Stacked prompt-to-image generations of rolling the latent input noise, leading to prompt-to-video. No ControlNet usage yet.
2D pan animation (noise crystal method). ControlNet used from here on out.
Parallax dolly effect. Some lighting inconsistencies.
Recrystallization for 2D pan. Transformation happens after 75% of the way through diffusion process.
Recrystallization for parallax. Transformation happens after 50% of the way through diffusion process, allowing lighting to be more stable. Noise masking used. Partial generation of the old skewed rock can be seen.
Example where noise crystal method excels (no occlusions). Fence blur created through latent linear interpolation.
Liquid noise method (using VAE upscale and flow maps). Noising is done to reduce VAE artifacts.
Liquid noise method (using VAE upscale and flow maps). Instead of noising, variance reduction is done to reduce VAE artifacts.
Zooming, showing sub-pixel movement capability with liquid noise (some artifacts visible).
Rotation, showing sub-pixel movement capability with liquid noise (some artifacts visible).
Attempt at rock parallax using liquid noise with layers. The varying speeds for each row cause some flickering.
Image-to-video using flow maps which move limbs and mouth. Image from Adventure Time. Use permitted by exceptions to copyright.
3-layer image-to-video scene (animating with layers). Turbulent hot air motion; tank motion.
Comparison of video-to-video style transfer without and with noise tracking. Note the facial distortion in the sample without noise tracking.