Jul 2023 – 2024
Image-to-video era begins
Image-to-video becomes the professional workflow: art-direct the frame, then add motion.
Rather than describing a shot in words, artists started with a still image (e.g. Midjourney or Stable Diffusion), then asked a video model to bring it to life. Art direction moved into the image, and tools appeared for painting where motion should happen (e.g. Motion Brush), setting a start and end frame, and steering the camera. This is still the backbone of most AI filmmaking. Two upscalers played a key role in the process: Magnific's creative upscaler added lost or missing details on input images, and Topaz Video AI cleaned up the low-resolution clips after generation.
Tutorials8
How to use Runway's Motion Brush
AI Video School · YouTube · Nov 2023
How it worked
- Art-direct still images in Midjourney (v5.2, then style references in 2024) or SDXL, then upscale in Magnific (from November 2023): its creativity, HDR and resemblance sliders hallucinate skin pores, fabric and set detail into the frame before the video model sees it, which is why so many 2024 films look sharper than the models that made them.
- Feed the still to Gen-2, Pika or Stable Video Diffusion with a short motion prompt and Director Mode camera values.
- Paint motion: Runway Motion Brush (November 2023, one region) and Multi Motion Brush (January 2024, five regions with x/y/z and ambient).
- Generate multiple takes per shot (5-10 was common), keep the coherent shots, then upscale to HD, denoise, and smooth the frame rate using Topaz Video AI.
- Add dialogue via Pika Lip Sync with ElevenLabs.
- 2024 added keyframes (Luma Dream Machine start and end frames, Kling start/end frame control), Kling 1.5 Motion Brush with six trajectories, Runway Gen-3 Alpha at ten seconds with Act-One performance capture, and Hailuo's human motion.
- Generate the background as its own plate: erase the character (Runway Erase & Replace), animate character and environment separately, key the character out with the green-screen tool and layer them, then keyframe its position by hand (Timmy, April 2024).
- First and last frame came back around in 2025 (Kling 2.1, then Veo 3.1 Frames to Video): author both endpoint stills and let the model bridge a longer, controlled shot.
Example Films5
More making-ofs2
How films were actually made, from the people who made them.
Related on the timeline8