# Image-to-video era begins

Period: Jul 2023 – 2024 · Era: 2024 · The Great Convergence

Image-to-video becomes the professional workflow: art-direct the frame, then add motion.

Rather than describing a shot in words, artists started with a still image (e.g. Midjourney or Stable Diffusion), then asked a video model to bring it to life. Art direction moved into the image, and tools appeared for painting where motion should happen (e.g. Motion Brush), setting a start and end frame, and steering the camera. This is still the backbone of most AI filmmaking. Two upscalers played a key role in the process: Magnific's creative upscaler added lost or missing details on input images, and Topaz Video AI cleaned up the low-resolution clips after generation.

## How it worked

1. Art-direct still images in Midjourney (v5.2, then style references in 2024) or SDXL, then upscale in Magnific (from November 2023): its creativity, HDR and resemblance sliders hallucinate skin pores, fabric and set detail into the frame before the video model sees it, which is why so many 2024 films look sharper than the models that made them.
2. Feed the still to Gen-2, Pika or Stable Video Diffusion with a short motion prompt and Director Mode camera values.
3. Paint motion: Runway Motion Brush (November 2023, one region) and Multi Motion Brush (January 2024, five regions with x/y/z and ambient).
4. Generate multiple takes per shot (5-10 was common), keep the coherent shots, then upscale to HD, denoise, and smooth the frame rate using Topaz Video AI.
5. Add dialogue via Pika Lip Sync with ElevenLabs.
6. 2024 added keyframes (Luma Dream Machine start and end frames, Kling start/end frame control), Kling 1.5 Motion Brush with six trajectories, Runway Gen-3 Alpha at ten seconds with Act-One performance capture, and Hailuo's human motion.
7. Generate the background as its own plate: erase the character (Runway Erase & Replace), animate character and environment separately, key the character out with the green-screen tool and layer them, then keyframe its position by hand (Timmy, April 2024).
8. First and last frame came back around in 2025 (Kling 2.1, then Veo 3.1 Frames to Video): author both endpoint stills and let the model bridge a longer, controlled shot.

## Why it mattered

Authorship moved from the prompt to the still frame, putting art direction back in the artist's hands and making motion controls the new interface. It also fixed the shape of the pipeline: a creative upscaler on the frame, a video upscaler on the clip.

## Why it faded

It never faded; it is still the base professional workflow. Its weakness, one anchor image cannot carry a character across shots, is exactly what the multi-reference era attacks.

## Tools of the era

- Midjourney (2022): https://midjourney.com
- Magnific (2023): https://magnific.ai
- Runway Motion Brush (2023): https://runway.com
- Stable Video Diffusion (2023): https://stability.ai/news/stable-video-diffusion-open-ai-video-model
- Luma Dream Machine (2024): https://lumalabs.ai/dream-machine
- Kling 1.x (2024): https://klingai.com
- Runway Gen-3 Alpha (2024): https://runway.com/research/introducing-gen-3-alpha
- Runway Act-One (2024): https://runway.com/research/introducing-act-one
- Hailuo video-01 (2024): https://hailuoai.video
- Pika (2023): https://pika.art
- Topaz Video AI (2022): https://www.topazlabs.com

## Example videos and posts

- How to use Runway's Motion Brush · AI Video School: https://www.youtube.com/watch?v=soZbh9O25ig
- Runway Gen-2 Multi-Motion Brush Tutorial · AI Video School: https://www.youtube.com/watch?v=P41bfQrsgDY
- Genesis: 316 prompts, 310 clips, 44 used · Nicolas Neubert: https://x.com/iamneubert/status/1684989102213476359
- Battalion: image-to-video and text-to-video only · Dave Clark: https://x.com/Diesol/status/1839351148412825605
- The AI film era has begun: Dead Sea masterclass and cost breakdown · Theoretically Media: https://www.youtube.com/watch?v=fZ4z4Z2GT2Y
- Create cinematic AI videos with Runway Gen-3 · Curious Refuge: https://www.youtube.com/watch?v=gyCg0yv3Njw
- Mastering Luma Dream Machine, basics to keyframes · Theoretically Media: https://www.youtube.com/watch?v=lbXZ4YmVNE8
- Can AI make a Star Wars film? Midjourney to Pika to ElevenLabs · Curious Refuge: https://www.youtube.com/watch?v=qvoxme5Vq8g
- Consistent characters in Midjourney (--cref) · Curious Refuge: https://www.youtube.com/watch?v=7BO_TocisAc
- Making Seamless Loops with Runway Gen-2 · Runway: https://www.youtube.com/watch?v=sTsft7XCqO8
- Clean plate, green screen and layered animation in Runway · Timmy: https://x.com/IXITimmyIXI/status/1781365664009142506
- A Modern Man: Kling 2.1 first and last frame · Dave Clark: https://x.com/Diesol/status/1959701086979358815

## On the timeline

- [Magnific AI](https://aifilmhistory.org/entry/magnific-ai-launch) (2023-11-27)
- [Topaz Video AI v3](https://aifilmhistory.org/entry/topaz-video-ai-v3) (2022-10)
- [Stable Video Diffusion](https://aifilmhistory.org/entry/stable-video-diffusion) (2023-11)
- [Luma Dream Machine](https://aifilmhistory.org/entry/luma-dream-machine) (2024-06-12)
- [Kling 1.0](https://aifilmhistory.org/entry/kling-1-0) (2024-06)
- [Runway Gen-3 Alpha](https://aifilmhistory.org/entry/runway-gen3-alpha) (2024-06)
- [Hailuo Video-01](https://aifilmhistory.org/entry/hailuo-video-01) (2024-08)
- [Kling 1.5](https://aifilmhistory.org/entry/kling-1-5) (2024-09-19)
- [Our T2 Remake](https://aifilmhistory.org/entry/our-t2-remake) (2024-03-06)
- [Genesis](https://aifilmhistory.org/entry/genesis-neubert) (2023-07-26)
- [Dead Sea](https://aifilmhistory.org/entry/dead-sea) (2024-07-11)

## Sources

- Magnific launch post: https://magnific.substack.com/p/finally-we-are-launching-magnific
- Topaz Video AI v3.0 release notes: https://community.topazlabs.com/t/topaz-video-ai-v3-0-0/35818
- Motion Brush: https://www.marktechpost.com/2023/11/17/runways-new-motion-brush-feature-in-gen-2-will-allow-to-add-controlled-movement-to-your-generations/
- Multi Motion Brush: https://venturebeat.com/ai/runway-gen-2-adds-multiple-motion-controls-to-ai-videos-with-multi-motion-brush
- Luma Keyframes: https://radiancefields.com/luma-ai-launches-keyframes-for-dream-machine
- Runway Act-One: https://runway.com/research/introducing-act-one
- Hailuo image-to-video: https://venturebeat.com/ai/hailuo-gets-feature-competitive-launching-image-to-video-ai-generation-capability
- Theoretically Media: films and tool lists: https://theoreticallymedia.com/films/

---
Source: https://aifilmhistory.org/workflows/image-to-video