Sep 2022 – 2023
Shoot it, then restyle it
Video-to-video: borrowing coherence from live footage.
Instead of asking a model to invent motion, artists filmed something real and let the model repaint it. Pose, depth and edges from the footage kept the shot coherent while the style changed completely, from live actors to anime, in a single pass.
Tutorials6
AnimateDiff + ControlNet vid2vid
Community tutorial · YouTube · Nov 2023
How it worked
- Shoot or source footage and extract frames with ffmpeg.
- Run preprocessors: OpenPose or DWPose skeletons, MiDaS or Zoe depth, lineart or canny edges.
- Train a DreamBooth or LoRA on the target style or character.
- Generate: per-frame img2img with ControlNet then EbSynth or TemporalKit to smooth; or WarpFusion's flow-warped, masked frames; or, from autumn 2023, AnimateDiff in sixteen-frame sliding windows with ControlNet stacks, IP-Adapter for style and prompt travel for scheduled changes.
- Deflicker in DaVinci Resolve, upscale in Topaz, composite backgrounds.
- Swap the performer: Viggle 1.0 (JST-1, March 20, 2024) took a driving video plus a character image and returned the character doing the motion; 2.0 two weeks later made it free, sharper and better with faces. A.I.Warper's Lil Yachty walk-out template turned it into the meme workflow of that spring, and the same character-onto-footage move came back as Act-One and Wan-Animate in 2025.
Example Films3
Related on the timeline1