# Shoot it, then restyle it

Period: Sep 2022 – 2023 · Era: 2023 · The Cambrian Explosion

Video-to-video: borrowing coherence from live footage.

Instead of asking a model to invent motion, artists filmed something real and let the model repaint it. Pose, depth and edges from the footage kept the shot coherent while the style changed completely, from live actors to anime, in a single pass.

## How it worked

1. Shoot or source footage and extract frames with ffmpeg.
2. Run preprocessors: OpenPose or DWPose skeletons, MiDaS or Zoe depth, lineart or canny edges.
3. Train a DreamBooth or LoRA on the target style or character.
4. Generate: per-frame img2img with ControlNet then EbSynth or TemporalKit to smooth; or WarpFusion's flow-warped, masked frames; or, from autumn 2023, AnimateDiff in sixteen-frame sliding windows with ControlNet stacks, IP-Adapter for style and prompt travel for scheduled changes.
5. Deflicker in DaVinci Resolve, upscale in Topaz, composite backgrounds.
6. Swap the performer: Viggle 1.0 (JST-1, March 20, 2024) took a driving video plus a character image and returned the character doing the motion; 2.0 two weeks later made it free, sharper and better with faces. A.I.Warper's Lil Yachty walk-out template turned it into the meme workflow of that spring, and the same character-onto-footage move came back as Act-One and Wan-Animate in 2025.

## Why it mattered

It gave filmmakers the first real control paradigm (pose, depth, edges) and proved the \"shoot it, then restyle it\" production model.

## Why it faded

It needs footage and it is a flicker-management discipline. Native video models removed the driving plate; Runway brought vid2vid back inside Gen-3 Alpha and Act-One, where this lineage lives now.

## Tools of the era

- Runway Gen-1 (2023): https://runway.com/research/gen-1
- ControlNet (2023): https://github.com/lllyasviel/ControlNet
- WarpFusion (2022): https://github.com/Sxela/WarpFusion
- TemporalKit (2023): https://github.com/CiaraStrawberry/TemporalKit
- AnimateDiff (2023): https://animatediff.github.io
- EbSynth (2019): https://ebsynth.com
- Topaz Video AI (2022): https://www.topazlabs.com
- Viggle (2024): https://viggle.ai
- LivePortrait (2024): https://github.com/KwaiVGI/LivePortrait

## Example videos and posts

- AnimateDiff + ControlNet vid2vid · Community tutorial: https://www.youtube.com/watch?v=-6r48urZBLU
- TemporalKit · Ciara Rowles: https://www.youtube.com/watch?v=z3YNHiuvxyg
- Viggle 2.0: motion capture has never been so easy · Viggle AI: https://www.youtube.com/watch?v=fyfJESrlHZE
- How to make the Lil Yachty walk-out meme with Viggle · Prompt Revolution: https://www.youtube.com/watch?v=cPXbeeoZ4Do
- Control AI actors with LivePortrait · Curious Refuge: https://www.youtube.com/watch?v=xfzK_6cTs58
- Stylizing real footage in five minutes · techhalla: https://x.com/techhalla/status/1828747537651572999
- Me in AI · Martin Haerlin: https://www.youtube.com/watch?v=kwcGc96HwPI

## On the timeline

- [Runway Gen-1](https://aifilmhistory.org/entry/runway-gen1) (2023-02)
- [Viggle 2.0 and the walk-out memes](https://aifilmhistory.org/entry/viggle-2-0) (2024-04-03)
- [(Me in AI) *2](https://aifilmhistory.org/entry/me-in-ai-2) (2023-09-11)

## Sources

- Runway Gen-1 research: https://runway.com/research/gen-1
- ControlNet paper (ICCV 2023): https://openaccess.thecvf.com/content/ICCV2023/html/Zhang_Adding_Conditional_Control_to_Text-to-Image_Diffusion_Models_ICCV2023_paper.html
- Corridor Digital: Anime Rock, Paper, Scissors: https://www.corridordigital.com/season/60
- WarpFusion: https://github.com/Sxela/WarpFusion
- AnimateDiff-Evolved for ComfyUI: https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved
- Introducing Viggle 1.0: https://www.youtube.com/watch?v=Q7cTDnNjNhY
- Curious Refuge: LivePortrait tutorial: https://www.youtube.com/watch?v=xfzK_6cTs58

---
Source: https://aifilmhistory.org/workflows/video-to-video-structure