# Frame by frame

Period: Apr 2021 – mid 2022 · Era: 2015-2021 · Before the Dawn

Warping one image forward, then re-dreaming it toward a prompt.

Before video models, AI artists made animations one picture at a time. A program would nudge, zoom or rotate the previous frame, then redraw it to match a text prompt, hundreds of times over. Stitched together, those frames became the dreamy, ever-shifting films of 2021 and early 2022.

## How it worked

1. Write a prompt, or a dictionary of `{frame: prompt}` keyframes, in a Colab cell and pick a 2D or 3D animation mode.
2. Each frame starts from the previous one, warped by a camera schedule (`zoom`, `translation_x/y/z`, `rotation_3d_*` as `0:(1.02)` strings; 3D mode borrows depth from MiDaS).
3. The warped frame is re-optimized or re-diffused toward the prompt for a set number of steps (init strength and skip steps decide how much survives).
4. Wait hours on a Colab GPU while frames land in Google Drive, then assemble with ffmpeg and smooth 1–4 fps output with RIFE or FILM interpolation.
5. For live footage, feed each source frame as the init (Disco v4.1 "video input") or stylize a handful of keyframes and let EbSynth carry them across the shot.
6. Latent-walk videos went the other way: interpolate between seeds or prompt embeddings instead of warping, for the morphing "dreaming" look.

## Why it mattered

It invented the vocabulary, prompt keyframes, camera schedules, init strength, that every later tool inherited.

## Why it faded

Pixel-space diffusion took minutes per frame and everything flickered. Stable Diffusion (August 2022) made img2img ten to fifty times faster, and Deforum absorbed the schedule grammar within days.

## Tools of the era

- VQGAN+CLIP (2021): https://github.com/nerdyrodent/VQGAN-CLIP
- PyTTI (2021): https://github.com/pxl-pshr/pytti
- Disco Diffusion (2021): https://github.com/alembics/disco-diffusion
- EbSynth (2019): https://ebsynth.com

## Example videos and posts

- The Crow, Cannes 2022 winner, and how it was made · Glenn Marshall: https://www.youtube.com/watch?v=OPXo3yOtnZY

## On the timeline

- [VQGAN+CLIP Goes Viral](https://aifilmhistory.org/entry/vqgan-clip) (2021-04)
- [The Crow](https://aifilmhistory.org/entry/the-crow) (2022-09)
- [Disco Diffusion v1](https://aifilmhistory.org/entry/disco-diffusion) (2021-10-29)

## Sources

- Disco Diffusion changelog: https://github.com/alembics/disco-diffusion/blob/main/README.md
- VQGAN+CLIP animations notebook: https://github.com/chigozienri/VQGAN-CLIP-animations
- PyTTI 5 beta announcement: https://www.patreon.com/posts/pytti-5-beta-is-58923136
- Sagans: Coherence with Unreal Engine 5: https://www.unrealengine.com/spotlights/words-camera-action-creating-an-ai-music-video-with-unreal-engine-5
- A face transformed with Stable Diffusion and EbSynth: https://80.lv/articles/a-human-face-transformed-with-stable-diffusion-ebsynth

---
Source: https://aifilmhistory.org/workflows/frame-by-frame-clip-animation