AnimateDiff Part 1: Text2Video

AnimateDiff Part 1: Text2Video

AnimateDiff is text-to-video stable diffusion based method that aims to enhance a pre-existing text-to-image model by adding a motion modeling module. It is designed to understand how things move in animation.

Motion module: why is it important?

Implementation notes: under an official repository, the motion modules architecture is defined in a script here and initialized using .yaml files here.

The module is:

1. Initialized and appended to the frozen base model.
2. Trained on video clips to acquire a meaningful motion prior.
3. Once trained, it’s injected into any personalized version derived from the same base model.

Reason 1: Personalized T2V model

It’s crucial for individuals and businesses to minimize costs during training or fine-tuning, especially when dealing with vast amounts of data. Various methods have been introduced (Textual Inversion, DreamBooth, LoRA) to empower users to incorporate new domains into pre-trained text-to-inage models with a limited number of images.

In the case of AnimateDiff, we have personalized text-to-video.

Keeping the pre-trained weights of the base model intact (SD base model) allows the motion module to be trained and the feature space of the original model unchanged (preserving what was learned previously).

Reason 2: Consistency over frames

The goal of motion-module is to add smoothness and consistency across the animation frames in each batch.

As outlined in the paper: “newly inserted motion module operates across frames in each batch to achieve motion smoothness and content consistency in the animation clips”.

Prompt example: a medium shot of a single sunflower in a grass field, moving in a wind, consistent and smooth movements, vintage 90s anime, 4k resolution, unreal 5 engine render, burnt orange and neutral colors.

How motion module is designed

• Left: insertion of trained motion module between model’s image layers.
• Right: module architecure with zero-initialized output layer.

Fig. 3, https://arxiv.org/abs/2307.04725

1. Motion module requires 5D video tensor input: batch × channels × frames × height × width.

• To ensure the compatibility of SD with motion module, the special technique was introduced: each 2D convolution and attention layer in the original image model were transfored into spatial-only pseudo-3D layers (Guo et al., 2023, p. 5).

2. The process involves reshaping the spatial dimensions of the feature map, projecting it, and passing it through self-attention blocks using the Attention(Q, K, V) operation.

• The core design element is “Vanilla temporal transformer”, which consists of **self-attention blocks** working along the time axis. It helps capturing the temporal relationships between features at the same location across different frames.
• The output projection layer is initialized with zeros.

Usage and parameters overview

Here are the tasks you are able to perform with AnimateDiff:

• Text2Video — generate a video from prompt: the best reference would be the official repo.
• Img2Video — animation with a given image can be found here. Implementation is based on official repo with introducing image latents to animation pipeline.
• Video2Video — animation with a given video (works in pair with ControlNet): a guide for ComfyUI.

Default motion modules can be found under HF AnimateDiff repo.

Practicle example

Let’s say we want to generate an animation with a prompt below.

((epic)), ((Nolan-esque)), ((cinematic masterpiece)), vast golden wheat fields stretching to the horizon, a lone girl standing amidst the swaying stalks, sunlight casting long shadows, a gentle breeze stirring the golden sea, an enigmatic aura, time-bending moments, emotions unraveling in each frame, a story untold, the fusion of nature's beauty and human

Running inference on AnimateDiff can be performed through various methods. In this article, we will focus on a convenient approach using the Automatic1111 WebUI. Similar processes is possible with ComfyUI and via API calls to automatic, for instance, using this extension.

Using default parameters

We start with default parameters, leaving FPS at 8 and choosing DDIM sampling method.

Default parameters

Output using default parameters

Choosing parameters for extended animation

Let’s try to make animation longer, e.g. up to 3 seconds at 24 FPS.

1. Context batch size — determines the number of frames passed into the motion module at once. The motion module for SD1.5 were trained with 16 frames, hence the best results expected at this value.
2. Number of frames:

• Value 0 means output number of frames == context batch size
• If the value is smaller than context batch size (excluding 0), the output GIF will only include the first set of frames, while the subsequent frames will be saved as PNGs.

3. Interp X makes the animation smoother by inserting additional frames between each original frame through frame interpolation. Using this parmeter increases the output frame rate.

To illustrate, suppose we want to extend an animation to 3 seconds at 24 FPS using Interp X. The formula for calculation is:

Interp X = (Desired Output Frame Rate / FPS) — 1

By setting Interp X to 1, we can use an FPS of 12 and still achieve 24 frames as output.

Then, the numer of frames can be obtained using:

*Number of frames = (Duration +1*) FPS***

In this our case we have: (3 + 1) * 12 = 37 frames

4. Overlap — controls the overlapping frames in a context, mostly benificial in cases where number of frames is greater than the context batch size.

5. Closed loop set to A to makes the last frame identical to the first frame, and prompt travel is interpolated to establish a closed loop.

6. Exploring different Sampling methods and Upscalers can yield improvements in results. For example, opting for DDIM as sampler may enhance generation speed, whereas DPM++ 2M frequently strikes a balance between speed and quality.

Parameters for extended animation

Output using tuned parameters

Using motion LoRA

Motion LoRAs, employed similarly to standard LoRAs, can introduce camera movement to the video. The corresponding LoRA checkpoints can be found in the same folder as the motion modules here. To integrate Motion LoRAs, trigger words are utilized.

For instance, to infuse a dynamic element like a gradual “tilt up” motion, append **lora:v2_lora_TiltUp:0.2** to the end of the prompt.

To introduce additional motion, the “prompt travel” technique can be applied at frames 0 and 10. Consequently, the prompt would appear as follows:

((cinematic masterpiece)), golden wheat fields bathed in sunlight, peaceful scenery 0: lone girl slowly walking 10: head rotation

lora:v2_lora_TiltUp:0.2

motion LoRA + prompt travel

Closing words

In this part of AnimateDIff series we’ve explored the Motion Module and its role in creating personalized text-to-video models and ensuring consistency across frames.

Moving forward, we’ll delve further into image2video and video2video cases, examining parameters for enhanced animation generation.

Stay tuned for insights into how AnimateDiff adapts to different scenarios, and how these methods can be employed in your animation generation.

References

1. Guo, et al. (2023). “AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning.”
2. Stable Diffusion Art (2023) “AnimateDiff: Easy text-to-video.” Official Website
3. “Official implementation of AnimateDiff.” GitHub repository
4. “AnimateDiff for Stable Diffusion WebUI.” GitHub repository