Flow Models for Image & Video Generation

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Stable Diffusion 3 Architecture
5. Flux: Rectified Flow in Production
6. Classifier-Free Guidance with Flows
7. Text Conditioning Mechanisms
8. Few-Step Distillation
9. Video Generation via Temporal Flows
10. Flow vs Diffusion: When to Use Which
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Describe the SD3 architecture and its use of flow matching.
2. Explain classifier-free guidance in the flow matching framework.
3. Compare distillation approaches for few-step flow models.
4. Extend flow matching to temporal (video) generation.
5. Evaluate when flow models outperform diffusion models.

Notation

• CFG — classifier-free guidance
• $w$ — guidance scale
• $v_\theta(\mathbf{x}, t, c)$ — conditioned velocity field

Core Intuition

Modern production text-to-image models (SD3, Flux, DALL-E 3) have largely switched from the VP-SDE diffusion formulation to flow matching. Why? Straighter trajectories → fewer sampling steps → faster generation. The architecture (DiT/MMDiT) and conditioning (text encoders, CFG) remain similar, but the underlying generative process uses rectified flow instead of DDPM/DDIM.

Stable Diffusion 3 Architecture

Key choices:

1. Flow matching with linear interpolation path.
2. MMDiT (Multimodal DiT): Separate streams for text and image tokens with cross-attention.
3. Rectified flow training objective: $\|v_\theta(\mathbf{x}_t, t, c) - (x_1 - x_0)\|^2$.
4. Latent space: 16-channel VAE latent (vs 4-channel in SD 1.5/2.1).
5. Text encoders: CLIP-L, CLIP-G, and T5-XXL for rich conditioning.

Architecture (MMDiT block):

• Image tokens: self-attention + cross-attention with text.
• Text tokens: self-attention + cross-attention with image.
• Bidirectional interaction (both modalities attend to each other).

Sampling: 20-50 Euler steps with CFG.

Flux: Rectified Flow in Production

Flux (Black Forest Labs): Streamlined flow matching architecture:

• Single-stream DiT (no separate text/image streams).
• Rotary positional embeddings for both text and image.
• Linear flow matching with simple Euler sampling.
• Guidance distillation for CFG-free inference.

Key innovations:

1. Guidance-distilled model: train a student that doesn't need CFG (saves 2x compute at inference).
2. Timestep-shifted schedule: allocate more steps to $t \approx 0$ (high noise) where the velocity changes rapidly.
3. 12B parameter model achieving state-of-the-art quality.

Classifier-Free Guidance with Flows

Standard CFG (diffusion): $\tilde{\epsilon} = (1+w)\epsilon_\theta(x, c) - w\epsilon_\theta(x, \emptyset)$.

CFG for flows: Same principle, applied to velocity:

Interpretation: Amplify the direction toward the conditioned manifold; suppress the unconditional component.

Training: Drop conditioning with probability $p_{\text{drop}} = 0.1$ during training (same as diffusion).

Guidance scale: Typical $w = 3$–$7$ (lower than diffusion's $w = 7.5$–$15$ because flow paths are already better directed).

Text Conditioning Mechanisms

Multi-encoder approach (SD3/Flux):

1. CLIP-L (77 tokens): Semantic understanding, style control.
2. CLIP-G (77 tokens): Higher-capacity semantics.
3. T5-XXL (256 tokens): Detailed text understanding, compositional reasoning.

Conditioning injection:

• Cross-attention: Standard method. Image tokens attend to text tokens.
• Adaptive LayerNorm (adaLN): Scale and shift image features based on text embedding. More parameter-efficient.
• Concatenation: Append text tokens to image token sequence. Used in Flux's single-stream architecture.

Few-Step Distillation

Goal: Reduce 50 steps → 1-4 steps while maintaining quality.

Progressive distillation:

1. Teacher: 50-step model.
2. Student trained to match 2 teacher steps in 1 step.
3. Repeat: new teacher (25 steps) → new student (1 step matches 2).
4. After $\log_2(50) \approx 6$ rounds: 1-step student.

Adversarial distillation (SDXL Turbo, SD3 Turbo):

• Discriminator judges if generated images are realistic.
• Student learns to produce realistic images in 1-4 steps.
• Often combined with reconstruction loss.

Consistency distillation:

• Train student to map any $\mathbf{x}_t$ directly to $\mathbf{x}_1$.
• Uses the teacher ODE trajectory as ground truth.
• Natural fit for rectified flows (already near-straight).

Video Generation via Temporal Flows

Extension to video: Add temporal dimension to the flow:

Temporal attention: Alternate between:

• Spatial attention (within each frame).
• Temporal attention (across frames at each spatial position).

Causal generation: Generate frames autoregressively; condition on previous frames for consistency.

Challenges:

• Memory: $F$ frames × full resolution is expensive.
• Temporal coherence: objects must persist across frames.
• Motion: velocity field must encode both spatial and temporal dynamics.

Flow vs Diffusion: When to Use Which

Use flow matching when:

• Few-step generation is critical (real-time applications).
• Training simplicity is valued (no noise schedule tuning).
• Latent-space generation (already common for images).

Use diffusion (VP-SDE) when:

• Maximum sample quality is needed (SDE sampling with many steps).
• Established infrastructure exists (pre-trained diffusion models).
• Pixel-space generation with specific noise schedules tuned.

In practice: The distinction is disappearing. SD3 and Flux use flow matching; DALL-E 3 uses a hybrid. The architecture (DiT, U-Net) and conditioning matter more than the ODE vs SDE choice.

Common Pitfalls

Pitfall 1. Using too many Euler steps with rectified flows. After proper training, 20-28 steps suffice for SD3-quality results. More steps waste compute.

Pitfall 2. Applying large CFG guidance scales ($w > 10$) with flow models. Flows are already well-directed; excessive guidance causes saturation and artifacts. Use $w = 3$–$5$.

Pitfall 3. Expecting flow matching alone to fix composition errors. Text understanding comes from the text encoder and cross-attention, not the generative process.

Summary

• SD3/Flux: Production flow matching systems with DiT/MMDiT architectures.
• CFG for flows: Same principle as diffusion; lower guidance scales needed.
• Distillation: Progressive, adversarial, or consistency-based for 1-4 step generation.
• Video: Temporal extension with spatial-temporal attention.
• Flow vs diffusion: Converging in practice; flow matching wins on speed, diffusion on maximum quality with many steps.

Exercises

Exercise 1. Compare the FLOPs per sampling step for SD3 (DiT-XL/2, 650M) with 28 steps vs SD 1.5 (U-Net, 860M) with 50 DDIM steps.

Exercise 2. Derive the CFG-augmented flow equation and show it's equivalent to modifying the score with guidance.

Exercise 3. Design a progressive distillation schedule for a 32-step flow model. How many distillation rounds and what's the total training cost?

Exercise 4. For video generation with 16 frames at 512x512: compute the memory requirements for the latent tensor (16-channel VAE, batch size 1).

Exercise 5. Explain why guidance-distilled models (Flux) produce better results than applying post-hoc CFG (hint: distribution mismatch during training vs inference).