Classifier-Free Guidance

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Classifier Guidance: Background
6. Classifier-Free Guidance: Formulation
7. Joint Training Procedure
8. Score Function Derivation
9. The Guidance Scale
10. Quality-Diversity Tradeoff
11. Computational Cost
12. Worked Examples
13. Connection to the Broader Curriculum
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. State classifier guidance via Bayes' rule on the score function.
2. Derive classifier-free guidance (CFG) as implicit classifier amplification without a separate model.
3. Explain joint training with random condition dropout.
4. Prove the equivalence between guided noise prediction and modified score $\nabla \log p(\mathbf{x}_t \mid c) + w \nabla \log p(c \mid \mathbf{x}_t)$.
5. Analyze the quality-diversity tradeoff as a function of guidance scale $w$.
6. Compute the 2× inference overhead of CFG.

Prerequisites

• Reverse Process — $\boldsymbol{\epsilon}_\theta$, score connection
• Forward Diffusion

Notation

• $c$ — Conditioning variable (class, text, etc.)
• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)$ — Conditional noise prediction
• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ — Unconditional prediction
• $w$ — Guidance scale
• $\varnothing$ — Null / dropped condition

Core Intuition

Conditional generation requires sampling from $p(\mathbf{x} \mid c)$ — e.g., images given text prompts. Classifier guidance (Dhariwal & Nichol, 2021) modifies the score using a separately trained classifier $p(c \mid \mathbf{x}_t)$ at every noise level — expensive and limited to tasks with classifiers.

Classifier-free guidance (Ho & Salimans, 2022) trains a single noise predictor in both conditional and unconditional modes, then extrapolates between them at inference. No separate classifier needed — yet equivalent to amplifying an implicit classifier gradient.

CFG is the standard technique in Stable Diffusion, DALL·E 2, and Imagen, enabling high-fidelity text-to-image generation.

Series context. Volume III, Chapter 10, Part III.

Classifier Guidance: Background

Theorem 1 (Bayesian Score Decomposition).

Definition 1 (Classifier-Guided Score).

where $w_{\text{CG}} > 1$ amplifies conditioning.

Limitation. Requires training $p(c \mid \mathbf{x}_t)$ on noisy inputs at all $t$ — cumbersome for text conditioning.

Classifier-Free Guidance: Formulation

Definition 2 (CFG Noise Prediction).

equivalently:

$w$ is the guidance scale; $\varnothing$ is null condition (empty prompt).

Joint Training Procedure

Definition 3 (Condition Dropout). During training, with probability $p_{\text{uncond}}$ (typically 0.1–0.2), replace condition $c$ with $\varnothing$:

Proposition 1. The same network $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c_{\text{train}})$ learns:

• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)$ — conditional denoising
• $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)$ — unconditional denoising

Using standard Reverse Process loss (13).

Score Function Derivation

From Reverse Process: $\boldsymbol{\epsilon}_\theta \propto -\nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)$.

Proposition 2 (Implicit Classifier Gradient).

In noise form:

Theorem 2 (CFG as Amplified Classifier Guidance). CFG (3) corresponds to modified score:

Equivalent to classifier guidance with scale $(1 + w)$ without explicit classifier.

Proof. Substitute (6) into guided score; rearrange to (3). $\blacksquare$

The Guidance Scale

• 0 — Standard conditional sampling
• 1–4 — Moderate enhancement
• 7.5 — Stable Diffusion default
• $\gg 1$ — High fidelity, low diversity, artifacts

Proposition 3. At $w = 0$: $\tilde{\boldsymbol{\epsilon}} = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, c)$ — pure conditional.

Quality-Diversity Tradeoff

Theorem 3 (Sharpened Distribution). CFG samples from effective distribution:

Proof sketch. From (8): $\log \tilde{p} = (1+w)\log p(\mathbf{x}|c) - w\log p(\mathbf{x}) + C$. $\blacksquare$

Consequences:

• High $w$: concentrates on high $p(c \mid \mathbf{x})$ regions — sharp, mode-collapsed
• Low $w$: closer to true conditional — diverse but less aligned
• CLIP score increases with $w$; FID has optimal $w$

Computational Cost

Proposition 4. Each sampling step requires two forward passes: conditional and unconditional. Total inference cost $\approx 2\times$ baseline.

Both passes can be batched: stack $(\mathbf{x}_t, \mathbf{x}_t)$ with conditions $(c, \varnothing)$.

Worked Examples

Example 1: $w = 7.5$

$\tilde{\boldsymbol{\epsilon}} = -6.5\,\boldsymbol{\epsilon}_\varnothing + 7.5\,\boldsymbol{\epsilon}_c$ — strong push toward conditional.

Example 2: Identical Predictions

If $\boldsymbol{\epsilon}_c = \boldsymbol{\epsilon}_\varnothing$, CFG has no effect regardless of $w$.

Connection to the Broader Curriculum

• Reverse Process — base sampling
• Bayes' Theorem — score decomposition
• Text conditioning via cross-attention (future article)
• RAG — conditioning in LLM domain

Common Pitfalls and Misconceptions

Pitfall 1: $w = 0$ is not unconditional — it's conditional without guidance.

Pitfall 2: Too-high $w$ causes oversaturation and artifacts.

Pitfall 3: Forgetting condition dropout during training makes $\boldsymbol{\epsilon}_\varnothing$ undefined.

Pitfall 4: Confusing $w$ in (3) vs. $w_{\text{CG}}$ in (2) — related by offset.

Research Perspective

CFG (Ho & Salimans, 2022). Applied at scale in Stable Diffusion, Imagen. Research: guidance distillation (single-pass), autoguidance, negative prompting as $c_{\text{neg}}$.

Summary of Takeaways

• CFG — $\tilde{\boldsymbol{\epsilon}} = (1+w)\boldsymbol{\epsilon}_c - w\boldsymbol{\epsilon}_\varnothing$
• Implicit classifier — $\boldsymbol{\epsilon}_\varnothing - \boldsymbol{\epsilon}_c$
• Effective distribution — $\tilde{p} \propto p(\mathbf{x} — c)^{1+w}/p(\mathbf{x})^w$
• Training — Drop condition with prob. $p_{\text{uncond}}$
• Cost — 2× forward passes

Next: Tokenization (Volume III, LLMs)

Exercises

Exercise 1. Derive (8) from (3) using score–noise identity.

Exercise 2. Prove (9) from modified score.

Exercise 3. What happens as $w \to \infty$?

Exercise 4. Design training schedule for $p_{\text{uncond}}$.

Exercise 5. Compare CFG to Logistic Regression classifier guidance conceptually.

Exercise 6. Batch CFG: write combined forward pass dimensions.

Exercise 7. Negative prompting as alternative condition.

Exercise 8. Why does CFG improve CLIP score but harm diversity?