DDIM & Accelerated Sampling

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The DDPM Sampling Bottleneck
5. DDIM: Deterministic Sampling
6. The DDIM Update Rule
7. Probability Flow ODE
8. DPM-Solver: High-Order ODE Solvers
9. Consistency Models
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Derive the DDIM update rule and show it's a non-Markovian generalization of DDPM.
2. Prove that DDIM with $\eta=0$ is deterministic (same noise → same image).
3. Connect DDIM to the probability flow ODE.
4. Explain how DPM-Solver achieves high quality in 10-20 steps.
5. Describe consistency models and single-step generation.

Notation

• $\eta \in [0, 1]$ — stochasticity parameter (DDIM)
• $\tau = (t_1, \ldots, t_S)$ — subsequence of timesteps for accelerated sampling
• $S$ — number of sampling steps ($S \ll T$)

Core Intuition

DDPM requires $T = 1000$ sequential denoising steps for high-quality samples — far too slow for practical use. DDIM shows that by making the process deterministic (removing stochasticity), we can skip timesteps without loss of quality. The key insight: the DDPM training objective is compatible with a broader family of reverse processes, including non-Markovian ones that allow arbitrary step sizes.

The DDPM Sampling Bottleneck

DDPM sampling: $\mathbf{x}_{t-1} = \frac{1}{\sqrt{1-\beta_t}}(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)) + \sigma_t\mathbf{z}$.

Problem: Must go through all $T=1000$ steps sequentially. Each step requires one neural network evaluation. Total: 1000 forward passes for one image.

Naive subsampling (skip steps): Use every $k$-th step. Quality degrades rapidly — the stochastic noise accumulates incorrectly.

DDIM: Deterministic Sampling

Key insight (Song et al., 2020): The DDPM training loss depends only on the marginals $q(\mathbf{x}_t|\mathbf{x}_0)$, not on the joint $q(\mathbf{x}_{1:T}|\mathbf{x}_0)$. We can define a different (non-Markovian) joint that has the same marginals but allows larger steps.

DDIM defines:

When $\sigma_t = 0$: completely deterministic (no noise added). When $\sigma_t = \sqrt{(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_t)}\sqrt{\beta_t}$: recovers DDPM.

The DDIM Update Rule

Setting $\eta \in [0,1]$ as the interpolation parameter:

Interpretation:

• First predict $\hat{\mathbf{x}}_0$ from $\mathbf{x}_t$ using the noise estimate.
• Then interpolate toward $\hat{\mathbf{x}}_0$ at the noise level of step $t-1$.
• Optionally add noise $\sigma_t$ for stochasticity.

Accelerated sampling: Choose a subsequence $\tau = (t_1, \ldots, t_S)$ with $S \ll T$. Apply the DDIM update only at these timesteps. Works because the formula is valid for any step size.

Probability Flow ODE

In the continuous-time limit, DDIM ($\eta=0$) corresponds to the probability flow ODE:

where $f, g$ define the forward SDE. This ODE has the same marginals $p_t(\mathbf{x})$ as the stochastic process but is deterministic.

Consequence: Any ODE solver (Euler, Runge-Kutta, etc.) can be used for sampling. Higher-order solvers = fewer steps for same quality.

DPM-Solver: High-Order ODE Solvers

DPM-Solver (Lu et al., 2022): Apply high-order methods to the probability flow ODE.

DPM-Solver-2 (second-order): Uses two function evaluations per step (analogous to Heun's method):

DPM-Solver++ (third-order): Three evaluations per step, achieving excellent quality in 10–20 steps.

Result: DPM-Solver with 20 steps ≈ DDPM with 1000 steps in FID quality.

Consistency Models

Radical approach (Song et al., 2023): Learn a model that maps any noisy point $\mathbf{x}_t$ directly to $\mathbf{x}_0$ in a single step.

Consistency property: For any two points on the same ODE trajectory, the model should produce the same output:

Training: Distill from a pre-trained diffusion model or train from scratch with the consistency loss.

Result: Single-step generation (one forward pass → image). Quality below multi-step diffusion but dramatically faster.

Common Pitfalls

Pitfall 1. Using DDPM-trained models with DDIM sampling without any modification. It works (same training objective), but models trained with noise prediction may not be optimal for few-step DDIM — v-prediction parameterization works better.

Pitfall 2. Setting $S$ too small without adjusting the noise schedule. For $S < 10$ steps, the effective noise schedule is very aggressive; requires schedule adjustment.

Pitfall 3. Assuming consistency models replace diffusion. They trade quality for speed — multi-step diffusion still produces better samples when latency allows.

Summary

• DDIM generalizes DDPM to non-Markovian reverse processes with tunable stochasticity.
• At $\eta=0$: deterministic mapping from noise to image (invertible).
• Allows arbitrary step skipping: 50 steps ≈ 1000 steps in quality.
• Probability flow ODE connects to classical numerical methods.
• DPM-Solver (high-order ODE solver): 10–20 steps for excellent quality.
• Consistency models: single-step generation via trajectory consistency.

Exercises

Exercise 1. Derive the DDIM update (equation 2) from the non-Markovian forward process definition.

Exercise 2. Verify that DDIM with $\sigma_t = \sqrt{(1-\bar{\alpha}_{t-1})/(1-\bar{\alpha}_t)}\sqrt{\beta_t}$ recovers the DDPM update.

Exercise 3. For a linear noise schedule, compute the effective noise levels when sampling with $S=10$ evenly-spaced steps from $T=1000$.

Exercise 4. Explain why DDIM is invertible (can encode images to noise and back) while DDPM is not.

Exercise 5. Derive the second-order DPM-Solver update rule from the midpoint method applied to the probability flow ODE.