Score Matching & Score-Based Models

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Score Function
5. Why Scores Instead of Densities?
6. Explicit Score Matching
7. Denoising Score Matching
8. Langevin Dynamics Sampling
9. Noise-Conditional Score Networks (NCSN)
10. Connection to Diffusion Models
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Define the score function and explain why it avoids the normalization constant.
2. Derive denoising score matching from first principles.
3. Explain Langevin dynamics and prove it converges to the target distribution.
4. Connect score-based models to DDPM through the noise prediction equivalence.
5. Explain why multi-scale noise is needed for effective score estimation.

Notation

• $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ — score function
• $\mathbf{s}_\theta(\mathbf{x}, \sigma)$ — learned score network
• $\sigma_1 > \sigma_2 > \ldots > \sigma_L$ — noise schedule
• $\epsilon$ — step size for Langevin dynamics

Core Intuition

We want to learn a generative model of data $p_{\text{data}}(\mathbf{x})$, but directly estimating the density requires computing an intractable normalization constant $Z = \int p(\mathbf{x})d\mathbf{x}$. The score function $\nabla_\mathbf{x}\log p(\mathbf{x})$ is the gradient of the log-density — it tells us the direction of increasing probability at any point, and crucially, it doesn't depend on $Z$.

The Score Function

Properties:

• Points toward modes of the distribution.
• Magnitude indicates steepness of the density landscape.
• Independent of normalization: $\nabla_\mathbf{x}\log\frac{p^*(\mathbf{x})}{Z} = \nabla_\mathbf{x}\log p^*(\mathbf{x})$.

For Gaussian $p(\mathbf{x}) = \mathcal{N}(\boldsymbol{\mu}, \sigma^2\mathbf{I})$:

The score points from $\mathbf{x}$ toward the mean $\boldsymbol{\mu}$, scaled by $1/\sigma^2$.

Why Scores Instead of Densities?

Problem with density estimation: For energy-based models $p_\theta(\mathbf{x}) = e^{-E_\theta(\mathbf{x})}/Z_\theta$, computing $Z_\theta = \int e^{-E_\theta(\mathbf{x})}d\mathbf{x}$ is intractable in high dimensions.

Score bypasses $Z$: Training a score model $\mathbf{s}_\theta(\mathbf{x}) \approx \nabla_\mathbf{x}\log p(\mathbf{x})$ requires no normalization constant.

Sampling with scores: Once we have the score, we can generate samples using Langevin dynamics — following the score uphill with noise injection.

Explicit Score Matching

Objective (Fisher divergence):

Problem: We don't know $\nabla_\mathbf{x}\log p_{\text{data}}(\mathbf{x})$ — that's what we're trying to learn!

Hyvärinen's trick: Using integration by parts (Stein's identity):

This only involves the model $\mathbf{s}_\theta$ and data samples — no ground-truth score needed. But computing $\text{tr}(\nabla_\mathbf{x}\mathbf{s}_\theta)$ (the Jacobian trace) is expensive for high-dimensional $\mathbf{x}$.

Denoising Score Matching

Key insight (Vincent, 2011): Instead of matching the data score, match the score of noisy data.

Perturb data: $\tilde{\mathbf{x}} = \mathbf{x} + \sigma\boldsymbol{\epsilon}$, where $\boldsymbol{\epsilon} \sim \mathcal{N}(0, \mathbf{I})$.

The score of the noisy distribution has a simple form:

Denoising score matching objective:

Equivalence: Minimizing DSM is equivalent to minimizing ESM for the noisy distribution. As $\sigma \to 0$, we recover the true data score.

Langevin Dynamics Sampling

Given the score $\mathbf{s}_\theta(\mathbf{x})$, generate samples via:

Theorem: As $\epsilon \to 0$ and $t \to \infty$, $\mathbf{x}_t$ converges to a sample from $p(\mathbf{x})$ (under mild regularity conditions).

Intuition: The score term pushes $\mathbf{x}$ toward high-density regions; the noise term ensures exploration and prevents collapse to modes.

Noise-Conditional Score Networks (NCSN)

Problem with single-noise DSM: At low noise, the score is inaccurate in low-density regions (no training data there). At high noise, the score is accurate everywhere but blurs fine details.

Solution (Song & Ermon, 2019): Train a single network on multiple noise levels:

with $\sigma_1 > \ldots > \sigma_L$ and $\lambda(\sigma_i) = \sigma_i^2$ (normalization).

Annealed Langevin dynamics: Sample by running Langevin at each noise level sequentially, from largest $\sigma_1$ (coarse structure) to smallest $\sigma_L$ (fine details).

Connection to Diffusion Models

Theorem (Song et al., 2021): DDPM's noise prediction $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)$ and the score function are related by:

The DDPM training objective (predict noise) is exactly denoising score matching (up to a weighting factor).

Unified view: Both diffusion models and score-based models learn the score at different noise levels. They differ only in how they use the score for sampling (discrete steps vs Langevin vs SDEs).

Common Pitfalls

Pitfall 1. Training a score network without noise conditioning. In low-density regions, the score is poorly estimated, causing Langevin dynamics to diverge.

Pitfall 2. Using too few Langevin steps per noise level. Insufficient mixing leads to poor sample quality, especially at lower noise levels.

Pitfall 3. Confusing the score with the gradient of the loss. The score $\nabla_\mathbf{x}\log p(\mathbf{x})$ is a function of the data point, not the model parameters.

Summary

• The score function $\nabla_\mathbf{x}\log p(\mathbf{x})$ avoids the intractable normalization constant.
• Denoising score matching provides a tractable training objective: predict the noise direction.
• Langevin dynamics converts scores into samples by gradient ascent with noise.
• Multi-scale noise (NCSN) ensures accurate scores everywhere.
• DDPM noise prediction = denoising score matching (equivalent formulations).

Exercises

Exercise 1. Compute the score function for a mixture of two Gaussians $p(\mathbf{x}) = \frac{1}{2}\mathcal{N}(\mu_1, \sigma^2) + \frac{1}{2}\mathcal{N}(\mu_2, \sigma^2)$.

Exercise 2. Prove that denoising score matching (equation 6) is equivalent to explicit score matching for the noisy distribution.

Exercise 3. Derive the stationary distribution of Langevin dynamics (equation 7) using the Fokker-Planck equation.

Exercise 4. Show that $\mathbf{s}_\theta(\mathbf{x}_t, t) = -\boldsymbol{\epsilon}_\theta/\sqrt{1-\bar{\alpha}_t}$ by differentiating the DDPM forward process.

Exercise 5. Explain why $\lambda(\sigma_i) = \sigma_i^2$ is the appropriate weighting in the NCSN objective.