Normalizing Flows: Foundations

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Change of Variables Formula
5. Designing Invertible Transformations
6. Coupling Layers (RealNVP)
7. Autoregressive Flows (MAF/IAF)
8. Residual Flows
9. Training via Maximum Likelihood
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Derive the change of variables formula and the role of the Jacobian determinant.
2. Explain coupling layers and why their Jacobian is triangular.
3. Compare autoregressive flows (fast density, slow sampling) vs inverse autoregressive (fast sampling, slow density).
4. Derive the maximum likelihood training objective for normalizing flows.
5. Analyze the expressiveness-computation tradeoff.

Notation

• $\mathbf{z} \sim p_Z(\mathbf{z}) = \mathcal{N}(0, \mathbf{I})$ — base distribution
• $f_\theta: \mathbb{R}^d \to \mathbb{R}^d$ — invertible transformation
• $\mathbf{x} = f_\theta(\mathbf{z})$ — generated sample
• $\det\mathbf{J}_f$ — Jacobian determinant of $f$

Core Intuition

A normalizing flow transforms a simple distribution (Gaussian) into a complex one (data distribution) through a chain of invertible functions. Because each function is invertible, we can compute the exact likelihood of any data point — unlike VAEs (approximate) or GANs (no likelihood). The cost: invertibility constrains the architecture.

The Change of Variables Formula

If $\mathbf{x} = f(\mathbf{z})$ where $f$ is a diffeomorphism:

Log-likelihood:

Composition of flows: $f = f_K \circ f_{K-1} \circ \ldots \circ f_1$:

Designing Invertible Transformations

Requirements:

1. Must be invertible (bijective).
2. Jacobian determinant must be efficiently computable (not $O(d^3)$).
3. Must be expressive enough to model complex distributions.

Strategies for efficient Jacobian:

• Triangular Jacobian: $\det =$ product of diagonal → $O(d)$.
• Fixed-point iteration for inversion.
• Structured matrices (orthogonal, permutation).

Coupling Layers (RealNVP)

Split input $\mathbf{x} = [\mathbf{x}_1, \mathbf{x}_2]$:

where $s, t$ are arbitrary neural networks (no invertibility constraint on them).

Inverse (trivial):

Jacobian: Lower triangular → $\det = \prod_i \exp(s_i(\mathbf{x}_1)) = \exp(\sum_i s_i)$. Cost: $O(d)$.

Expressiveness: Alternate which dimensions are "frozen" ($\mathbf{x}_1$) vs "transformed" ($\mathbf{x}_2$) across layers.

Autoregressive Flows (MAF/IAF)

Masked Autoregressive Flow (MAF):

• Density evaluation: $O(1)$ passes (all $x_i$ known → parallel).
• Sampling: $O(d)$ sequential steps (must compute $x_1, x_2, \ldots$ in order).

Inverse Autoregressive Flow (IAF): Reverse the direction:

• Sampling: $O(1)$ passes (fast).
• Density: $O(d)$ sequential steps (slow).

Tradeoff: MAF for density estimation; IAF for fast generation.

Residual Flows

where $g_\theta$ is Lipschitz-constrained ($\|g\|_{\text{Lip}} < 1$) to ensure invertibility.

Jacobian: Use the matrix determinant lemma or Hutchinson's trace estimator:

Advantage: More expressive than coupling layers (transforms all dimensions simultaneously).

Training via Maximum Likelihood

No adversarial training, no reconstruction loss, no variational bound — exact maximum likelihood.

Common Pitfalls

Pitfall 1. Using too few coupling layers. Each layer only transforms half the dimensions; many layers needed for full expressiveness.

Pitfall 2. Forgetting the log-det-Jacobian term. Without it, the model ignores volume changes and produces poor density estimates.

Pitfall 3. Assuming flows scale to high dimensions easily. The invertibility constraint limits expressiveness; flows typically underperform diffusion models on image generation.

Summary

• Normalizing flows: exact likelihood via change of variables.
• Coupling layers: Split-transform-merge; triangular Jacobian, $O(d)$ cost.
• Autoregressive: Full expressiveness but sequential in one direction.
• Residual flows: Lipschitz-constrained for invertibility; trace estimation for Jacobian.
• Training: straightforward maximum likelihood.

Exercises

Exercise 1. Derive the Jacobian determinant for a coupling layer (equation 4).

Exercise 2. Prove that a composition of invertible functions is invertible with Jacobian = product of Jacobians.

Exercise 3. Show that MAF is equivalent to a Gaussian autoregressive model with learned mean and variance.

Exercise 4. For a residual flow with $\|g\|_{\text{Lip}} = 0.9$: bound the condition number of the Jacobian.

Exercise 5. Compare the sample quality (FID) achievable by normalizing flows vs diffusion models on CIFAR-10. Explain the gap.