Change of Variables & Jacobian Determinants

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Change of Variables Formula
5. The Jacobian Matrix
6. Computing Determinants Efficiently
7. Triangular Jacobians
8. The Matrix Determinant Lemma
9. Hutchinson's Trace Estimator
10. Composition of Flows
11. When Exact Likelihood Is Tractable
12. Common Pitfalls
13. Summary
14. Exercises

Learning Objectives

1. Derive the multivariate change of variables formula from first principles.
2. Explain the geometric meaning of the Jacobian determinant (volume change).
3. Identify architectures that yield efficient (triangular, low-rank) Jacobians.
4. Apply the matrix determinant lemma to residual connections.
5. Use Hutchinson's estimator for stochastic log-det computation.

Notation

• $f: \mathbb{R}^d \to \mathbb{R}^d$ — diffeomorphism (smooth, invertible)
• $\mathbf{J}_f(\mathbf{x}) = \partial f / \partial\mathbf{x} \in \mathbb{R}^{d \times d}$ — Jacobian matrix
• $|\det\mathbf{J}_f|$ — absolute value of Jacobian determinant

Core Intuition

When we stretch, rotate, or warp space, probability density must change to keep total probability equal to 1. The Jacobian determinant measures exactly how much a transformation $f$ locally stretches or compresses volume. If $f$ doubles the volume at some point, the density must halve there. This fundamental principle connects invertible transformations to density estimation.

The Change of Variables Formula

One dimension: If $x = f(z)$ and $f$ is monotonic:

Multiple dimensions: If $\mathbf{x} = f(\mathbf{z})$ where $f$ is a diffeomorphism:

Equivalently, in the "forward" direction:

Proof sketch: Consider a small volume element $d\mathbf{z}$ around $\mathbf{z}$. Under $f$, it maps to a volume element $d\mathbf{x} = |\det\mathbf{J}_f| \cdot d\mathbf{z}$. Since probability is conserved: $p_X(\mathbf{x})d\mathbf{x} = p_Z(\mathbf{z})d\mathbf{z}$.

The Jacobian Matrix

For $f: \mathbb{R}^d \to \mathbb{R}^d$:

Geometric interpretation:

• $|\det\mathbf{J}| > 1$: local volume expansion.
• $|\det\mathbf{J}| < 1$: local volume contraction.
• $|\det\mathbf{J}| = 1$: volume-preserving (like rotations).
• $|\det\mathbf{J}| = 0$: degenerate (not invertible at this point).

Computational cost: Naively $O(d^3)$ to compute the determinant. For $d = 4096$ (typical hidden dimension), this is prohibitive. Flow architectures are designed to make this cheaper.

Computing Determinants Efficiently

Strategy 1: Triangular matrices. $\det(\mathbf{T}) = \prod_i T_{ii}$. Cost: $O(d)$.

Strategy 2: Block-triangular.

Strategy 3: Rank-1 updates (Matrix determinant lemma).

Strategy 4: Trace (for log-det of near-identity). If $\mathbf{J} = \mathbf{I} + \epsilon\mathbf{A}$:

Triangular Jacobians

Autoregressive transformations naturally yield triangular Jacobians:

The Jacobian is lower-triangular. Determinant = product of diagonal:

Examples: MAF, IAF, autoregressive neural spline flows.

Coupling layers (RealNVP/Glow): Split dimensions; Jacobian is block-triangular with identity blocks. Determinant = product of scale factors.

The Matrix Determinant Lemma

For residual connections $f(\mathbf{x}) = \mathbf{x} + g(\mathbf{x})$:

If $\mathbf{J}_g$ has low rank: $\mathbf{J}_g = \mathbf{UV}^T$ where $\mathbf{U}, \mathbf{V} \in \mathbb{R}^{d \times r}$:

Cost: $O(r^2 d + r^3)$ instead of $O(d^3)$. Practical when $r \ll d$.

Hutchinson's Trace Estimator

When $\mathbf{J}$ is not structured, use stochastic estimation:

For log-det via power series:

Cost: $O(Kd)$ per sample (each $\mathbf{J}_g\mathbf{v}$ is a JVP, computable via autodiff). Biased estimate unless $K \to \infty$.

Composition of Flows

For $f = f_K \circ \cdots \circ f_1$:

Log-likelihood:

Each layer contributes independently to the log-determinant. This is why flows are composed of many simple layers.

When Exact Likelihood Is Tractable

Exact likelihood requires:

1. Invertibility of $f$ (to compute $f^{-1}(\mathbf{x})$ for the base density term).
2. Efficient Jacobian determinant (triangular, low-rank, or estimable).

Always tractable: Coupling flows, autoregressive flows, linear flows. Approximately tractable: Residual flows (Hutchinson's estimator), continuous flows (trace estimator). Not tractable: GANs, standard VAE decoders, diffusion with stochastic sampling.

Common Pitfalls

Pitfall 1. Forgetting the absolute value: $|\det\mathbf{J}|$. Without it, negative determinants (orientation-reversing maps) give negative "probabilities."

Pitfall 2. Computing the full Jacobian matrix then taking determinant. This is $O(d^2)$ memory + $O(d^3)$ compute. Always exploit structure.

Pitfall 3. Confusing the forward and inverse Jacobians. $\det\mathbf{J}_{f^{-1}} = 1/\det\mathbf{J}_f$. Using the wrong one flips the sign of the log-likelihood.

Summary

• Change of variables: $p_X(\mathbf{x}) = p_Z(f^{-1}(\mathbf{x})) \cdot |\det\mathbf{J}_{f^{-1}}|$.
• Jacobian determinant measures local volume change.
• Efficient computation: triangular ($O(d)$), low-rank (MDL), stochastic (Hutchinson's).
• Composition: log-det sums over layers.
• Architecture design is driven by making the Jacobian tractable.

Exercises

Exercise 1. Derive the change of variables for the 2D rotation $f(\mathbf{x}) = \mathbf{R}_\theta\mathbf{x}$. Show that $|\det\mathbf{J}| = 1$.

Exercise 2. For a coupling layer with affine transform: compute the full Jacobian matrix and verify it's triangular.

Exercise 3. Use the matrix determinant lemma to compute $\det(\mathbf{I} + \mathbf{uv}^T)$ for $\mathbf{u} = [1, 0, 0]^T, \mathbf{v} = [2, 3, 1]^T$.

Exercise 4. Estimate $\text{tr}(\mathbf{A})$ using Hutchinson's estimator with 5 random vectors for a known $3 \times 3$ matrix. Compute the variance.

Exercise 5. Prove that the Jacobian determinant of the composition $f_2 \circ f_1$ equals $\det\mathbf{J}_{f_2} \cdot \det\mathbf{J}_{f_1}$.