Coupling Flows: NICE, RealNVP & Glow

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. NICE: Additive Coupling
5. RealNVP: Affine Coupling
6. Glow: 1x1 Convolutions
7. Multi-Scale Architecture
8. Permutations & Channel Mixing
9. Training & Architecture Details
10. Limitations of Coupling Flows
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Derive NICE's additive coupling and show its volume-preserving property.
2. Extend to RealNVP's affine coupling with learnable scaling.
3. Explain Glow's invertible 1x1 convolution and LU decomposition.
4. Design a multi-scale architecture with early factoring-out.
5. Analyze why coupling flows require many layers for expressiveness.

Notation

• $\mathbf{x} = [\mathbf{x}_1, \mathbf{x}_2]$ — split input
• $s(\cdot), t(\cdot)$ — scale and translation networks (arbitrary)
• $\mathbf{W} \in \mathbb{R}^{c \times c}$ — invertible 1x1 convolution

Core Intuition

Coupling flows solve the expressiveness-tractability dilemma by splitting input into two halves: one half passes through unchanged (providing a "conditioner"), while the other half is transformed based on the first. This gives a triangular Jacobian (cheap determinant) while still allowing arbitrarily complex transformations. Stack many such layers, alternating which half is transformed.

NICE: Additive Coupling

NICE (Dinh et al., 2014): The simplest coupling:

where $m$ is any neural network (no constraints).

Inverse (trivial):

Jacobian:

Volume-preserving: $\det = 1$ always. The transformation cannot expand or contract regions — only shift them. Limited expressiveness without learned scaling.

RealNVP: Affine Coupling

RealNVP (Dinh et al., 2017): Add learnable scaling:

Inverse:

Log-determinant:

Key improvement over NICE: The $\exp(s)$ scaling allows the model to expand/contract volumes — essential for modeling distributions with varying density.

Scale network $s$: Outputs log-scale factors. No constraint on output range; $\exp$ ensures positivity.

Translation network $t$: Shifts values. Combined with scaling: affine transformation of the second half, conditioned on the first.

Glow: 1x1 Convolutions

Problem: Coupling layers only transform half the dimensions per layer. Need to mix information between halves.

Glow (Kingma & Dhariwal, 2018): Replace fixed permutations with learned invertible 1x1 convolutions:

Log-determinant: $\log|\det\mathbf{J}| = h \cdot w \cdot \log|\det\mathbf{W}|$ for spatial dimensions $h \times w$.

LU decomposition for efficiency: Parameterize $\mathbf{W} = \mathbf{PLU}$:

• $\mathbf{P}$: fixed permutation matrix.
• $\mathbf{L}$: lower triangular with ones on diagonal.
• $\mathbf{U}$: upper triangular.

Multi-Scale Architecture

Key design pattern: Factor out half the dimensions at each scale level:

1. Level 1: Process at full resolution. After $K$ steps, split: $\mathbf{z}_1$ (factored out) + remaining.
2. Level 2: Squeeze (2x2 → channels). Process. Split: $\mathbf{z}_2$ + remaining.
3. Level L: Final level. All remaining dimensions become $\mathbf{z}_L$.

Final latent: $\mathbf{z} = [\mathbf{z}_1, \mathbf{z}_2, \ldots, \mathbf{z}_L]$.

Benefits:

• Early dimensions model coarse structure, later dimensions model fine details.
• Reduces computation (fewer dimensions at deeper levels).
• Improves training stability (gradients flow at multiple scales).

Each "step" consists of:

1. ActNorm (learned batch normalization).
2. Invertible 1x1 conv.
3. Affine coupling layer.

Permutations & Channel Mixing

Why permutations matter: A coupling layer leaves $\mathbf{x}_1$ unchanged. Without mixing, the same dimensions are always "frozen." After $K$ layers with the same split, half the dimensions have never been directly transformed.

Options (increasing quality):

1. Fixed reversal: Alternate which half is frozen. Simplest.
2. Fixed random permutation: Better mixing but still static.
3. Learned 1x1 conv (Glow): Optimal mixing, adds minimal overhead.

Training & Architecture Details

Base distribution: Isotropic Gaussian $\mathcal{N}(0, \mathbf{I})$.

Loss: Negative log-likelihood:

Coupling network architecture ($s, t$):

• For images: 3-layer CNN with 512 channels (residual blocks).
• Separate networks for $s$ and $t$, or shared with split output.
• NO constraint on $s, t$ (they don't need to be invertible).

Typical configuration (Glow on faces):

• 3 scale levels, $K=32$ steps per level.
• Total: 96 coupling layers + 96 1x1 convs + 96 ActNorms.

Limitations of Coupling Flows

1. Expressiveness per layer: Only transforms half the dimensions → needs many layers.
2. Topology: Cannot change the topology of the distribution (homeomorphism). Can't map disconnected regions to connected ones in finite layers.
3. Sample quality: Inferior to diffusion models and GANs on image generation (FID scores typically 2-3x higher).
4. Dimension mismatch: Input and output must have the same dimensionality (no compression).

Common Pitfalls

Pitfall 1. Using too few coupling layers. For CIFAR-10 (32x32x3 = 3072 dims): need at least 32 coupling layers for reasonable quality.

Pitfall 2. Not using ActNorm. Without proper normalization, the scale network $s$ can output extreme values early in training, causing numerical instability.

Pitfall 3. Forgetting to alternate the split pattern. Without alternation, half the dimensions are never directly transformed.

Summary

• NICE: Additive coupling; volume-preserving; limited expressiveness.
• RealNVP: Affine coupling; learned scaling; triangular Jacobian.
• Glow: LU-decomposed 1x1 conv for optimal channel mixing.
• Multi-scale: Factor out dimensions progressively; coarse-to-fine modeling.
• Training: Direct MLE; no adversarial loss, no reconstruction.
• Typical: 32-96 coupling layers for image generation.

Exercises

Exercise 1. Prove that two consecutive NICE layers with opposite splits can represent any invertible affine transformation.

Exercise 2. For a Glow model on 64x64x3 images with 4 levels and $K=16$ steps: count total parameters for coupling networks (assume 3-layer CNN with 512 channels each).

Exercise 3. Show that the multi-scale factored-out variables $\mathbf{z}_1$ at the first level correspond to coarse image features.

Exercise 4. Derive the memory cost of storing all intermediate activations for a Glow model with $L$ levels and $K$ steps (needed for gradient computation).

Exercise 5. Compare the bits-per-dimension (BPD) achievable by Glow vs a VAE on CIFAR-10. Explain why exact likelihood doesn't always mean better generation.