Residual Flows & iResNet

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Invertible Residual Networks
5. Lipschitz Constraint for Invertibility
6. Log-Determinant via Power Series
7. Unbiased Estimation: Russian Roulette
8. Spectral Normalization
9. Comparison with Coupling Flows
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. State the sufficient condition for a residual network to be invertible.
2. Derive the power series expansion for the log-determinant of $\mathbf{I} + \mathbf{J}_g$.
3. Explain the Russian roulette estimator for unbiased truncation.
4. Apply spectral normalization to enforce the Lipschitz constraint.
5. Compare free-form (residual) vs structured (coupling/autoregressive) flows.

Notation

• $\text{Lip}(g)$ — Lipschitz constant of $g$
• $\sigma_1(\mathbf{W})$ — spectral norm (largest singular value) of $\mathbf{W}$
• $N \sim \text{Geometric}(p)$ — random truncation point

Core Intuition

Coupling and autoregressive flows constrain architecture severely (splits, masking, triangular structure). Residual flows take a different approach: use a standard ResNet $f(\mathbf{x}) = \mathbf{x} + g(\mathbf{x})$ and constrain $g$ to have Lipschitz constant less than 1. This guarantees invertibility via the Banach fixed-point theorem, while allowing ALL dimensions to interact simultaneously.

Invertible Residual Networks

i-ResNet (Behrmann et al., 2019): The residual mapping

is invertible if $\text{Lip}(g) < 1$.

Proof: $f$ is a contraction mapping perturbation of identity. By Banach fixed-point theorem, $f^{-1}$ exists and can be found iteratively:

Converges geometrically: $\|\mathbf{x}^{(t)} - \mathbf{x}^*\| \leq \text{Lip}(g)^t \cdot \|\mathbf{x}^{(0)} - \mathbf{x}^*\|$.

Lipschitz Constraint for Invertibility

Sufficient condition: $\text{Lip}(g) < 1$, i.e., $\|g(\mathbf{x}) - g(\mathbf{y})\| \leq c\|\mathbf{x} - \mathbf{y}\|$ for all $\mathbf{x}, \mathbf{y}$ with $c < 1$.

For a neural network $g$ with layers $g = g_L \circ \cdots \circ g_1$:

where $\phi_l$ is the activation function ($\text{Lip}(\text{ReLU}) = 1$, $\text{Lip}(\text{ELU}) \leq 1$).

Enforcement: Normalize each weight matrix so $\sigma_1(\mathbf{W}_l) \leq c^{1/L}$ where $c < 1$.

Log-Determinant via Power Series

The Jacobian is $\mathbf{J}_f = \mathbf{I} + \mathbf{J}_g$. Log-determinant:

Convergence: Guaranteed when $\text{Lip}(g) < 1$ (spectral radius of $\mathbf{J}_g$ is less than 1).

Stochastic estimation using Hutchinson:

Each $\mathbf{J}_g^k\mathbf{v}$ computed by $k$ JVPs (Jacobian-vector products via autodiff). Total cost: $O(K \cdot d)$ for $K$-term truncation.

Unbiased Estimation: Russian Roulette

Problem: Truncating the series at $K$ terms gives a BIASED estimate.

Solution: Russian roulette estimator. Draw $N \sim p(n)$ (e.g., geometric distribution), then:

Property: $\mathbb{E}[\widehat{\log\det}] = \log\det(\mathbf{I} + \mathbf{J}_g)$ (unbiased).

Variance: Depends on distribution $p(n)$. Geometric with $p=0.5$ gives reasonable variance with average $\sim 2$ terms evaluated.

Spectral Normalization

Method to enforce $\sigma_1(\mathbf{W}) \leq c$:

Power iteration to estimate $\sigma_1$:

One iteration per training step (amortized). Cost: two matrix-vector products.

Practical target: Set $c = 0.97$ per layer; with $L=10$ layers: $\text{Lip}(g) \leq 0.97^{10} \approx 0.74 < 1$.

Comparison with Coupling Flows

When to use residual flows: When modeling complex high-dimensional densities where coupling flows would need impractically many layers.

Common Pitfalls

Pitfall 1. Setting Lipschitz constant too close to 1 (e.g., 0.99). Inversion converges slowly and the power series needs many terms. Use $\text{Lip}(g) \leq 0.7$–$0.9$.

Pitfall 2. Using only 1 Hutchinson vector for trace estimation. The variance is high; use 5-10 vectors for stable training.

Pitfall 3. Forgetting that inversion is iterative. If you need fast exact inversion (e.g., for real-time generation), coupling flows are better despite lower expressiveness.

Summary

• Residual flows: $f(\mathbf{x}) = \mathbf{x} + g(\mathbf{x})$ with $\text{Lip}(g) < 1$.
• Invertibility via Banach fixed-point theorem; iterative inversion.
• Log-det via power series + Hutchinson's trace estimator.
• Russian roulette gives unbiased estimates without infinite series.
• Spectral normalization enforces Lipschitz constraint efficiently.
• More expressive per layer than coupling flows; less exact.

Exercises

Exercise 1. For $g(\mathbf{x}) = 0.5\mathbf{Ax}$ with $\|\mathbf{A}\| = 0.9$: compute $\text{Lip}(g)$ and verify invertibility.

Exercise 2. Compute the first 3 terms of the power series for $\log\det(\mathbf{I} + 0.5\mathbf{A})$ where $\mathbf{A} = [[0.2, 0.1], [0.3, 0.4]]$.

Exercise 3. How many fixed-point iterations are needed to invert $f(\mathbf{x}) = \mathbf{x} + 0.8g(\mathbf{x})$ to accuracy $10^{-6}$?

Exercise 4. Design a Russian roulette estimator with geometric($p=0.3$) distribution. Compute expected number of terms and prove unbiasedness.

Exercise 5. Compare the wall-clock time for training a 10-layer residual flow vs 30-layer Glow on CIFAR-10 (estimate FLOPs per training step).