Residual Connections & Skip Connections

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Degradation Problem
5. The Residual Learning Framework
6. Gradient Flow Analysis
7. The Unraveled View
8. Dynamical Systems Interpretation
9. Dense Connections (DenseNet)
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Explain the degradation problem and why adding layers can increase training error.
2. Derive how skip connections create direct gradient pathways.
3. Prove that a ResNet with $L$ blocks can be viewed as an ensemble of $2^L$ paths.
4. Interpret residual blocks as Euler discretization of an ODE.
5. Analyze the effect of residual connections on the loss landscape.

Notation

• $\mathbf{h}^{(l)}$ — hidden state at layer $l$
• $F^{(l)}(\cdot)$ — residual function (nonlinear transformation at block $l$)
• $L$ — total number of residual blocks

Core Intuition

A plain network of depth $L$ requires gradients to traverse $L$ nonlinear layers. Each layer's Jacobian can shrink or amplify gradients. A residual connection adds the identity: $\mathbf{h}^{(l+1)} = \mathbf{h}^{(l)} + F^{(l)}(\mathbf{h}^{(l)})$. This creates a "gradient highway" — the identity path carries gradients directly, regardless of what happens in $F$.

The Degradation Problem

Observation (He et al., 2015). Training error of a 56-layer plain network is higher than a 20-layer network — not due to overfitting (training error, not just test error, degrades).

Why? The deeper network has strictly more representational capacity. It could learn the identity for the extra layers. But optimization fails: the gradient signal is too weak or chaotic to find good solutions when traversing many composed nonlinearities.

The Residual Learning Framework

Instead of learning $H(\mathbf{x})$ directly, learn the residual $F(\mathbf{x}) = H(\mathbf{x}) - \mathbf{x}$:

Key insight: If the optimal function is close to identity, learning a small $F \approx 0$ is easier than learning $H \approx \mathbf{I}$ from scratch. The skip connection provides the "default" identity mapping.

Gradient Flow Analysis

For a ResNet with $L$ blocks, the output is:

The gradient of the loss with respect to an intermediate layer:

Using the chain rule through residual blocks:

Expanding the product:

The leading $\mathbf{I}$ term ensures the gradient never vanishes — even if all $\partial F^{(k)}/\partial \mathbf{h}^{(k)} \to 0$, the gradient flows directly through the identity.

The Unraveled View

Theorem (Veit et al., 2016). A ResNet with $L$ blocks is equivalent to an ensemble of $2^L$ paths of different lengths.

Proof. Expanding (2) recursively:

Each subset $S$ defines a path that passes through blocks in $S$ and skips the rest. There are $2^L$ such paths. Most effective paths have length $\sim L/2$ (binomial distribution peaks).

Implication: ResNets exhibit graceful degradation — removing individual blocks has small effect because most paths remain intact.

Dynamical Systems Interpretation

The update $\mathbf{h}^{(l+1)} = \mathbf{h}^{(l)} + F^{(l)}(\mathbf{h}^{(l)})$ is an Euler discretization of the ODE:

with step size $\Delta t = 1$. This perspective leads to:

• Neural ODEs (Chen et al., 2018): use adaptive ODE solvers instead of fixed layers.
• Depth as continuous time: deeper networks = longer integration time.
• Stability analysis: if $\lVert\partial F/\partial\mathbf{h}\rVert < 1$, the system is contractive and stable.

Dense Connections (DenseNet)

DenseNet concatenates all previous features:

Every layer has direct access to all earlier features. This maximizes information flow but increases memory ($O(L^2d)$ features at the final layer).

Common Pitfalls

Pitfall 1. Expecting skip connections to help shallow networks. The benefit is primarily for deep networks ($L > 10$) where gradient degradation is significant.

Pitfall 2. Adding skip connections with dimension mismatch. When $\dim(\mathbf{h}^{(l)}) \neq \dim(F^{(l)})$, use a linear projection: $\mathbf{h}^{(l+1)} = \mathbf{W}_s\mathbf{h}^{(l)} + F^{(l)}(\mathbf{h}^{(l)})$.

Pitfall 3. Not initializing residual blocks near zero. For very deep networks, if $F^{(l)}$ outputs are large at initialization, the sum in (2) grows as $O(L)$ causing instability.

Summary

• Skip connections solve the degradation problem by providing identity shortcuts.
• The gradient contains a direct $\mathbf{I}$ term — it never vanishes regardless of depth.
• ResNets can be viewed as ensembles of $2^L$ paths — inherently robust.
• The ODE interpretation $d\mathbf{h}/dt = F(\mathbf{h})$ connects depth to continuous dynamics.
• Residual connections smooth the loss landscape, making optimization easier.

Exercises

Exercise 1. For a plain network $\mathbf{h}^{(L)} = f_L \circ \cdots \circ f_1(\mathbf{h}^{(0)})$, show that the gradient is $\prod_l \mathbf{J}_l$. How does this differ from equation (4)?

Exercise 2. Prove that a ResNet with $L$ blocks has $2^L$ unique paths from input to output.

Exercise 3. For the ODE $d\mathbf{h}/dt = -\mathbf{A}\mathbf{h}$ with positive definite $\mathbf{A}$, show the solution is contractive: $\lVert\mathbf{h}(t)\rVert \leq \lVert\mathbf{h}(0)\rVert$.

Exercise 4. Show that Pre-Norm ResNets have gradient norm $\lVert\partial\mathcal{L}/\partial\mathbf{h}^{(l)}\rVert \geq \lVert\partial\mathcal{L}/\partial\mathbf{h}^{(L)}\rVert$ for all $l$.

Exercise 5. Compute the memory cost of DenseNet vs ResNet with $L$ blocks and hidden dimension $d$.