Dropout & Its Bayesian Interpretation

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Dropout: Formulation
5. Inverted Dropout
6. Dropout as Ensemble Averaging
7. Effect on Co-adaptation
8. MC Dropout: Bayesian Interpretation
9. Dropout in Different Architectures
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Define dropout mathematically as multiplication by a random binary mask.
2. Derive the expected output and show it equals the deterministic scaled output.
3. Explain dropout as an implicit ensemble of $2^n$ subnetworks.
4. Derive the MC Dropout uncertainty estimate.
5. Understand when and where to apply dropout (and when not to).

Notation

• $p$ — dropout probability (probability of zeroing a unit)
• $\mathbf{m} \sim \text{Bernoulli}(1-p)^d$ — binary mask vector
• $\mathbf{h} \in \mathbb{R}^d$ — hidden activations
• $\tilde{\mathbf{h}} = \mathbf{m} \odot \mathbf{h}$ — dropped activations

Core Intuition

During training, dropout randomly "turns off" each neuron with probability $p$. This forces the network to not rely on any single neuron — every neuron must be useful independently. At test time, all neurons are active but scaled, effectively averaging over all possible subnetworks. This acts as a powerful regularizer that reduces co-adaptation between neurons.

Dropout: Formulation

Training: For each forward pass, sample a mask $\mathbf{m} \in \{0,1\}^d$ with $m_i \overset{\text{iid}}{\sim} \text{Bernoulli}(1-p)$:

The output of the next layer: $\mathbf{z} = \mathbf{W}\tilde{\mathbf{h}} + \mathbf{b}$.

Test time (naive): Use the full network but scale: $\mathbb{E}[\tilde{\mathbf{h}}] = (1-p)\mathbf{h}$.

So at test time: multiply weights by $(1-p)$.

Inverted Dropout

In practice, scale during training instead (avoids modifying test-time code):

Now $\mathbb{E}[\tilde{\mathbf{h}}] = \mathbf{h}$ — the expected output matches the deterministic network. At test time, no modification is needed.

Dropout as Ensemble Averaging

A network with $d$ units and dropout creates $2^d$ possible subnetworks (one for each binary mask). Each forward pass trains one subnetwork. The test-time output (with scaling) approximates the geometric average of all $2^d$ subnetworks' predictions.

Theorem (Baldi & Sadowski, 2013). For linear networks, the dropout-averaged output exactly equals the scaled deterministic output. For nonlinear networks, it is an approximation.

Ensemble perspective:

This is intractable to compute directly but is approximated by the scaled deterministic forward pass.

Effect on Co-adaptation

Without dropout, neurons can "co-adapt" — one neuron compensates for another's mistakes. This creates fragile representations. With dropout:

• Each neuron must be independently useful (since its partners may be absent).
• Features become more robust and transferable.
• Effective model capacity is reduced (regularization).

Analogy: Training a team where random members are absent each day forces everyone to be versatile.

MC Dropout: Bayesian Interpretation

Theorem (Gal & Ghahramani, 2016). Dropout at test time approximates inference in a deep Gaussian process — a form of approximate Bayesian inference.

MC Dropout prediction: Run $T$ forward passes at test time with dropout active:

Uncertainty estimate:

where $\tau^{-1}$ is a model precision term related to weight decay.

Interpretation: The variance across dropout samples captures epistemic uncertainty — high variance indicates the model is unsure because different subnetworks disagree.

Dropout in Different Architectures

• Fully connected layers: Standard dropout with $p \in [0.2, 0.5]$.
• Convolutional layers: Spatial dropout — drop entire feature maps rather than individual pixels. Standard dropout on individual pixels is too aggressive (adjacent pixels are correlated).
• Recurrent layers: Apply to non-recurrent connections only (not hidden-to-hidden). Variational dropout: same mask across time steps.
• Transformers: Applied to attention weights and after FFN layers. Typical $p = 0.1$.

Common Pitfalls

Pitfall 1. Using dropout with batch normalization. BN estimates batch statistics; dropout changes the distribution. At test time, the BN running statistics are inconsistent with the non-dropped network. Use one or the other.

Pitfall 2. Applying dropout before the final layer without inverted scaling. The logits will be systematically lower at test time.

Pitfall 3. Setting $p$ too high. $p > 0.5$ means most of the network is dropped — underfitting. For large models, $p = 0.1$–$0.3$ is typical.

Summary

• Dropout randomly zeros activations during training: $\tilde{\mathbf{h}} = \mathbf{m} \odot \mathbf{h}/(1-p)$.
• Acts as an implicit ensemble of $2^d$ subnetworks.
• Prevents co-adaptation — neurons must be independently useful.
• MC Dropout at test time provides uncertainty estimates (approximate Bayesian inference).
• Apply dropout rates of 0.1–0.5 depending on model size and task.

Exercises

Exercise 1. Show that inverted dropout (equation 2) has $\mathbb{E}[\tilde{\mathbf{h}}] = \mathbf{h}$.

Exercise 2. For a linear network $y = \mathbf{w}^T\tilde{\mathbf{h}}$ with dropout, compute $\text{Var}(y)$ and show it provides an implicit L2 penalty on $\mathbf{w}$.

Exercise 3. Prove that the number of possible subnetworks with $d$ droppable units is $2^d$.

Exercise 4. Implement MC Dropout conceptually: for a classification problem, explain how to obtain predictive entropy from $T$ stochastic forward passes.

Exercise 5. Derive the equivalent weight decay strength $\lambda$ corresponding to dropout rate $p$ for a single linear layer (Gal & Ghahramani, 2016).