Online & Mutual Distillation

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Deep Mutual Learning (DML)
5. Born-Again Networks (BAN)
6. Co-Distillation
7. Online Knowledge Distillation (ONE)
8. Ensemble Distillation
9. When Online Beats Offline
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Explain how two models can teach each other without a pre-trained teacher.
2. Derive the DML objective and its connection to mutual regularization.
3. Describe born-again networks and the surprising improvement across generations.
4. Analyze when online distillation outperforms standard offline distillation.
5. Design ensemble distillation pipelines for deployment.

Notation

• $\theta_1, \theta_2$ — parameters of two peer networks
• $p_1, p_2$ — their respective predictions
• Generation $k$ — the $k$-th iteration of born-again training

Core Intuition

Standard distillation requires a large pre-trained teacher — expensive to train and maintain. Online distillation eliminates this requirement: two (or more) models teach each other simultaneously during training. Surprisingly, two identical models training together outperform each individually, because each model's "noise" provides regularization for the other. It's like study buddies who learn better together than alone.

Deep Mutual Learning (DML)

Zhang et al. (2018): Train two networks simultaneously, each teaching the other:

Key insight: Even though neither model is better than the other, the KL term provides:

1. Regularization: Each model is pulled toward the other (prevents overconfidence).
2. Diversity: Different initializations → different mistakes → complementary knowledge.
3. Smoothing: The peer's soft predictions are a form of label smoothing.

Result: Both models outperform training alone by 1-2% on CIFAR/ImageNet.

Extension to $M$ peers: Each model is distilled from the average of all others:

Born-Again Networks (BAN)

Furlanello et al. (2018): Iteratively distill a model into itself:

1. Generation 0: Train model $\theta_0$ from scratch.
2. Generation 1: Train $\theta_1$ (same architecture) distilled from $\theta_0$.
3. Generation 2: Train $\theta_2$ distilled from $\theta_1$.
4. Continue...

Surprising result: $\theta_1 > \theta_0$, $\theta_2 > \theta_1$ (consistent improvement across generations).

Why it works:

• The teacher provides smoothed labels → implicit regularization.
• Different random initialization finds a different (potentially better) local minimum.
• The label smoothing effect prevents the new model from overfitting.

Saturation: Improvements diminish after 2-3 generations. Typical gain: 0.5-1% total.

Ensemble Born-Again: Average predictions from all generations: $p = \frac{1}{K}\sum_k p_{\theta_k}$. Better than any individual generation.

Co-Distillation

Anil et al. (2018, Google): Distill between models training on different shards of data:

1. Train $M$ models on different data shards simultaneously.
2. Periodically: each model generates soft labels on other models' shards.
3. Each model trains on: its own hard labels + others' soft labels.

Connection to distributed training: Co-distillation is an alternative to gradient aggregation in data-parallel training. Instead of averaging gradients, average soft predictions.

Advantages over AllReduce:

• Communication-efficient (send soft labels, not gradients).
• Asynchronous (no synchronization barrier).
• Heterogeneous (models can have different architectures/sizes).

Online Knowledge Distillation (ONE)

Zhu et al. (2018): Create a "multi-branch" network with shared backbone and $M$ prediction heads:

Training: Each branch is distilled from the ensemble:

Deployment: Remove branches; keep only the shared backbone + one head. The backbone has been improved by the ensemble training signal.

Advantage: Single-model training cost (shared backbone) with ensemble benefit.

Ensemble Distillation

Combine multiple models into a single deployable model:

Why ensemble teachers are powerful:

• Ensembles provide better-calibrated predictions (uncertainty).
• The diversity among ensemble members encodes model uncertainty.
• The student learns both the prediction AND the uncertainty.

Uncertainty distillation: Beyond average predictions, distill the VARIANCE of the ensemble:

When Online Beats Offline

Online (DML, co-distillation) is better when:

• No pre-trained teacher available (fresh training from scratch).
• Training budget allows only one training run.
• Models must be different architectures (heterogeneous DML).
• Data is distributed across shards (federated/distributed settings).

Offline (standard KD) is better when:

• A high-quality teacher already exists.
• Significant capacity gap (teacher much larger than student).
• The teacher provides information the student CANNOT discover on its own.

Common Pitfalls

Pitfall 1. Using DML with very different capacity models. If one model is much larger, it dominates and the smaller model just follows — losing the mutual benefit. Use similar-capacity peers.

Pitfall 2. Too many born-again generations. After 2-3 generations, the model converges to a fixed point. Additional generations waste compute with negligible improvement.

Pitfall 3. Not using temperature in online distillation. Even with peers (not a strong teacher), temperature $T=3$–$5$ on the KL term improves results by revealing inter-class structure.

Summary

• DML: Two peers teach each other via KL; both improve 1-2%.
• Born-Again: Self-distillation across generations; consistent improvement.
• Co-Distillation: Distill across data shards (communication-efficient distributed training).
• ONE: Multi-branch with shared backbone; ensemble signal improves single model.
• Ensemble distillation: Compress $M$ models into 1 model.
• Online methods eliminate the need for a pre-trained teacher.

Exercises

Exercise 1. For DML with 2 ResNet-32 models on CIFAR-100: estimate the additional training cost (FLOPs) vs single model training.

Exercise 2. Prove that the DML objective encourages the two models to agree (KL = 0 when identical) while the CE ensures they remain accurate.

Exercise 3. Train born-again networks for 4 generations on MNIST. Plot accuracy vs generation number. Does it converge?

Exercise 4. For co-distillation with 4 models on 4 data shards: compute the communication cost vs AllReduce for a 7B model.

Exercise 5. Design an experiment to determine whether born-again improvement comes from (a) implicit label smoothing, (b) different initialization, or (c) curriculum effect.