Self-Distillation & Internal Knowledge Transfer

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Be Your Own Teacher (BYOT)
5. Layer-Wise Self-Distillation
6. Label Refinery
7. Snapshot Distillation
8. PS-KD: Progressive Self-Knowledge Distillation
9. Why Self-Distillation Works
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Explain how a model can improve by teaching itself.
2. Describe BYOT's auxiliary classifier mechanism.
3. Derive label refinery's iterative self-improvement.
4. Connect self-distillation to implicit regularization.
5. Identify when self-distillation provides meaningful gains.

Notation

• $f_l$ — feature at layer $l$
• $C_l$ — auxiliary classifier at layer $l$
• $p_{\text{prev}}$ — predictions from previous epoch/checkpoint

Core Intuition

Self-distillation seems paradoxical: how can a model teach itself something it doesn't already know? The answer: knowledge is distributed unevenly. Deep layers have better features than shallow ones. Late-training predictions are better than early-training ones. By making the model's "better parts" teach its "worse parts," overall performance improves. It's like revising your own notes — you consolidate and improve understanding.

Be Your Own Teacher (BYOT)

Zhang et al. (2019): Attach auxiliary classifiers at intermediate layers:

Training objective: Each intermediate classifier is distilled from the final (deepest) classifier:

Effect on the backbone:

• Intermediate features are pushed to be more discriminative (they must support classification).
• The final classifier provides smooth labels to intermediate classifiers.
• Backbone features improve at ALL layers (not just the final one).

Deployment: Remove auxiliary classifiers. Only the backbone + final head is used. But the backbone is better because of the self-distillation training.

Layer-Wise Self-Distillation

Deeper layers teach shallower ones:

where $\text{sg}$ is stop-gradient (deeper layer is treated as the "teacher").

DINO/BYOL connection: Self-supervised methods like DINO use a similar principle — the momentum encoder (EMA of the student) acts as the teacher. This IS self-distillation in the temporal dimension.

Effect: Each layer tries to approximate the representation of the next layer, creating a hierarchical regularization that encourages smooth feature progression.

Label Refinery

Bagherinezhad et al. (2018): Iteratively improve labels using the model's own predictions:

1. Epoch 1: Train on hard labels → get predictions $p_1$.
2. Epoch 2: Train on soft labels $p_1$ (+ hard label) → get predictions $p_2$.
3. Epoch $k$: Train on $p_{k-1}$ → get $p_k$.

Why it helps: The model's own predictions are smoother than hard labels (implicit label smoothing). Each generation produces better-calibrated soft labels for the next.

Connection to born-again: Label refinery is born-again networks within a single training run (using checkpoints instead of retrained models).

Snapshot Distillation

Use historical checkpoints as teachers:

where $p_{t-\Delta}$ is the model's prediction $\Delta$ steps ago.

Exponential moving average (EMA) teacher:

The EMA model is smoother (averaged over many steps) and provides better-calibrated soft labels than the current model.

PS-KD: Progressive Self-Knowledge Distillation

Kim et al. (2021): Progressively increase self-distillation strength during training:

where $\alpha_t$ increases from 0 to $\alpha_{\max}$ over training.

Schedule: $\alpha_t = \alpha_{\max} \cdot (t / T)$ (linear increase).

Rationale:

• Early training: model predictions are poor → don't trust them ($\alpha \approx 0$).
• Late training: model predictions are good → use them as soft labels ($\alpha \approx \alpha_{\max}$).

Why Self-Distillation Works

Theoretical perspectives:

1. Regularization: Soft labels from the model prevent overconfidence on individual training examples.

2. Implicit ensemble: EMA teacher or historical predictions average over many optimization steps → smoother than any single checkpoint.

3. Dark knowledge from itself: The model's soft predictions on CORRECT examples encode learned similarities (which wrong classes are likely), providing richer gradient signal.

4. Feature compression: Intermediate distillation forces early layers to compress information efficiently (can't rely on later layers fixing bad representations).

5. Calibration: Self-distilled models are better calibrated (confidence matches accuracy).

Common Pitfalls

Pitfall 1. Self-distilling from the first epoch's predictions. Early predictions are near-random — using them as soft labels adds noise, not signal. Wait until the model has reasonable accuracy (after 20-30% of training).

Pitfall 2. Using $\alpha_{\text{self}} = 1.0$ (only self-distillation, no hard labels). Without grounding in hard labels, the model can drift (accumulate errors across self-distillation steps). Always maintain some hard label signal.

Pitfall 3. Self-distilling a model that's already overconfident. If the model outputs near-one-hot predictions, self-distillation provides minimal benefit (soft labels are already near-hard). Use temperature or earlier checkpoints.

Summary

• BYOT: Deeper layers teach shallower via auxiliary classifiers.
• Label refinery: Iteratively improve labels from own predictions.
• Snapshot distillation: EMA or historical checkpoints as teachers.
• PS-KD: Progressive increase of self-distillation strength.
• Why it works: Regularization, implicit ensemble, calibration improvement.
• Typical gain: 0.5-1.5% accuracy improvement with zero additional teacher cost.

Exercises

Exercise 1. For a 50-layer ResNet with BYOT classifiers at layers 16, 33, 50: estimate the additional training cost (FLOPs for auxiliary classifiers vs backbone).

Exercise 2. Implement label refinery for CIFAR-10 over 5 epochs. Track calibration (ECE) across iterations.

Exercise 3. Compare EMA teacher ($\beta=0.999$) vs snapshot teacher ($\Delta=1000$ steps) in terms of staleness and smoothness.

Exercise 4. Prove that self-distillation with temperature $T$ is equivalent to label smoothing with a specific smoothing parameter (derive the equivalence).

Exercise 5. Design a self-distillation strategy for GPT-2: what would the "auxiliary classifiers" be, and how would you apply layer-wise self-distillation to a causal language model?