Model Distillation

Dark Knowledge & Information in Soft Labels

Why soft labels carry exponentially more information than hard labels: the information-theoretic analysis, dark knowledge in incorrect class probabilities, inter-class similarity structure, and the implicit curriculum from teacher confidence.

Knowledge Distillation: Theory & Methods

Transferring knowledge from teacher to student: soft targets, temperature scaling, feature-based distillation, relation-based distillation, and the theory of why distillation works better than training from scratch.

Feature-Based & Relation-Based Distillation

Beyond logit matching: FitNets for intermediate layers, attention transfer, contrastive representation distillation (CRD), relational knowledge distillation (RKD), and when feature matching outperforms logit distillation.

Online & Mutual Distillation

Distillation without a pre-trained teacher: deep mutual learning, online knowledge distillation, born-again networks, co-distillation, and ensemble distillation — when and why these outperform standard teacher-student frameworks.

Data-Free & Zero-Shot Distillation

Distillation without access to original training data: generating synthetic data from the teacher, DeepInversion, contrastive data-free methods, and when zero-shot distillation is necessary (privacy, data deletion, proprietary models).

Task-Specific Distillation: Detection, Segmentation & NLP

Adapting distillation to specific tasks: distilling object detectors (feature pyramid matching), segmentation models (pixel-level KD), NLP models (TinyBERT, DistilBERT), and task-aware loss design.

Self-Distillation & Internal Knowledge Transfer

When the model teaches itself: self-distillation within a single network, deeper layers teaching shallower ones, auxiliary classifiers as teachers, label refinery, and why self-distillation improves even large models.

LLM Distillation & Synthetic Data

Distilling large language models: on-policy vs off-policy distillation, synthetic data generation, step-by-step distillation, constitutional AI as distillation, and open-source model distillation pipelines.

Speculative Decoding & Draft Model Distillation

Training efficient draft models for speculative decoding: distilling small models to match large model token distributions, acceptance rate optimization, MedUSA heads, and Eagle-style draft architectures.

Reward & Preference Distillation

Distilling alignment: transferring reward models, DPO as implicit distillation, RLHF teacher-student transfer, constitutional AI self-distillation, and maintaining alignment when compressing models.

Vision Model Distillation: CNN & ViT Compression

Distilling vision models: CNN-to-CNN, ViT-to-CNN, CNN-to-ViT knowledge transfer, attention map distillation, patch-level knowledge transfer, and achieving MobileNet-speed with ResNet-quality.

Distillation for Deployment: End-to-End Compression Pipelines

Production distillation: combining distillation with quantization and pruning, on-device model compression, latency-constrained architecture search with distillation, and building a complete model compression pipeline.