Distillation for Deployment: End-to-End Compression Pipelines

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Complete Compression Pipeline
5. Distillation + Quantization
6. Distillation + Pruning
7. NAS-Guided Distillation
8. On-Device Deployment Constraints
9. Production Case Studies
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Design end-to-end compression pipelines combining distillation, quantization, and pruning.
2. Explain the optimal ordering of compression techniques.
3. Apply latency-constrained NAS with distillation-based training.
4. Handle deployment constraints (memory, latency, power).
5. Evaluate the compound effect of multiple compression techniques.

Notation

• $C$ — compression ratio (original size / compressed size)
• $\Delta$ — accuracy degradation from compression
• $L$ — latency constraint

Core Intuition

Real deployment rarely uses just one compression technique. Maximum compression comes from combining multiple methods: distill into a smaller architecture, prune redundant weights, then quantize to low precision. The key is ORDERING — each technique should be applied when it's most effective, and distillation acts as "error recovery" between aggressive compression steps.

The Complete Compression Pipeline

Optimal ordering for maximum compression with minimum quality loss:

1. Architecture distillation (reduce parameters 5-10x).
2. Structured pruning (remove redundant heads/layers, 1.5-2x).
3. Quantization-aware training (reduce precision, 2-4x).
4. Post-training quantization (further compression without training).

Total compression: $5 \times 1.5 \times 4 = 30$x with 2-5% quality loss.

Why this order works:

• Architecture change first (biggest impact, needs full retraining anyway).
• Pruning next (identifies redundancy in the distilled model).
• QAT (model learns to handle quantization before final freeze).
• PTQ last (cheap, final squeeze for deployment).

Distillation + Quantization

QAT-KD: Quantization-aware training WITH teacher guidance:

Why combined is better:

• QAT alone: model must figure out good quantized weights from hard labels.
• QAT + KD: teacher provides smooth targets that guide the quantized model to the right solution space.

Typical improvement: 0.5-1% accuracy over QAT alone at INT4.

Order matters:

• Distill THEN quantize: Better (model is already compact and well-structured).
• Quantize THEN distill: Worse (quantization errors compound before distillation can help).

Distillation + Pruning

Pruning-aware distillation: Train sparse student with teacher guidance:

Two approaches:

1. Prune then distill: Prune the teacher → distill the pruned teacher into same-architecture student → the student learns a DIFFERENT sparse structure.

2. Distill then prune: Distill into compact architecture → prune the student further → fine-tune with teacher guidance.

Approach 2 is generally better: Distillation first ensures the student has good weights; pruning then removes the least important ones.

NAS-Guided Distillation

Find the optimal student architecture automatically:

1. Define search space (channel widths, layer depths, kernel sizes).
2. Train a super-network with distillation from teacher.
3. Search for sub-networks meeting latency constraint:

Once-for-all (OFA) networks: Train a single large network that supports many sub-networks. Deploy different sub-networks based on device capability.

EfficientNet recipe:

1. NAS finds baseline architecture.
2. Compound scaling determines size variants (B0-B7).
3. Distillation from larger variant trains smaller variants.

On-Device Deployment Constraints

Mobile (phone, edge):

• Memory: 1-4 GB RAM available for model.
• Latency: 20-50ms per inference.
• Power: 2-5W budget.
• Solution: MobileNet-scale architecture + INT8 quantization + distillation from server model.

Browser (WebAssembly):

• Memory: Limited by tab.
• Compute: No GPU (or WebGPU).
• Solution: Tiny models (under 50MB) + INT8 + WASM SIMD.

Server (GPU):

• Memory: 24-80 GB VRAM.
• Latency: 10-100ms budget.
• Throughput: maximize tokens/sec.
• Solution: Moderate compression (INT4/INT8), optimized kernels.

Production Case Studies

Case 1: BERT → DistilBERT (Hugging Face):

• Teacher: BERT-base (110M, 12 layers).
• Student: DistilBERT (66M, 6 layers).
• Method: Triple loss (MLM + distillation + cosine embedding).
• Result: 97% quality, 60% smaller, 2x faster.

Case 2: GPT-4 → Phi-2 (Microsoft):

• Teacher: GPT-4 (estimated trillions of parameters).
• Student: Phi-2 (2.7B).
• Method: Synthetic data from GPT-4 + SFT.
• Result: Phi-2 outperforms many 7B+ models on reasoning tasks.

Case 3: Stable Diffusion → SDXL Turbo (Stability AI):

• Teacher: 50-step SDXL.
• Student: 1-4 step model.
• Method: Adversarial distillation + progressive distillation.
• Result: Real-time image generation.

Common Pitfalls

Pitfall 1. Compressing in one step (100x reduction at once). Gradual compression with distillation between steps yields much better quality. Each step should be 3-5x.

Pitfall 2. Optimizing for FLOPs instead of actual latency. Memory bandwidth, kernel efficiency, and hardware utilization matter more than theoretical FLOPs.

Pitfall 3. Not profiling on target hardware. A model that's fast on A100 may be slow on mobile due to operations not supported by the mobile accelerator.

Summary

• Optimal pipeline: Distill → prune → QAT → PTQ (each step recovers from previous).
• Distillation + quantization: Teacher guides QAT for better low-bit quality.
• NAS + distillation: Automatically find optimal student architecture.
• 30x compression achievable with 2-5% quality loss using combined techniques.
• Always profile on target hardware — theoretical compression doesn't equal actual speedup.
• Production examples: DistilBERT, Phi-2, SDXL Turbo.

Exercises

Exercise 1. Design a compression pipeline for deploying a 70B LLM on a smartphone (8GB RAM, 6 TOPS). Specify each step and expected size reduction.

Exercise 2. Compare the accuracy-latency Pareto frontier for: (a) distillation-only, (b) quantization-only, (c) distillation + quantization.

Exercise 3. For NAS-guided distillation: estimate the search cost (GPU-hours) for finding an optimal student from a space of $10^{12}$ architectures.

Exercise 4. Compute the total compression ratio for: ResNet-152 → MobileNet-V2 (architecture 5x) → 50% pruning (2x) → INT4 quantization (4x).

Exercise 5. Design an A/B test to validate that a compressed model meets production quality requirements. Specify metrics, sample size, and decision criteria.