Data-Free & Zero-Shot Distillation

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Why Data-Free Distillation
5. DeepInversion: Synthesizing Training Data
6. Generative Adversarial Data-Free Distillation
7. Contrastive Data-Free Methods
8. Meta-Learning for Few-Shot Distillation
9. Quality Comparison
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Identify scenarios requiring data-free distillation.
2. Derive DeepInversion's optimization objective for data synthesis.
3. Explain adversarial generator-based data-free methods.
4. Compare data-free vs data-dependent distillation quality.
5. Design a data-free distillation pipeline respecting privacy constraints.

Notation

• $\hat{x}$ — synthesized (generated) input
• $\mathcal{R}(\hat{x})$ — regularization on synthesized data
• $\text{BN}_l$ — batch normalization statistics at layer $l$

Core Intuition

What if you have a trained teacher but NO access to the training data? This happens with: (1) proprietary data (medical, financial), (2) data deletion requirements (GDPR right to be forgotten), (3) API-only access (GPT-4 — no weights, only outputs). Data-free distillation generates synthetic training data from the teacher itself, or uses the teacher's internal statistics to create informative training signals.

Why Data-Free Distillation

Scenario 1: Privacy compliance. Hospital trains model on patient data. Wants to deploy compressed model to edge devices. Cannot share patient data for student training. Must distill without original data.

Scenario 2: API distillation. Teacher is GPT-4 (only API access). Can query it but never see its weights or training data. Must create training data by querying.

Scenario 3: Data deletion. Original training data must be deleted (legal requirement). Teacher model remains. Need to create a smaller version without the deleted data.

Scenario 4: Continual learning. Old model was trained on data no longer available. New model must retain old knowledge without replaying old data.

DeepInversion: Synthesizing Training Data

Yin et al. (2020): Generate images that activate the teacher's internal statistics:

Batch Normalization regularization:

Forces synthetic data to match the running mean/variance stored in BN layers — ensuring it "looks like" training data to the teacher.

Image regularization: Total variation + $\ell_2$ norm to encourage natural-looking images.

Adaptive DeepInversion: Add a term to maximize student-teacher disagreement (generate samples that are HARD for the student):

Generative Adversarial Data-Free Distillation

Use a generator network to produce training data:

Generator $G$: Trained to produce inputs that maximize teacher-student disagreement. Student $S$: Trained to match teacher on generated inputs.

Adversarial dynamic:

• Generator creates hard examples → student struggles.
• Student improves on hard examples → generator must find harder ones.
• Convergence: student matches teacher on all reachable inputs.

Advantage over DeepInversion: Generator amortizes the optimization — doesn't need per-batch pixel-level optimization. Much faster data generation.

Contrastive Data-Free Methods

CMI (Fang et al., 2021): Generate diverse synthetic data via contrastive learning:

The second term enforces diversity: generated samples must be different from each other (minimum distance $\epsilon$).

Without diversity: Generator collapses to producing the same "hard" image repeatedly. Student overfits to one type of difficulty.

Meta-Learning for Few-Shot Distillation

When you have a TINY amount of data (10-100 examples):

MetaDistill: Use meta-learning to optimize how the student uses each precious example:

1. Inner loop: Update student on synthetic data from teacher.
2. Outer loop: Optimize the data generation to maximize student's performance on the few real examples.

Result: With just 1% of original data + data-free generation, achieves 90%+ of full-data distillation quality.

Quality Comparison

For ResNet-34 → ResNet-18 on CIFAR-100:

Key insight: Data-free distillation approaches (but doesn't match) full-data distillation. It still beats training the student from scratch without any teacher.

Common Pitfalls

Pitfall 1. Assuming data-free distillation equals full-data quality. There's typically a 2-5% gap. Use it only when data access is truly impossible.

Pitfall 2. Not enforcing diversity in generated samples. Without diversity regularization, generators collapse to a few prototype images.

Pitfall 3. Using DeepInversion without BN statistics. If the teacher doesn't have BatchNorm (e.g., transformers with LayerNorm), DeepInversion's primary regularization is unavailable. Use adversarial methods instead.

Summary

• Data-free distillation: When original training data is unavailable.
• DeepInversion: Optimize inputs to match teacher's BN statistics.
• Adversarial: Generator creates hard examples; student learns from them.
• Contrastive: Enforce diversity to prevent generator collapse.
• Quality gap: 2-5% below full-data distillation; still better than no distillation.
• Use cases: privacy, API-only access, data deletion compliance.

Exercises

Exercise 1. For a teacher with 10 BN layers: how many statistics are stored, and what do they represent about the training data?

Exercise 2. Design a data-free distillation pipeline for a medical imaging model where patient data has been deleted.

Exercise 3. Derive the optimal generator update that maximizes student-teacher disagreement (equation 4).

Exercise 4. Compare the wall-clock time of DeepInversion (500 iterations per batch) vs adversarial generation (1 forward pass per batch).

Exercise 5. Can data-free distillation be used to "steal" a proprietary API model? Discuss the ethical and legal implications.