LLM Distillation & Synthetic Data

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Distillation for LLMs
5. On-Policy vs Off-Policy
6. Step-by-Step Distillation
7. Synthetic Data Generation
8. Constitutional AI as Distillation
9. Practical Pipelines
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Distinguish on-policy and off-policy LLM distillation.
2. Explain step-by-step distillation (rationale + answer).
3. Describe synthetic data pipelines for LLM training.
4. Connect constitutional AI to self-distillation.
5. Design a practical distillation pipeline for a specific use case.

Notation

• $\pi_T$ — teacher model (large)
• $\pi_S$ — student model (small)
• $\mathcal{D}_{\text{synth}}$ — synthetic training data from teacher

Core Intuition

Distilling an LLM isn't just about matching output logits — it's about transferring the teacher's reasoning ability. The most effective approach: have the teacher generate reasoning traces (chain-of-thought) and high-quality responses, then fine-tune the student on this synthetic data. This is how most competitive open-source models are created.

Distillation for LLMs

Challenge: Token-level KL divergence is expensive (need teacher logits for every token position) and doesn't capture reasoning quality.

Practical approach: Sequence-level distillation — have the teacher generate complete responses, then SFT the student on them:

This is simply SFT on teacher-generated data.

On-Policy vs Off-Policy

Off-policy: Teacher generates responses; student trains on them regardless of student's own distribution.

On-policy: Student generates; teacher scores/corrects; student trains on corrected outputs.

On-policy advantages: Addresses distribution mismatch — student sees its own mistakes and learns to correct them. More expensive (requires iterative generation).

Step-by-Step Distillation

Idea (Hsieh et al., 2023): Distill not just the answer but the reasoning:

1. Teacher generates: rationale $r$ + answer $a$ for each question $q$.
2. Student trained on: $\log\pi_S(r, a | q)$.

Why it helps: The rationale provides "intermediate supervision" — the student learns HOW to reason, not just WHAT to answer. A 770M student with step-by-step distillation can outperform a 540B teacher with standard prompting.

Synthetic Data Generation

Pipeline:

1. Curate diverse seed prompts (cover many topics/styles).
2. Teacher generates responses with quality filtering:
   • Self-consistency: generate multiple times, keep consistent ones.
   • Verifier: use a separate model to score quality.
   • Format filtering: ensure responses meet format requirements.
3. Augment with instruction variations (rephrase prompts).
4. Train student on the filtered synthetic dataset.

Scale: Typical synthetic datasets: 100K–1M examples. More diverse prompts matter more than more examples per prompt.

Examples: Alpaca (52K from GPT-4), WizardLM (evolved instructions), Orca (progressive complexity).

Constitutional AI as Distillation

CAI (Anthropic): The model distills from itself via critique and revision:

1. Generate initial response.
2. Self-critique: "Does this response violate [principle]?"
3. Self-revise: "Rewrite to fix the violation."
4. Train on the revised responses.

This is self-distillation: The model's critique ability (from pre-training on ethical text) is transferred into its generation ability. The "teacher" is the model's own evaluation capacity.

Practical Pipelines

For a domain-specific chatbot (e.g., medical):

1. Collect 10K domain questions.
2. Generate responses with GPT-4 (teacher).
3. Filter: have domain experts score 10%, train quality classifier, filter rest.
4. SFT a 7B model on the filtered data.
5. Optionally: DPO with preference pairs (GPT-4 vs student responses).

For a code model:

1. Generate code solutions with teacher.
2. Execute and test — keep only passing solutions (verifiable).
3. Include test cases as additional training signal.
4. Student learns: problem → solution → tests.

Common Pitfalls

Pitfall 1. Distilling without quality filtering. Teacher models produce bad outputs too; training on them teaches the student bad habits.

Pitfall 2. Insufficient prompt diversity. Training on 1M responses to similar prompts is worse than 100K responses to diverse prompts.

Pitfall 3. Ignoring the capacity gap. A 1B student cannot replicate a 70B teacher's reasoning. Target tasks within the student's capacity, or use a larger student.

Summary

• LLM distillation = SFT on teacher-generated data (sequence-level).
• Step-by-step: Include reasoning traces for better student learning.
• On-policy distillation addresses distribution mismatch but is more expensive.
• Synthetic data pipelines: Seed prompts → teacher generation → quality filtering → student SFT.
• Constitutional AI is self-distillation: model's critique ability → generation ability.

Exercises

Exercise 1. Compare the training cost of token-level KD vs sequence-level distillation for a 70B teacher and 7B student processing 100K examples.

Exercise 2. Design a quality filtering pipeline for synthetic code data that doesn't require human annotation.

Exercise 3. Explain why a 7B model distilled from GPT-4 can outperform a 7B model trained from scratch on the same task, even though both see the same number of tokens.

Exercise 4. Derive the optimal number of teacher-generated responses per prompt to maximize student performance (considering diversity vs noise).

Exercise 5. Design an on-policy distillation loop with DPO: describe each iteration's data generation and training steps.