ML

Full Fine-tuning & Instruction Tuning

Adapting pre-trained models to downstream tasks: full parameter fine-tuning, instruction tuning (FLAN, Alpaca), supervised fine-tuning (SFT), learning rate schedules, catastrophic forgetting, and multi-task fine-tuning.

Intermediate

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. Full Fine-tuning
  5. Instruction Tuning
  6. SFT: Supervised Fine-Tuning for Chat
  7. Learning Rate & Schedule
  8. Catastrophic Forgetting
  9. Multi-Task Fine-Tuning
  10. Common Pitfalls
  11. Summary
  12. Exercises

Learning Objectives

  1. Design a full fine-tuning pipeline for task adaptation.
  2. Explain instruction tuning and the FLAN/T0 methodology.
  3. Derive optimal learning rate schedules for fine-tuning.
  4. Identify and mitigate catastrophic forgetting.
  5. Balance multi-task training for broad capability.

Notation

  • θ0\theta_0 — pre-trained parameters
  • ηft\eta_{\text{ft}} — fine-tuning learning rate
  • Dtask\mathcal{D}_{\text{task}} — downstream task dataset

Core Intuition

Pre-training learns general language understanding; fine-tuning specializes it for a specific use case. Full fine-tuning updates ALL parameters — maximum flexibility but risks forgetting pre-trained knowledge and overfitting to small datasets. Instruction tuning is a special form that teaches the model to follow instructions across many tasks, creating a general-purpose assistant.

Full Finetuning Dynamics

PretrainedNew task: 80%Old task: 75%Catastrophic forgetting: higher LR improves new task but destroys old task performance
LR
0.00010
PretrainedFine-tuned
Explore: Full finetuning moves all weights in loss landscape toward the new task. Higher learning rates cause catastrophic forgetting of pretrained capabilities.

Full Fine-tuning

Update all parameters on task-specific data:

θ∗=arg⁡min⁡θLtask(θ;Dtask),θ initialized at θ0.(1)\theta^* = \arg\min_\theta \mathcal{L}_{\text{task}}(\theta; \mathcal{D}_{\text{task}}), \quad \theta \text{ initialized at } \theta_0. \tag{1}

When to use full fine-tuning:

  • Sufficient data (above 10K examples for above 1B models).
  • Maximum quality needed (no PEFT compromise).
  • Single-task deployment (don't need to switch between tasks).

Typical hyperparameters:

  • Learning rate: 10−510^{-5} to 5×10−55 \times 10^{-5} (10-100x smaller than pre-training LR).
  • Epochs: 2-5 (more risks overfitting).
  • Batch size: 32-128 (smaller than pre-training).
  • Warmup: 3-10% of total steps.

Instruction Tuning

FLAN (Wei et al., 2022): Fine-tune on a mixture of tasks phrased as instructions:

Input: "Translate the following to French: Hello, how are you?" Output: "Bonjour, comment allez-vous?"

Key design decisions:

  1. Task diversity: 62+ datasets across NLU, NLG, reasoning, translation.
  2. Instruction templates: Multiple phrasings per task (10+ templates).
  3. Chain-of-thought: Include reasoning traces for complex tasks.
  4. Task balancing: Limit large datasets; upsample small ones.

FLAN-T5/PaLM result: Instruction-tuned models outperform base models on UNSEEN tasks — zero-shot generalization from following diverse instructions.

Scaling instruction data: More task diversity (not more examples per task) drives improvement. 100 tasks × 1K examples > 10 tasks × 10K examples.

SFT: Supervised Fine-Tuning for Chat

The alignment pipeline's first step:

LSFT=−∑t=1∣y∣log⁡pθ(yt∣x,y<t),(2)\mathcal{L}_{\text{SFT}} = -\sum_{t=1}^{|y|} \log p_\theta(y_t | x, y_{<t}), \tag{2}

where xx is the prompt and yy is the desired response. Loss computed ONLY on the response tokens (not the prompt).

Data format: (system_prompt, user_message, assistant_response) conversations.

Key practices:

  • Data quality over quantity: 10K high-quality conversations outperform 1M noisy ones.
  • Diverse tasks: coding, math, creative writing, analysis, roleplay.
  • Multi-turn: Include multi-turn conversations for dialogue coherence.
  • Format: Train with the exact chat template used at inference.

Learning Rate & Schedule

The fine-tuning LR principle: Start small, decay quickly.

Linear decay with warmup:

η(t)={ηmax⁡⋅t/twt<twηmax⁡⋅(1−(t−tw)/(T−tw))t≥tw(3)\eta(t) = \begin{cases}\eta_{\max} \cdot t/t_w & t < t_w \\ \eta_{\max} \cdot (1 - (t-t_w)/(T-t_w)) & t \geq t_w\end{cases} \tag{3}

Cosine decay:

η(t)=ηmin⁡+ηmax⁡−ηmin⁡2(1+cos⁡πtT).(4)\eta(t) = \eta_{\min} + \frac{\eta_{\max} - \eta_{\min}}{2}\left(1 + \cos\frac{\pi t}{T}\right). \tag{4}

Layer-wise learning rate decay: Lower layers (general features) get smaller LR; upper layers (task-specific) get larger:

ηl=ηmax⁡⋅ξL−l,ξ=0.8–0.95.(5)\eta_l = \eta_{\max} \cdot \xi^{L-l}, \quad \xi = 0.8\text{--}0.95. \tag{5}

Catastrophic Forgetting

Problem: Fine-tuning on task A degrades performance on task B (previously learned).

Causes:

  • Small fine-tuning dataset overrides general knowledge.
  • High learning rate pushes parameters far from pre-trained values.
  • Limited diversity in fine-tuning data.

Mitigations:

  1. Lower learning rate (stay close to pre-trained initialization).
  2. Regularization: Add L2 penalty toward pre-trained weights: λ∥θ−θ0∥2\lambda\|\theta - \theta_0\|^2.
  3. Data replay: Mix fine-tuning data with pre-training-style data.
  4. EWC: Penalize changes to important parameters (Fisher information weighted).
  5. LoRA/PEFT: Freeze most parameters; only update a small subspace.

Multi-Task Fine-Tuning

Train on multiple tasks simultaneously:

L=∑kwkLk(θ;Dk).(6)\mathcal{L} = \sum_k w_k \mathcal{L}_k(\theta; \mathcal{D}_k). \tag{6}

Task balancing strategies:

  • Proportional: wk∝∣Dk∣w_k \propto |\mathcal{D}_k| (large datasets dominate).
  • Equal: wk=1/Kw_k = 1/K (each task gets equal influence).
  • Temperature sampling: p(k)∝∣Dk∣1/Tp(k) \propto |\mathcal{D}_k|^{1/T} with T>1T > 1 upsampling small tasks.
  • Dynamic (GradNorm): Adjust wkw_k based on task gradient magnitudes.

When multi-task helps: Tasks are related and share useful representations. When it hurts: Tasks conflict (opposite objectives) or one task dominates.

Common Pitfalls

Pitfall 1. Using pre-training learning rate for fine-tuning. Pre-training LR (10−410^{-4}) is too high; fine-tuning should be 10−510^{-5} or lower to avoid destroying pre-trained features.

Pitfall 2. Computing loss on prompt tokens during SFT. The model already knows how to process prompts from pre-training; only response tokens need learning. Including prompt in loss dilutes the training signal.

Pitfall 3. Fine-tuning for too many epochs on small datasets. After 3-5 epochs on 10K examples, the model overfits (training loss drops while eval loss rises). Use early stopping.

Summary

  • Full fine-tuning: Update all parameters; maximum quality; risk of forgetting.
  • Instruction tuning: Diverse tasks as instructions; enables zero-shot generalization.
  • SFT: Response-only loss on conversations; data quality is critical.
  • Learning rate: 10-100x smaller than pre-training; cosine/linear decay.
  • Catastrophic forgetting: Mitigate with low LR, regularization, or PEFT.
  • Multi-task: Balance datasets carefully; temperature sampling helps.

Exercises

Exercise 1. Design an instruction tuning mixture for a 7B model targeting strong code + math + conversation abilities. Specify datasets, ratios, and templates.

Exercise 2. Compute the optimal fine-tuning learning rate for a model pre-trained with peak LR 3×10−43 \times 10^{-4}, using the "1/100" rule.

Exercise 3. For a model fine-tuned on QA: measure catastrophic forgetting by evaluating on translation before and after fine-tuning.

Exercise 4. Compare full fine-tuning vs LoRA for a 13B model on a 50K-example task: estimate quality gap, memory, and training time.

Exercise 5. Derive the GradNorm algorithm for dynamically balancing 3 tasks with different gradient magnitudes.

Next

Pre-training Objectives: CLM, MLM & Beyond