Supervised Fine-Tuning & Instruction Tuning

Learning Objectives
Notation
Core Intuition
Full Fine-Tuning
Instruction Formatting
Loss Masking
Data Quality vs Quantity
Parameter-Efficient Fine-Tuning (PEFT)
Chat Templates & Multi-Turn
Common Pitfalls
Summary
Exercises

Learning Objectives

Explain why SFT is needed after pre-training.
Derive the loss masking strategy for instruction tuning.
Compare full fine-tuning vs LoRA/QLoRA in terms of quality and cost.
Analyze the LIMA result: "Less Is More for Alignment."
Design effective instruction tuning datasets.

Notation

$\mathcal{D}_{\text{SFT}} = \{(x_i, y_i)\}$ — instruction-response pairs
$\mathcal{L}_{\text{SFT}}$ — supervised fine-tuning loss
$\mathbf{m}_t$ — loss mask (0 for prompt tokens, 1 for response tokens)

Core Intuition

Pre-trained LLMs are powerful but not directly useful — they complete text, not follow instructions. SFT teaches the model the "format" of helpful interaction: given an instruction, produce a well-structured response. Remarkably, this requires relatively few high-quality examples (1K-10K) because the model already has the knowledge from pre-training — it just needs to learn when and how to apply it.

Supervised Fine-Tuning

0.00

Explore: SFT minimizes cross-entropy on task demonstrations. High learning rates shift weights far from pretraining, risking catastrophic forgetting of general capabilities.

Full Fine-Tuning

Update all model parameters on instruction data:

\mathcal{L}_{\text{SFT}} = -\sum_{(x,y) \in \mathcal{D}}\sum_{t=1}^{|y|}\log p_\theta(y_t|x, y_{<t}). \tag{1}

Cost: Same as pre-training in terms of memory (full optimizer states). For a 70B model: ~280 GB for AdamW states + gradients.

Risk: Catastrophic forgetting if SFT data is too narrow or training is too long.

Instruction Formatting

Standard format (Alpaca-style):

Below is an instruction. Write a response.

### Instruction:
{instruction}

### Response:
{response}

Modern chat format (ChatML):

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{instruction}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Special tokens: Models learn to recognize the structure from these delimiters. The format itself carries information about the expected behavior.

Loss Masking

Key principle: Only compute loss on response tokens, not prompt tokens:

\mathcal{L} = -\sum_t m_t \cdot \log p_\theta(x_t|x_{<t}), \quad m_t = \begin{cases}0 & t \in \text{prompt} \\ 1 & t \in \text{response}\end{cases}. \tag{2}

Why: The prompt is given at inference time — we don't need the model to "generate" it. Training on prompt tokens wastes capacity on learning to reproduce instructions.

Exception: Some training pipelines include prompt loss with a lower weight (e.g., 0.1) to maintain general language modeling ability.

Data Quality vs Quantity

LIMA (Zhou et al., 2023): 1,000 carefully curated examples achieve alignment quality comparable to 50K lower-quality examples.

Key findings:

Quality matters more than quantity for SFT.
Diverse task coverage matters more than multiple examples per task.
Human-written responses outperform model-generated ones (for SFT stage).

Optimal SFT data strategy:

1K–10K high-quality, diverse instruction-response pairs.
Cover: QA, math, coding, creative writing, reasoning, safety.
Each example should be a gold-standard response.

Parameter-Efficient Fine-Tuning (PEFT)

LoRA: Add low-rank adapters $\Delta\mathbf{W} = \mathbf{BA}$ ( $\mathbf{B} \in \mathbb{R}^{d \times r}, \mathbf{A} \in \mathbb{R}^{r \times d}$ ). Train only $\mathbf{A}, \mathbf{B}$ .

QLoRA: LoRA on a 4-bit quantized base model. Enables fine-tuning 70B models on a single GPU.

Quality comparison:

Full FT slightly outperforms LoRA on narrow tasks.
LoRA is nearly equivalent for instruction tuning.
QLoRA loses $<1\%$ quality vs full LoRA with $8\times$ memory savings.

Chat Templates & Multi-Turn

Multi-turn format:

<|im_start|>user
First question<|im_end|>
<|im_start|>assistant
First response<|im_end|>
<|im_start|>user
Follow-up<|im_end|>
<|im_start|>assistant
Second response<|im_end|>

Loss masking for multi-turn: Only compute loss on assistant turns. User messages are treated as context (masked).

System prompts: Prepended instructions that define the model's persona and behavior rules. Trained by including system prompts in SFT data.

Common Pitfalls

Pitfall 1. Training too long on SFT data. Overfitting on small SFT datasets is rapid (often 1-3 epochs is optimal). Beyond that: memorization, loss of generalization.

Pitfall 2. Not masking prompt tokens. Including prompt in the loss teaches the model to generate instruction-like text, contaminating the response distribution.

Pitfall 3. Using only one type of instruction. Models fine-tuned only on QA will refuse creative writing. Diversity of task types is critical.

Summary

SFT teaches format and style, not knowledge (knowledge comes from pre-training).
Loss masking on response tokens only is crucial for quality.
Quality > quantity: 1K excellent examples can suffice (LIMA).
PEFT (LoRA/QLoRA) makes SFT practical for large models on limited hardware.
Multi-turn training with chat templates for conversational models.

Exercises

Exercise 1. For a 7B model with LoRA rank $r=16$ applied to all attention matrices ( $4d^2$ total): compute the percentage of trainable parameters.

Exercise 2. Design an SFT dataset of 1000 examples covering 10 task categories. Specify the distribution.

Exercise 3. Explain why training on the prompt tokens can hurt instruction-following ability.

Exercise 4. Compare the memory requirements for full fine-tuning vs QLoRA ( $r=64$ ) for a 70B model.

Exercise 5. Analyze the tradeoff between SFT data diversity and per-task depth: when is it better to have 100 tasks × 10 examples vs 10 tasks × 100 examples?