Pre-training Objectives & Data

Learning Objectives
Notation
Core Intuition
Next-Token Prediction (CLM)
Masked Language Modeling (MLM)
Span Corruption (T5)
UL2: Unified Objective
Data Curation
Compute-Optimal Training (Chinchilla)
Common Pitfalls
Summary
Exercises

Learning Objectives

Compare CLM, MLM, and span corruption in terms of compute efficiency and downstream performance.
Derive the Chinchilla scaling law for optimal model/data ratio.
Explain the role of data deduplication and quality filtering.
Analyze the "tokens per parameter" ratio for optimal training.
Describe the UL2 framework and its mode switching.

Notation

$N$ — model parameters
$D$ — training tokens
$C = 6ND$ — approximate training FLOPs
$L(N, D)$ — loss as function of scale

Core Intuition

The pre-training objective defines what the model learns from raw text. Next-token prediction (the dominant approach) is remarkably powerful: by predicting the next word in every context, the model implicitly learns grammar, facts, reasoning, and even code. But the data matters as much as the objective — garbage in, garbage out.

Pretraining Objectives

Explore: MLM masks random tokens and predicts them bidirectionally (BERT). CLM predicts the next token autoregressively with causal masking (GPT). CLM enables generation; MLM excels at understanding.

Next-Token Prediction (CLM)

\mathcal{L}_{\text{CLM}} = -\frac{1}{T}\sum_{t=1}^T \log p_\theta(x_t | x_1, \ldots, x_{t-1}). \tag{1}

Properties:

Every token is a training signal (no masking waste).
Natural for generation (model produces text left-to-right).
Implicitly learns: syntax, semantics, facts, reasoning, code, math.

Why it works so well: Predicting the next token in diverse text requires understanding the deep structure of language. "The capital of France is ___" requires factual knowledge. "If x > 0 then ___" requires logical reasoning.

Used in: GPT series, LLaMA, Mistral, Claude, all modern LLMs.

Masked Language Modeling (MLM)

\mathcal{L}_{\text{MLM}} = -\sum_{t \in \mathcal{M}} \log p_\theta(x_t | \mathbf{x}_{\backslash\mathcal{M}}), \tag{2}

where $\mathcal{M}$ is the set of masked positions (15% of tokens).

Compute efficiency issue: Only 15% of tokens provide training signal (the masked ones). CLM uses 100% of tokens.

Advantage: Bidirectional context — masked positions see both left and right context. Better for understanding tasks.

Used in: BERT, RoBERTa, DeBERTa. NOT used for modern LLMs (can't generate).

Span Corruption (T5)

Replace random spans with sentinel tokens; predict the original spans:

\text{Input: "The \langle X\rangle sat on \langle Y\rangle"} \to \text{Target: "\langle X\rangle cat \langle Y\rangle the mat"} \tag{3}

Properties:

Encoder-decoder architecture.
More efficient than MLM (shorter targets than full sequence).
Natural for seq2seq tasks (translation, summarization).

Used in: T5, mT5, Flan-T5.

UL2: Unified Objective

UL2 (Tay et al., 2022): Train with a mixture of objectives using mode tokens:

[R] Regular CLM (causal, left-to-right).
[S] Short span corruption (like T5, spans 2-5 tokens).
[X] Extreme span corruption (longer spans, more aggressive).

The model learns to switch behavior based on the mode token prefix.

Result: One model that's good at both generation (via [R] mode) and understanding (via [S]/[X] modes).

Data Curation

Modern pre-training datasets:

Web crawl (CommonCrawl, ~60%)
Books (5-10%)
Code (GitHub, ~10%)
Scientific papers (5%)
Wikipedia (5%)
Curated high-quality text (5-10%)

Key processing steps:

1. Deduplication: Remove near-duplicate documents (MinHash, suffix arrays). Duplicates cause memorization and reduce diversity.

2. Quality filtering: Classify web pages as high/low quality using a model trained on curated data. Remove: porn, spam, machine-generated text, boilerplate.

3. Toxicity filtering: Remove highly toxic content to reduce harmful outputs.

4. Domain mixing: Oversample high-quality domains (books, code, wikipedia) relative to their natural frequency.

Compute-Optimal Training (Chinchilla)

Chinchilla scaling law (Hoffmann et al., 2022):

L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E, \tag{4}

with $\alpha \approx 0.34, \beta \approx 0.28$ .

Optimal allocation: For fixed compute $C = 6ND$ :

N_{\text{opt}} \propto C^{0.5}, \quad D_{\text{opt}} \propto C^{0.5}. \tag{5}

Rule of thumb: Optimal training uses ~20 tokens per parameter ( $D \approx 20N$ ).

Implication: GPT-3 (175B params, 300B tokens) was undertrained. Chinchilla (70B params, 1.4T tokens) achieves same performance with less compute.

Post-Chinchilla practice: Most labs now overtrain (100-200 tokens/param) because inference cost depends on $N$ , not $D$ .

Common Pitfalls

Pitfall 1. Training on undeduplicated data. Even 1% duplication rates cause measurable memorization of duplicated content and waste compute.

Pitfall 2. Following Chinchilla exactly for production models. Chinchilla optimizes for training compute, but inference-optimal models are smaller and trained longer.

Pitfall 3. Ignoring data mixture ratios. The ratio of code:text:math dramatically affects downstream capabilities. Too much code hurts language; too little hurts reasoning.

Summary

CLM (next-token prediction) is the dominant objective: 100% token efficiency, natural for generation.
MLM provides bidirectional context but wastes 85% of tokens and can't generate.
Data quality matters as much as model size: dedup, filter, balance domains.
Chinchilla: Optimal is $D \approx 20N$ ; modern practice overtrains for inference efficiency.
UL2: Unified objective mixing CLM + span corruption.

Exercises

Exercise 1. Compute the training FLOPs for LLaMA-70B trained on 2T tokens using $C = 6ND$ .

Exercise 2. For a compute budget of $10^{24}$ FLOPs: derive the Chinchilla-optimal $N$ and $D$ .

Exercise 3. Explain why MLM with 15% masking is less compute-efficient than CLM (in terms of gradient signal per FLOP).

Exercise 4. Design a data mixture for a 7B model targeting strong code + math + language performance.

Exercise 5. Derive the "inference-optimal" training regime where we minimize total cost (training + serving $K$ queries) rather than just training cost.