Pre-training Objectives: CLM, MLM & Beyond

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Causal Language Modeling (CLM)
5. Masked Language Modeling (MLM)
6. Prefix Language Modeling
7. Denoising Objectives (T5/BART)
8. UL2: Unifying Language Learning
9. Contrastive Pre-training (CLIP)
10. Choosing the Right Objective
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Derive CLM and MLM loss functions and their information-theoretic interpretations.
2. Explain why CLM dominates for generative models while MLM excels at understanding.
3. Describe T5's span corruption and BART's denoising objectives.
4. Explain UL2's mixture-of-denoisers approach.
5. Choose the pre-training objective for a given downstream application.

Notation

• $x_{<t}$ — tokens before position $t$ (causal context)
• $x_{\backslash M}$ — tokens with masked positions excluded
• $\mathcal{M}$ — set of masked positions

Core Intuition

The pre-training objective defines WHAT the model learns from raw text. CLM (predict next token) teaches left-to-right generation — ideal for chatbots and code completion. MLM (predict masked tokens) teaches bidirectional understanding — ideal for classification and information extraction. The choice fundamentally shapes the model's capabilities, and modern approaches (UL2) combine multiple objectives.

Causal Language Modeling (CLM)

Objective: Predict the next token given all previous tokens:

Properties:

• Autoregressive: Each token depends only on previous tokens (causal mask).
• Exact likelihood: Provides exact log-probability of sequences.
• Generation: Natural fit for text generation (sample token by token).

Information-theoretic view: Minimizing CLM loss = maximizing the log-likelihood = minimizing the cross-entropy between model and true data distribution = minimizing KL divergence $D_{\text{KL}}(p_{\text{data}} \| p_\theta)$.

Models: GPT series, LLaMA, Mistral, Claude, Gemini.

Efficiency: Every token is a training signal (100% token utilization). For a sequence of length $T$: $T$ predictions per sequence.

Masked Language Modeling (MLM)

Objective (BERT): Predict randomly masked tokens from bidirectional context:

Masking strategy (BERT): 15% of tokens masked:

• 80%: Replace with [MASK].
• 10%: Replace with random token.
• 10%: Keep original (model must decide if token is correct).

Properties:

• Bidirectional: Each prediction uses FULL context (left AND right).
• Not generative: Cannot naturally generate text (masking is artificial).
• Understanding: Excels at classification, NER, QA — tasks needing comprehension.

Token utilization: Only 15% of tokens provide training signal (vs 100% for CLM). Need more data/steps for same effective training.

Models: BERT, RoBERTa, DeBERTa, ELECTRA.

Prefix Language Modeling

Hybrid: Bidirectional encoding of a prefix, autoregressive generation of the rest:

where $x_{\leq L}$ is encoded bidirectionally and $x_{>L}$ is generated autoregressively.

Attention mask:

• Prefix tokens ($t \leq L$): Full bidirectional attention.
• Generation tokens ($t > L$): Causal attention (can see prefix + previous generated).

Use case: Conditional generation — encode a context (question, document) bidirectionally, then generate response autoregressively.

Models: UniLM, GLM, PaLM (partially).

Denoising Objectives (T5/BART)

T5 Span Corruption: Replace random spans with sentinel tokens; predict the spans:

Input: "The <X> sat on the <Y>." Target: "<X> cat <Y> mat </s>"

BART Denoising: Multiple corruption strategies + full reconstruction:

• Token masking (like MLM).
• Token deletion (model figures out what's missing).
• Sentence permutation.
• Document rotation.
• Span masking.

Encoder-decoder architecture: Encoder processes corrupted input bidirectionally; decoder generates clean output autoregressively.

Advantage: Combines bidirectional understanding (encoder) with generation ability (decoder). Good at both comprehension and generation tasks.

UL2: Unifying Language Learning

Tay et al. (2022): Mix multiple denoising objectives with different "modes":

R-Denoiser (Regular): Short spans, 15% corruption. → Understanding tasks. S-Denoiser (Sequential): Causal prefix → generation. → Generation tasks. X-Denoiser (Extreme): Long spans, 50% corruption. → Long-range reasoning.

Mode tokens: Prepend [R], [S], or [X] to indicate which denoiser is being used. Model learns to adapt behavior based on mode.

Mixture: Randomly select mode per example: $p(R)=0.5, p(S)=0.25, p(X)=0.25$.

Result: Single model that excels at BOTH understanding and generation — matching specialized models on their respective strengths.

Contrastive Pre-training (CLIP)

For multimodal models: Learn aligned representations across modalities:

Properties:

• No text generation (contrastive only).
• Learns shared embedding space for images and text.
• Enables zero-shot classification and retrieval.

Choosing the Right Objective

Common Pitfalls

Pitfall 1. Using MLM for a generative task. BERT-style models cannot naturally generate text — they can fill blanks but not write continuations.

Pitfall 2. Training CLM with too short context. CLM benefits enormously from long context during pre-training. Truncating to 512 tokens wastes potential.

Pitfall 3. Assuming more pre-training data always helps. With a fixed compute budget, there's an optimal data-to-parameters ratio (Chinchilla scaling). Too much data with too small a model hits diminishing returns.

Summary

• CLM: Next-token prediction; autoregressive; best for generation (GPT, LLaMA).
• MLM: Masked prediction; bidirectional; best for understanding (BERT).
• Prefix LM: Bidirectional prefix + autoregressive generation; good for conditional tasks.
• T5/BART: Encoder-decoder denoising; good at both understanding and generation.
• UL2: Mixes all modes; single model excels everywhere.
• The pre-training objective is the single most important design decision.

Exercises

Exercise 1. Compute the bits-per-character of a CLM model with perplexity 15 on English text.

Exercise 2. For MLM with 15% masking: how many training tokens are needed to match CLM's effective training signal on 1T tokens?

Exercise 3. Design a pre-training objective for a code model that needs both code completion (left-to-right) and code understanding (bidirectional).

Exercise 4. Prove that minimizing CLM loss is equivalent to maximizing the ELBO on the data log-likelihood.

Exercise 5. Compare UL2 vs separate CLM + MLM models: compute the parameter efficiency (performance per parameter) for each approach.