Scaling Laws for Language Models

Learning Objectives
Prerequisites
Notation
Core Intuition
Empirical Scaling Laws
Loss as a Function of Compute
Loss as a Function of Model Size and Data
The Chinchilla Optimal Allocation
Kaplan vs. Chinchilla Recommendations
Breaking Scaling Laws
Worked Examples
Connection to the Broader Curriculum
Common Pitfalls and Misconceptions
Research Perspective
Summary of Takeaways
Exercises

Learning Objectives

After reading this chapter, you should be able to:

State the power-law form of scaling: $L(N) \propto N^{-\alpha_N}$ , $L(D) \propto D^{-\alpha_D}$ .
Define training compute $C \approx 6ND$ FLOPs and the compute-optimal frontier.
Derive Chinchilla's optimal allocation: train smaller models on more tokens.
Contrast Kaplan et al. (2020) vs. Hoffmann et al. (2022) recommendations.
Explain implications for Tokenization and pretraining budget.

Prerequisites

Maximum Likelihood Estimation — cross-entropy loss
Tokenization — tokens as data unit

Notation

$N$ — Number of model parameters
$D$ — Training tokens (dataset size)
$C$ — Compute budget (FLOPs)
$L$ — Cross-entropy loss (nats per token)
$\alpha, \beta, \gamma$ — Empirical scaling exponents

Core Intuition

LLM performance improves predictably with scale — not linearly, but via power laws. Doubling compute yields predictable (diminishing) loss reduction. Scaling laws quantify these relationships, enabling budget allocation: given a compute budget, how large should the model be and how many tokens should it see?

The Chinchilla result (Hoffmann et al., 2022) overturned prior wisdom: for compute-optimal training, model size and data should scale equally — most models were undertrained.

Series context. Volume III, Chapter 12 (Pretraining).

Neural Scaling Laws

log C

21.00

ParamsData

Explore: Loss scales as a power law L ∝ C^(-α) with compute. Chinchilla found optimal allocation: scale params and data equally (~20 tokens/param).

Empirical Scaling Laws

Definition 1 (Cross-Entropy Loss). Pretraining minimizes:

L = -\frac{1}{N_{\text{tokens}}}\sum_i \log p_\theta(x_i \mid x_{<i}). \tag{1}

Definition 2 (Power Law). Empirically, loss scales as:

L(x) = A x^{-\alpha} + L_\infty, \tag{2}

where $x \in \{N, D, C\}$ (parameters, data tokens, compute), $\alpha > 0$ , $L_\infty$ is irreducible loss.

Loss as a Function of Compute

Definition 3 (Training Compute). Approximate FLOPs:

C \approx 6 N D, \tag{3}

where $N$ = non-embedding parameters, $D$ = training tokens. Factor 6 accounts for forward + backward pass.

Theorem 1 (Kaplan et al., 2020). For optimal allocation at fixed $C$ :

L(C) \propto C^{-\alpha_C}, \quad \alpha_C \approx 0.05. \tag{4}

Smooth log-linear improvement over orders of magnitude in compute.

Loss as a Function of Model Size and Data

Separate scaling:

L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \tag{5}

with $\alpha_N \approx 0.076$ , $\alpha_D \approx 0.095$ (Chinchilla estimates).

Combined (Hoffmann et al.):

L(N, D) = E + \frac{A}{N^{\alpha_N}} + \frac{B}{D^{\alpha_D}}, \tag{6}

where $E$ is irreducible entropy of natural language.

The Chinchilla Optimal Allocation

Problem. Minimize $L(N, D)$ subject to $C = kND$ (fixed compute).

Theorem 2 (Chinchilla Optimality). At compute-optimal training:

N_{\text{opt}} \propto C^{a}, \quad D_{\text{opt}} \propto C^{b}, \quad a \approx b \approx 0.5. \tag{7}

Corollary 1. Parameters and tokens should scale equally with compute: $D_{\text{opt}} \approx 20 N_{\text{opt}}$ (rough rule of thumb).

Proof sketch. Lagrange optimization on (6) with constraint $ND = C/k$ . Equal scaling exponents when both terms contribute equally at optimum. $\blacksquare$

Kaplan vs. Chinchilla Recommendations

Kaplan et al. — Scale $N$ faster than $D$ — Larger models, less data
Chinchilla — Scale $N$ and $D$ equally — Smaller models, ~4× more data

Example. GPT-3 (175B params, 300B tokens) was undertrained by Chinchilla standards. Optimal: ~70B params on ~1.4T tokens for same compute.

Breaking Scaling Laws

Proposition 1 (Limits). Scaling laws are empirical — may break due to:

Data exhaustion (finite high-quality text)
Architecture bottlenecks
Inference cost constraints
Diminishing returns on downstream tasks vs. pretraining loss

Emergent abilities (Wei et al.) may appear at scale — not captured by smooth loss curves alone.

Worked Examples

Example 1: Compute Budget

$C = 10^{24}$ FLOPs. Chinchilla: $N_{\text{opt}} \approx 10^{11}$ , $D_{\text{opt}} \approx 2 \times 10^{12}$ tokens.

Example 2: Power Law

If $L(N) = 1000 \cdot N^{-0.076}$ , doubling $N$ reduces loss by factor $2^{0.076} \approx 1.055$ (5.5%).

Connection to the Broader Curriculum

Tokenization — token counting for $D$
LoRA — efficient adaptation post-pretraining
DPO — alignment after scale
Linear Regression — log-log linear fits

Common Pitfalls and Misconceptions

Pitfall 1: Confusing pretraining loss with downstream task performance.

Pitfall 2: Using Kaplan scaling in post-Chinchilla era.

Pitfall 3: Ignoring inference cost when scaling $N$ .

Pitfall 4: $C = 6ND$ is approximate (depends on architecture).

Research Perspective

Kaplan et al. (2020). Chinchilla (Hoffmann et al., 2022). Llama 2/3 follow Chinchilla-optimal ratios. Open questions: data quality scaling, mixture-of-experts compute accounting.

Summary of Takeaways

Compute — $L \propto C^{-\alpha_C}$
Parameters — $L \propto N^{-\alpha_N}$
Data — $L \propto D^{-\alpha_D}$
Chinchilla — $N_{\text{opt}}, D_{\text{opt}} \propto C^{0.5}$

Next: DPO

Exercises

Exercise 1. Fit power law to given $(N, L)$ data points in log-log space.

Exercise 2. Derive Chinchilla optimum from (6) with Lagrange multipliers.

Exercise 3. For $C = 10^{23}$ , estimate $N_{\text{opt}}, D_{\text{opt}}$ .

Exercise 4. Compare GPT-3 training to Chinchilla recommendation.

Exercise 5. Why is $\alpha_D > \alpha_N$ significant?

Exercise 6. Relate scaling laws to Bias–Variance.

Exercise 7. MoE: how does effective $N$ differ from total parameters?

Exercise 8. When do scaling laws predict diminishing returns?