Scaling Laws & Compute-Optimal Training

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Kaplan Scaling Laws (OpenAI)
5. Chinchilla Optimal Training
6. The Compute-Optimal Frontier
7. Over-Training for Inference
8. Predicting Performance
9. Emergent Abilities & Phase Transitions
10. Planning a Training Run
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. State the power-law scaling relationships for loss vs compute, data, and parameters.
2. Derive Chinchilla-optimal allocation between model size and data.
3. Explain why over-training (beyond Chinchilla) is optimal for inference-heavy deployment.
4. Use scaling laws to predict performance of larger models.
5. Plan a training run given a fixed compute budget.

Notation

• $N$ — number of parameters
• $D$ — number of training tokens
• $C$ — total compute (FLOPs) $\approx 6ND$
• $L$ — loss (cross-entropy)

Core Intuition

Language model performance follows surprisingly predictable power laws: double the compute → fixed reduction in loss. This means we can predict how a 100B model will perform by training 1B models. The key question: given a fixed compute budget, how should you split it between a larger model vs more data? Chinchilla showed the answer: scale both equally ($N \propto D$), contrary to the "make models as large as possible" approach.

Kaplan Scaling Laws (OpenAI)

Kaplan et al. (2020): Loss follows power laws in $N$, $D$, and $C$:

Key findings:

• Performance improves smoothly with scale (no plateaus).
• Parameters matter more than data (larger model with less data > smaller with more).
• This led to training VERY large models on relatively little data (GPT-3: 175B params, 300B tokens).

Chinchilla Optimal Training

Hoffmann et al. (2022, DeepMind): Kaplan's conclusion was WRONG. Both parameters and data should scale equally:

The Chinchilla rule: Optimal tokens $\approx$ 20x parameters.

Implication: GPT-3 (175B params, 300B tokens) was undertrained. The same compute could have been used for a 70B model trained on 1.4T tokens — achieving BETTER performance.

The Compute-Optimal Frontier

Parametric loss model:

where:

• $A/N^\alpha$: reducible loss from limited model capacity.
• $B/D^\beta$: reducible loss from limited data.
• $E$: irreducible loss (noise in data).

Optimal allocation (minimize $L$ given $C = 6ND$):

Chinchilla found $a \approx b \approx 0.5$ (equal scaling).

Over-Training for Inference

Post-Chinchilla insight (LLaMA, 2023): Chinchilla minimizes loss for fixed TRAINING compute. But at deployment, inference cost scales with $N$ (not $D$). A smaller model trained on MORE data has:

• Same training compute.
• Same or better loss.
• MUCH cheaper inference.

LLaMA strategy: Train a 7B model on 1T tokens (143x Chinchilla ratio instead of 20x).

Over-training multiplier $M$:

For $M = 7$ (LLaMA-7B at 1T tokens): Uses 7x more training compute than Chinchilla-optimal, but inference is 7x cheaper than the Chinchilla-optimal 50B model.

When to over-train: When inference cost dominates (serving millions of users) vs training cost (one-time).

Predicting Performance

Using scaling laws for planning:

1. Train small models (100M-1B) at different scales.
2. Fit power law: $L = a \cdot N^{-b} + c$.
3. Extrapolate to target scale.

Reliability: Predictions are accurate to within 5% for 10x extrapolation. Less reliable for 100x+ extrapolation (emergent abilities may not be captured).

Downstream task prediction: Harder. Some tasks show smooth scaling; others show sudden "emergence" at specific scales.

Emergent Abilities & Phase Transitions

Wei et al. (2022): Some capabilities appear suddenly at scale:

• Chain-of-thought reasoning: emerges around 100B.
• Multi-step arithmetic: emerges around 10B.
• Code generation: emerges around 1B.

Debate: Are emergent abilities real (phase transitions) or artifacts of evaluation metrics (smooth underlying improvement, discontinuous metric)?

Schaeffer et al. (2023): Many "emergent" abilities disappear when using continuous metrics instead of threshold-based ones. The underlying capability scales smoothly; it just crosses evaluation thresholds at certain scales.

Planning a Training Run

Given budget $C$ (in FLOPs):

1. Decide target: Chinchilla-optimal (minimize loss) vs over-trained (minimize inference cost).
2. Compute $N$ and $D$: $N = C / (6D)$ given chosen $D$.
3. Estimate loss: $L = A/N^\alpha + B/D^\beta + E$ from small-scale experiments.
4. Validate: Train a 1/10 scale model first; verify scaling prediction.
5. Budget hardware: Total GPU-hours = $C / (\text{GPU FLOPS} \times \text{MFU})$.

Example: Budget = $10^{24}$ FLOPs. Chinchilla: 16B model, 320B tokens. Over-trained ($M=4$): 7B model, 560B tokens. On 1024 H100s at 50% MFU: $10^{24} / (10^{15} \times 0.5 \times 1024) \approx 23$ days.

Common Pitfalls

Pitfall 1. Blindly following Chinchilla ratios without considering deployment. For a model serving millions of requests, the 7x inference cost savings from over-training easily pays for the extra training compute.

Pitfall 2. Extrapolating scaling laws beyond data quality limits. Scaling laws assume constant data quality. At 10T+ tokens, data quality degrades (more duplicates, noise) — actual performance falls below power-law prediction.

Pitfall 3. Ignoring the constant $E$ (irreducible loss). No amount of scaling reduces loss below $E$ (entropy of natural language). Estimated $E \approx 1.7$ nats for English text.

Summary

• Power-law scaling: Loss decreases predictably with compute, parameters, and data.
• Chinchilla rule: Optimal tokens $\approx$ 20x parameters for minimum training loss.
• Over-training: Train smaller models longer for cheaper inference (LLaMA approach).
• Prediction: Small-scale experiments reliably predict large-scale performance.
• Emergence: Some capabilities appear at specific scales (debate on mechanism).
• Scaling laws are the most important tool for planning billion-dollar training runs.

Exercises

Exercise 1. Given a budget of $10^{23}$ FLOPs: compute the Chinchilla-optimal model size and training tokens.

Exercise 2. LLaMA-2 70B was trained on 2T tokens. Is this over-trained or under-trained by Chinchilla standards? By how much?

Exercise 3. Fit a power law $L = a \cdot N^{-b} + c$ to these data points: (100M, 3.5), (1B, 2.8), (10B, 2.3). Predict loss at 100B.

Exercise 4. Compare total cost (training + 1-year inference) for: (a) Chinchilla-optimal 50B model serving 1M requests/day, vs (b) over-trained 7B model at same quality.

Exercise 5. Design an experiment to test whether chain-of-thought reasoning is truly emergent (discontinuous) or appears emergent due to evaluation metrics.