Emergent Abilities & In-Context Learning

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. What Are Emergent Abilities?
5. In-Context Learning (ICL)
6. Chain-of-Thought Reasoning
7. Phase Transitions in Scaling
8. Theoretical Explanations
9. The Measurement Artifact Debate
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Define emergent abilities and distinguish from continuous improvements.
2. Formalize in-context learning as implicit Bayesian inference.
3. Explain how chain-of-thought extends effective computation.
4. Analyze the "mirage" hypothesis of emergence.
5. Connect emergence to transformer circuit complexity.

Notation

• $N$ — number of model parameters
• $C$ — compute budget (FLOPs)
• $\text{ICL}(k)$ — performance with $k$ in-context examples
• $p_\theta(y|x, \mathcal{D}_k)$ — model prediction given $k$ demonstrations

Core Intuition

Some abilities of language models appear to "switch on" suddenly at a certain scale — models below a threshold show near-random performance, while models above it succeed reliably. This is in-context learning (learning from examples in the prompt without gradient updates), chain-of-thought reasoning (solving multi-step problems by generating intermediate steps), and instruction following. Whether this represents genuine phase transitions or measurement artifacts is debated.

What Are Emergent Abilities?

Definition (Wei et al., 2022): An ability is emergent if it is "not present in smaller models but is present in larger models."

Examples:

• 3-digit addition: GPT-3 (175B) succeeds; GPT-3 (13B) fails.
• Multi-step arithmetic: Requires CoT prompting + sufficient scale.
• Code generation: Requires >50B parameters for non-trivial programs.

Characteristics of emergence:

• Unpredictable from smaller-scale trends.
• Sharp transition (not gradual improvement).
• Task-specific (some tasks emerge earlier than others).

In-Context Learning (ICL)

The model learns from examples in the prompt without weight updates:

Bayesian interpretation (Xie et al., 2022): ICL performs implicit Bayesian inference over a latent concept:

The demonstrations help the model identify the "concept" $c$ (e.g., sentiment analysis, translation), then apply it.

Transformer mechanism: Induction heads detect patterns like "A follows B" in the context and generalize to new instances. This requires at least 2-layer attention circuits.

Chain-of-Thought Reasoning

Key insight: Autoregressive transformers have limited computation per token (fixed depth). By generating intermediate reasoning steps, they effectively increase computation:

Theoretical result (Feng et al., 2023): Constant-depth transformers cannot solve certain problems (like composition of functions) directly, but CAN solve them with chain-of-thought (using intermediate tokens as scratch space).

Emergence: CoT only helps above a certain model scale ($\sim$100B params). Below this, models generate incoherent intermediate steps.

Phase Transitions in Scaling

Observation: Plotting accuracy vs log(compute), some tasks show:

• Flat at random (below threshold).
• Sharp jump to high accuracy (at threshold).
• Continued improvement (above threshold).

Proposed mechanisms:

1. Circuit formation: The model needs sufficient capacity to form specialized circuits (e.g., induction heads for ICL). Below capacity: circuits can't form.
2. Representation threshold: Certain concepts require minimum embedding dimension to be distinctly represented.
3. Memorization to generalization: Below threshold, model memorizes; above, it generalizes.

Theoretical Explanations

Induction head theory (Olsson et al., 2022):

• Two-layer attention circuits that implement ICL.
• Head 1: copies information from previous token.
• Head 2: uses copied info to predict the next token in a pattern.
• Formation requires sufficient model width and training data.

Grokking connection: Models sometimes memorize first, then generalize after more training — the "phase transition" may be a training dynamics phenomenon, not a capacity one.

The Measurement Artifact Debate

Schaeffer et al. (2023): Emergence may be a "mirage" caused by:

1. Nonlinear metrics: Accuracy is discontinuous (0 or 1 per example). Smooth metrics (log-likelihood, Brier score) often show continuous improvement.
2. Task difficulty thresholding: If a task requires composing $k$ skills, and each skill improves continuously, the conjunction appears to "emerge" sharply.

Counter-argument: Even with smooth metrics, some abilities (like ICL on novel task formats) genuinely require minimum scale for the relevant circuits to form.

Current consensus: Both views have merit. Some emergence is metric-dependent; some reflects genuine computational phase transitions.

Common Pitfalls

Pitfall 1. Assuming emergence means "unpredictable." While the task transition is sharp, the underlying capabilities (perplexity, calibration) improve smoothly. Emergence is often predictable with the right metric.

Pitfall 2. Attributing emergence solely to scale. Training data quality, diversity, and duration also determine when abilities emerge. A 7B model trained on diverse code data can outperform a 70B model trained without code.

Pitfall 3. Expecting all tasks to benefit from CoT. Simple lookup tasks (factual recall) don't benefit from CoT — it adds tokens without useful computation.

Summary

• Emergent abilities appear sharply at certain scales (ICL, CoT, instruction following).
• In-context learning as implicit Bayesian inference over latent concepts.
• Chain-of-thought extends effective computation by using intermediate tokens.
• Debate: Some emergence is real (circuit formation); some is metric artifact.
• Scale alone doesn't guarantee emergence — data quality and training matter.

Exercises

Exercise 1. Formalize in-context learning as Bayesian inference: derive $p(c|\mathcal{D}_k)$ for $k$ demonstrations of a classification task.

Exercise 2. Explain why accuracy shows sharp transitions while log-probability shows smooth improvement.

Exercise 3. Design an experiment to distinguish "real" emergence from a metric artifact for a multi-step reasoning task.

Exercise 4. Compute the effective computational depth of a 32-layer transformer generating a 10-step chain-of-thought.

Exercise 5. Describe the minimal 2-layer attention circuit that implements the copying pattern needed for ICL.