Curriculum Learning & Data Scheduling

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Classical Curriculum Learning
5. Data Mixing Schedules
6. Dynamic Data Selection
7. Anti-Curriculum and Self-Paced Learning
8. Practical Scheduling for LLMs
9. Common Pitfalls
10. Summary
11. Exercises

Learning Objectives

1. Explain why data ordering affects training outcomes.
2. Describe curriculum learning strategies (easy-to-hard, domain scheduling).
3. Analyze dynamic data selection methods (online batch selection).
4. Design a data mixing schedule for multi-domain pre-training.
5. Explain when anti-curriculum outperforms curriculum.

Notation

• $\mathcal{D} = \{\mathcal{D}_1, \ldots, \mathcal{D}_K\}$ — data domains
• $w_k(t)$ — mixing weight for domain $k$ at step $t$
• $\ell(\mathbf{x})$ — per-example loss

Core Intuition

Not all training data is equally useful at all stages of training. Early in training, simple examples help establish basic patterns; later, harder examples refine decision boundaries. Similarly, the ratio of code:math:text data should change over training to optimize final performance. Data scheduling determines WHAT the model sees WHEN.

Classical Curriculum Learning

Bengio et al. (2009): Present training examples in order of increasing difficulty.

Difficulty metrics:

• Loss-based: examples with lower loss (on a pre-trained model) are "easier."
• Length-based: shorter sequences first.
• Complexity-based: simpler linguistic structures first.

Effect: Faster convergence in early training. Sometimes better final performance. The model builds a "foundation" of simple patterns before tackling complex ones.

Formal: Define a scoring function $c(\mathbf{x})$ (complexity). At step $t$, train on $\{x : c(x) \leq \tau(t)\}$ where $\tau(t)$ increases over time.

Data Mixing Schedules

For multi-domain training (web, code, books, math):

Static mixing: Fixed proportions throughout training: $w = [0.6, 0.15, 0.15, 0.1]$ (web, code, books, math).

Dynamic mixing: Proportions change over training:

• Early: more web data (diverse, easy patterns).
• Late: more code/math (complex reasoning, structured).

DoReMi (Xie et al., 2023): Learn optimal mixing weights by training a small proxy model with different ratios and selecting weights that minimize worst-case domain loss.

Dynamic Data Selection

Online batch selection: At each step, score candidate batches and select the most useful one.

DSIR (Data Selection with Importance Resampling): Compute importance weights $p_{\text{target}}(x)/p_{\text{source}}(x)$ and upsample data similar to a high-quality target distribution.

DsDm (Data Selection via Data Models): Train a small model to predict which examples will most improve the large model's validation loss.

Influence functions: Estimate $\partial\mathcal{L}_{\text{val}}/\partial\mathbf{x}_i$ — how much each training example influences validation performance.

Anti-Curriculum and Self-Paced Learning

Anti-curriculum: Start with HARD examples first.

When it helps: When easy examples are redundant (model already "knows" them) and hard examples contain the most information.

Self-paced learning: Let the model choose its own curriculum based on current competence. At each step, focus on examples where the model is "almost right" (moderate loss) — not too easy, not too hard.

Practical Scheduling for LLMs

LLaMA-style: Static mixing with heavy web data (67% CommonCrawl).

GPT-4-style (speculated): Multi-phase training:

1. Phase 1: Broad pre-training on diverse web data.
2. Phase 2: Upweight high-quality data (books, code, curated).
3. Phase 3: Instruction tuning (SFT data).

Warmup-stable-decay for data quality:

• Warmup: slightly easier data (shorter, simpler).
• Stable: full data mixture.
• Decay: highest quality data only (for final quality boost).

Common Pitfalls

Pitfall 1. Aggressive curriculum (too easy for too long). The model overfits on simple patterns and then struggles to adapt when hard examples are introduced.

Pitfall 2. Changing data mixture without adjusting learning rate. A sudden shift in data distribution can destabilize training; pair data transitions with LR adjustments.

Pitfall 3. Assuming curriculum always helps. For well-shuffled large-scale pre-training, curriculum often provides minimal benefit over random ordering. It helps more for smaller datasets or specific capabilities.

Summary

• Curriculum learning: Easy-to-hard ordering for faster convergence.
• Data mixing: Static or dynamic proportions across domains.
• Dynamic selection: Choose the most useful batches based on model state.
• Anti-curriculum/self-paced: Sometimes hard-first or moderate-loss-first works better.
• For LLMs: multi-phase scheduling with quality ramp-up is standard practice.

Exercises

Exercise 1. Design a curriculum for training a code model: define difficulty metrics and the schedule.

Exercise 2. For a model training on 4 domains with equal size: derive the DoReMi objective for finding optimal mixing weights.

Exercise 3. Explain why random shuffling is a strong baseline for large-scale pre-training (hint: implicit curriculum from loss weighting).

Exercise 4. Compute the information gain from self-paced selection vs random selection for a model with bimodal loss distribution.

Exercise 5. Design a 3-phase training schedule for a 7B model targeting strong performance on both language and code tasks.