Continual Learning & Catastrophic Forgetting

Learning Objectives
Notation
Core Intuition
The Catastrophic Forgetting Problem
Elastic Weight Consolidation (EWC)
Experience Replay
Progressive Neural Networks
Continual Pre-training for LLMs
Model Merging as Continual Learning
Common Pitfalls
Summary
Exercises

Learning Objectives

Formalize catastrophic forgetting and measure its severity.
Derive EWC from a Bayesian perspective (Fisher information as importance).
Design experience replay buffers for continual learning.
Apply continual pre-training to update LLMs with new knowledge.
Use model merging techniques to combine task-specific adaptations.

Notation

$\mathcal{T}_1, \mathcal{T}_2, \ldots$ — sequence of tasks
$\mathbf{F}_k$ — Fisher information matrix after task $k$
$\theta_k^*$ — optimal parameters for task $k$

Core Intuition

Neural networks learn by adjusting all parameters — but this means learning task B overwrites the parameters that encoded task A. This "catastrophic forgetting" is the fundamental challenge of continual learning: how to accumulate knowledge over time without losing what was previously learned. Solutions range from protecting important parameters (EWC) to replaying old data (experience replay) to growing the network (progressive nets).

Continual Learning (EWC)

EWC λ

0.50

Task ATask BTask C

Explore: Continual learning faces catastrophic forgetting as new tasks overwrite old knowledge. EWC (Elastic Weight Consolidation) regularizes important weights to preserve prior task performance.

The Catastrophic Forgetting Problem

Formal definition: After training on tasks $\mathcal{T}_1, \ldots, \mathcal{T}_K$ sequentially:

\text{Forgetting}_k = \text{Perf}(\theta_k^*; \mathcal{T}_k) - \text{Perf}(\theta_K^*; \mathcal{T}_k), \quad k < K. \tag{1}

Why it happens in neural networks:

Shared representation: all tasks use the same parameters.
No explicit memory: unlike the brain, no separate episodic memory.
Gradient interference: gradients for new task point away from old task optimum.

Stability-plasticity dilemma:

Too stable → can't learn new tasks.
Too plastic → forgets old tasks.
Must balance both.

Elastic Weight Consolidation (EWC)

Kirkpatrick et al. (2017): Protect important parameters with a quadratic penalty:

\mathcal{L}_{\text{EWC}} = \mathcal{L}_{\text{new}}(\theta) + \frac{\lambda}{2}\sum_i F_i(\theta_i - \theta_i^*)^2, \tag{2}

where $F_i$ is the Fisher information of parameter $i$ (measuring its importance for old tasks).

Fisher information:

F_i = \mathbb{E}_{x \sim \mathcal{D}_{\text{old}}}\left[\left(\frac{\partial\log p(x|\theta)}{\partial\theta_i}\right)^2\right]. \tag{3}

Interpretation: Parameters with high Fisher are "important" (small changes cause large loss increase). EWC penalizes changes to these parameters heavily; unimportant parameters can change freely.

Bayesian view: EWC approximates the posterior $p(\theta | \mathcal{T}_1)$ as a Gaussian centered at $\theta_1^*$ with precision $F$ . Training on $\mathcal{T}_2$ with this prior prevents forgetting $\mathcal{T}_1$ .

Experience Replay

Simple but effective: Store a small buffer of old task data; mix with new task during training:

\mathcal{L} = \mathcal{L}_{\text{new}}(\theta; \mathcal{D}_{\text{new}}) + \alpha\mathcal{L}_{\text{replay}}(\theta; \mathcal{B}), \tag{4}

where $\mathcal{B}$ is a replay buffer of stored examples.

Buffer strategies:

Random: Store random subset of old data.
Herding: Store representative examples (closest to class mean).
Gradient-based: Store examples with high gradient magnitude.

Buffer size tradeoff: Larger buffer = less forgetting, more memory. Typical: 1-5% of old data.

For LLMs: Replay a fraction of pre-training data during fine-tuning (5-10% of each batch).

Progressive Neural Networks

Rusu et al. (2016): Don't overwrite — grow:

Train a model for task 1 (freeze it).
Add a new column (model) for task 2, with lateral connections from task 1.
Add another column for task 3, with connections from tasks 1 and 2.

Properties:

Zero forgetting (old parameters are frozen).
Positive forward transfer (new tasks can use old features via lateral connections).
Growing cost (one model per task — impractical for many tasks).

Continual Pre-training for LLMs

Problem: LLM knowledge becomes stale (trained on data up to a cutoff date). How to update without full retraining?

Approaches:

1. Continued pre-training: Resume pre-training on new data:

Use low learning rate (1/10 of original).
Mix new data with old data replay (5-20%).
Risk: slight degradation on old benchmarks.

2. Domain adaptation: Focus on new domain while preserving general ability:

Train on domain-specific corpus (medical, legal, code).
Replay general data to prevent catastrophic forgetting.

3. Knowledge editing: Modify specific facts without retraining:

ROME/MEMIT: Direct parameter editing for fact updates.
Limited to simple fact changes; doesn't generalize.

Model Merging as Continual Learning

Instead of sequential training: Train separate models for each task, then merge:

Linear merging (TIES, DARE):

\theta_{\text{merged}} = \theta_{\text{base}} + \sum_k \alpha_k(\theta_k - \theta_{\text{base}}). \tag{5}

Task arithmetic: Task vectors $\tau_k = \theta_k - \theta_{\text{base}}$ can be added:

\theta_{\text{multi}} = \theta_{\text{base}} + \sum_k \tau_k. \tag{6}

Advantage: No sequential training needed. Train tasks in parallel, merge at the end.

Limitation: Works best when task vectors are approximately orthogonal (non-conflicting).

Common Pitfalls

Pitfall 1. Using high learning rate for continual pre-training. This causes immediate forgetting. Use LR that's 5-10x lower than original pre-training peak.

Pitfall 2. EWC without sufficient Fisher estimation. Computing Fisher on too few examples gives noisy importance estimates. Use 1000+ samples.

Pitfall 3. Merging models trained with very different data distributions. If tasks are conflicting (e.g., "be verbose" vs "be concise"), linear merging produces a poor compromise.

Summary

Catastrophic forgetting: Learning new tasks degrades old task performance.
EWC: Quadratic penalty weighted by Fisher information (parameter importance).
Experience replay: Mix old data with new data during training.
Progressive nets: Grow the model; zero forgetting but linear cost.
Continual pre-training: Low LR + data replay to update LLM knowledge.
Model merging: Train separately, merge after — avoids sequential forgetting entirely.

Exercises

Exercise 1. Compute the Fisher information for a 2-class logistic regression model with weights $w = [1, -1]$ .

Exercise 2. For experience replay with 1% buffer: estimate the forgetting rate after 10 sequential tasks on CIFAR-100.

Exercise 3. Design a continual pre-training schedule to update LLaMA-2 (trained through 2023) with 2024 data. Specify LR, data mix, and evaluation strategy.

Exercise 4. Prove that linear model merging is optimal when task vectors are orthogonal (in parameter space).

Exercise 5. Compare the compute cost of: (a) retraining from scratch, (b) continual pre-training, (c) model merging for adding a new language to a multilingual LLM.