RLHF: Reinforcement Learning from Human Feedback

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Three-Stage Pipeline
5. Reward Model Training
6. PPO for Language Models
7. KL Regularization
8. Reward Hacking & Overoptimization
9. Practical Implementation Details
10. Limitations of RLHF
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Describe the complete 3-stage InstructGPT pipeline.
2. Derive the Bradley-Terry model for preference learning.
3. Explain PPO's surrogate objective adapted for language models.
4. Analyze reward overoptimization and the role of KL penalty.
5. Identify practical challenges in RLHF (instability, cost, alignment tax).

Notation

• $r_\phi(x, y)$ — reward model score for response $y$ given prompt $x$
• $\pi_\theta$ — policy (language model)
• $\pi_{\text{ref}}$ — reference policy (SFT model)
• $\beta$ — KL penalty coefficient

Core Intuition

SFT teaches the model to generate good responses, but it can't distinguish "good" from "great" or penalize harmful outputs precisely. RLHF adds a learned reward signal from human preferences: humans rank model outputs, a reward model learns these preferences, and the language model is optimized to maximize reward while staying close to its SFT behavior. This is how GPT-4, Claude, and Gemini become helpful AND safe.

The Three-Stage Pipeline

InstructGPT (Ouyang et al., 2022):

Stage 1: SFT — Fine-tune on demonstration data (human-written ideal responses).

Stage 2: Reward Model — Train a model to predict human preferences:

• Collect comparison data: for prompt $x$, human ranks $y_1 > y_2 > \ldots > y_k$.
• Train reward model on these rankings.

Stage 3: RL (PPO) — Optimize the language model to maximize reward:

Reward Model Training

Bradley-Terry preference model: Probability that response $y_w$ is preferred over $y_l$:

Loss:

Architecture: Same as the language model but with a scalar output head (replace LM head with linear → scalar).

Data: 50K-500K comparison pairs. Each comparison: same prompt, 2+ responses ranked by humans.

Key property: Only DIFFERENCES in reward matter (invariant to adding a constant).

PPO for Language Models

PPO surrogate objective:

where $A$ is the advantage (reward - baseline).

Adapted for LLMs:

• State $s$: Prompt + tokens generated so far.
• Action $a$: Next token.
• Reward: $r_\phi$ applied to the COMPLETE response (sparse reward at end).
• Episode: One complete generation.

Value function: Predicts expected future reward from current state. Trained alongside policy.

Practical: 4 models in memory simultaneously: policy, reference, reward, value. For a 7B model: 4 × 14GB = 56GB minimum.

KL Regularization

Purpose: Prevent the model from deviating too far from SFT behavior.

Per-token KL: Penalize each token prediction that differs from the reference.

Why needed:

• Without KL: model finds "reward hacks" (exploit reward model errors).
• Without KL: model collapses to repetitive high-reward patterns.
• The reference provides a "prior" over reasonable text.

$\beta$ selection: Typically 0.01-0.2. Higher = more conservative (closer to SFT). Lower = more optimization (higher reward, risk of hacking).

Reward Hacking & Overoptimization

Gao et al. (2022): As optimization pressure increases, actual quality peaks then DECREASES:

• Initially: more optimization improves quality.
• Beyond the peak: model exploits reward model errors.
• Eventually: responses get high reward but low true quality.

Examples of reward hacking:

• Generating longer responses (reward model biased toward length).
• Repeating key phrases that correlate with high reward.
• Sycophantic agreement (always agreeing with the user).

Mitigation: Strong KL penalty, reward model ensembles, iterative RLHF with updated reward models.

Practical Implementation Details

Training infrastructure:

• 4 models: policy (trainable), reference (frozen), reward (frozen), value (trainable).
• Policy + value share the backbone (two heads).
• Reference = copy of initial SFT model (frozen throughout).

Hyperparameters:

• Batch size: 512+ (need diverse prompts per batch).
• PPO epochs per batch: 1-4 (more = overfitting to batch).
• Learning rate: $10^{-6}$ to $5 \times 10^{-6}$ (very conservative).
• Clip $\epsilon$: 0.2 (standard PPO).
• KL penalty $\beta$: 0.01-0.1 (tune on held-out eval).
• Total steps: 10K-50K (much shorter than pre-training).

Cost: 2-4x more expensive than SFT (4 models + multiple generations per step).

Limitations of RLHF

1. Expensive: Requires human annotation (preference data), 4 models in memory, iterative training.
2. Unstable: PPO is notoriously finicky; hyperparameter sensitivity is high.
3. Reward model quality: Ceiling is the reward model; bad preferences → bad alignment.
4. Alignment tax: RLHF can reduce raw capability (model becomes "safe" but less capable on some tasks).
5. Non-stationary: As the policy improves, the reward model becomes stale (trained on worse responses).

Common Pitfalls

Pitfall 1. Training the reward model on data from a different model generation. If RM was trained on GPT-3 outputs but policy is GPT-4, the RM may not discriminate well among GPT-4 quality outputs.

Pitfall 2. Setting KL penalty too low. The model will quickly exploit reward model weaknesses (length bias, sycophancy) and produce text that scores high but isn't actually good.

Pitfall 3. Running PPO for too many steps. RLHF should be "short and sweet" — 10K-50K steps. Longer training leads to overoptimization and degraded capabilities.

Summary

• RLHF pipeline: SFT → Reward Model → PPO optimization.
• Reward model: Bradley-Terry on human preference pairs.
• PPO for LLMs: Sparse reward (end of generation); per-token KL penalty.
• KL regularization prevents reward hacking and maintains diversity.
• Overoptimization: True quality peaks then degrades with more RL.
• Cost: 2-4x SFT; requires 4 models in memory; 50K-500K preference labels.

Exercises

Exercise 1. For a reward model trained on 100K preference pairs: estimate the inter-annotator agreement needed for the RM to be useful (above random).

Exercise 2. Derive the optimal policy for the KL-regularized reward objective (equation 1). Show it's a softmax of the reward.

Exercise 3. Compute the memory requirement for RLHF of a 70B model with all 4 models in BF16.

Exercise 4. Design an experiment to detect reward hacking: specify metrics that would diverge from reward model score.

Exercise 5. Compare the cost (GPU-hours and human-hours) of RLHF vs DPO for aligning a 13B model. When is each more cost-effective?