ML

RLHF: Reinforcement Learning from Human Feedback

The full RLHF pipeline: reward modeling from comparisons, PPO optimization, KL-constrained policy updates, the Bradley-Terry model, and InstructGPT methodology.

Advanced

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. The Alignment Problem
  5. Stage 1: Supervised Fine-Tuning (SFT)
  6. Stage 2: Reward Modeling
  7. Stage 3: PPO Optimization
  8. The KL Penalty
  9. Connection to DPO
  10. Common Pitfalls
  11. Summary
  12. Exercises

Learning Objectives

  1. Describe the three-stage RLHF pipeline (SFT → RM → PPO).
  2. Derive the Bradley-Terry model for preference learning.
  3. Explain the PPO objective with KL constraint.
  4. Prove the connection between reward maximization and DPO.
  5. Analyze failure modes: reward hacking, mode collapse.

Notation

  • πθ\pi_\theta — policy (language model)
  • πref\pi_{\text{ref}} — reference policy (SFT model)
  • rϕ(x,y)r_\phi(x, y) — reward model
  • β\beta — KL penalty coefficient

Core Intuition

Pre-trained LLMs predict likely text, not helpful text. A model trained on the internet will happily generate toxic, incorrect, or harmful content if it's statistically likely. RLHF aligns the model with human preferences: first teach it what "good" outputs look like (reward model), then optimize it to produce those outputs (RL), while staying close to the pre-trained model (KL penalty) to preserve capabilities.

RLHF Pipeline

SFTReward ModelPPO LoopResponse score: 0.55 · Δpolicy: 0.30max E[r] − β·KL(π || π_ref)
Stage
1
KL β
0.10
Explore: RLHF: SFT on demonstrations → train reward model on comparisons → PPO fine-tunes policy against reward while KL penalty β keeps it close to the reference model.

The Alignment Problem

Pre-training objective: Maximize p(xt+1∣x1:t)p(x_{t+1}|x_{1:t}) — predict likely text.

Desired behavior: Produce helpful, harmless, honest responses.

Gap: Likely ≠ helpful. The most probable completion of "How do I hack..." might be instructions for hacking, not a refusal.

Solution: RLHF adds a layer of preference optimization on top of the pre-trained model.

Stage 1: Supervised Fine-Tuning (SFT)

Fine-tune the pre-trained model on high-quality demonstrations:

LSFT=−E(x,y)∼Ddemo[log⁡πθ(y∣x)].(1)\mathcal{L}_{\text{SFT}} = -\mathbb{E}_{(x,y) \sim \mathcal{D}_{\text{demo}}}\left[\log\pi_\theta(y|x)\right]. \tag{1}

Data: Human-written (prompt, response) pairs demonstrating ideal behavior.

Purpose: Shift the model's distribution toward helpful response format. Creates the reference policy πref\pi_{\text{ref}} and starting point for RL.

Stage 2: Reward Modeling

Data collection: For each prompt xx, generate two responses y1,y2y_1, y_2 and have humans label which is better: yw≻yly_w \succ y_l.

Bradley-Terry model: The probability of preferring ywy_w over yly_l:

p(yw≻yl∣x)=σ(rϕ(x,yw)−rϕ(x,yl)),(2)p(y_w \succ y_l | x) = \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)), \tag{2}

where σ\sigma is the sigmoid function.

Training objective:

LRM=−E(x,yw,yl)[log⁡σ(rϕ(x,yw)−rϕ(x,yl))].(3)\mathcal{L}_{\text{RM}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log\sigma(r_\phi(x, y_w) - r_\phi(x, y_l))\right]. \tag{3}

Architecture: Same as the language model but with the last token's hidden state projected to a scalar reward. Typically initialized from the SFT model.

Stage 3: PPO Optimization

Objective: Maximize reward while staying close to the reference policy:

max⁡πθEx∼D,y∼πθ(⋅∣x)[rϕ(x,y)]−β DKL(πθ∥πref).(4)\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)}\left[r_\phi(x, y)\right] - \beta\, D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}). \tag{4}

PPO implementation:

  1. Generate responses y∼πθ(⋅∣x)y \sim \pi_\theta(\cdot|x) for a batch of prompts.
  2. Score with reward model: R=rϕ(x,y)−βlog⁡πθ(y∣x)πref(y∣x)R = r_\phi(x, y) - \beta\log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}.
  3. Compute advantages using a value function.
  4. Update policy with clipped PPO objective:
LPPO=−E[min⁡(πθ(y∣x)πold(y∣x)A,clip(πθπold,1±ϵ)A)].(5)\mathcal{L}_{\text{PPO}} = -\mathbb{E}\left[\min\left(\frac{\pi_\theta(y|x)}{\pi_{\text{old}}(y|x)}A, \text{clip}\left(\frac{\pi_\theta}{\pi_{\text{old}}}, 1\pm\epsilon\right)A\right)\right]. \tag{5}

The KL Penalty

Purpose: Prevent the policy from diverging too far from the pre-trained model.

Without KL penalty: The model "reward hacks" — finds degenerate outputs that exploit reward model weaknesses (e.g., repeating a praised phrase endlessly).

Effect: The KL term βDKL(πθ∥πref)\beta D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}) penalizes outputs that would be very unlikely under the reference model.

Optimal solution (analytically):

π∗(y∣x)=1Z(x)πref(y∣x)exp⁡(r(x,y)β).(6)\pi^*(y|x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{r(x,y)}{\beta}\right). \tag{6}

The optimal policy is the reference policy reweighted by exponentiated reward.

Connection to DPO

DPO derives from RLHF: Substituting the optimal policy (equation 6) into the Bradley-Terry model:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x).(7)r(x, y) = \beta\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x). \tag{7}

Plugging into the preference loss eliminates the reward model entirely:

LDPO=−E[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))].(8)\mathcal{L}_{\text{DPO}} = -\mathbb{E}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right]. \tag{8}

DPO = RLHF without explicit reward modeling or RL. Same objective, simpler optimization.

Common Pitfalls

Pitfall 1. Reward hacking. The model finds outputs that score highly on the reward model but are low quality (e.g., excessive hedging, sycophancy). Mitigation: KL penalty, reward model ensembles.

Pitfall 2. Mode collapse. The policy collapses to a narrow set of "safe" responses, losing diversity. Mitigation: entropy bonus, temperature sampling during RL.

Pitfall 3. Reward model overoptimization. As the policy optimizes harder, reward model score increases but true quality decreases (Goodhart's law). Mitigation: early stopping, conservative β\beta.

Summary

  • RLHF pipeline: SFT → Reward Model → PPO optimization.
  • Reward model: Bradley-Terry preference model from human comparisons.
  • PPO: Maximize reward subject to KL constraint from reference policy.
  • KL penalty prevents reward hacking and preserves capabilities.
  • DPO is a simplified equivalent: skips RM and RL entirely.

Exercises

Exercise 1. Derive the optimal policy (equation 6) by solving the KL-constrained optimization problem.

Exercise 2. Show that the Bradley-Terry model is equivalent to logistic regression on reward differences.

Exercise 3. For β=0.1\beta = 0.1 and reward difference 2.0: compute the optimal policy's probability ratio π∗/πref\pi^*/\pi_{\text{ref}}.

Exercise 4. Explain why reward model quality degrades as the policy moves far from the training distribution.

Exercise 5. Derive DPO (equation 8) from the RLHF objective by substituting the optimal policy into the Bradley-Terry preference model.

Next

Context Length Extension