Direct Preference Optimization (DPO)

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Preference Data and Bradley-Terry Model
6. The RLHF Objective
7. Closed-Form Optimal Policy
8. Deriving the DPO Loss
9. Implicit Reward Model
10. DPO vs. RLHF Pipeline
11. Worked Examples
12. Connection to the Broader Curriculum
13. Common Pitfalls and Misconceptions
14. Research Perspective
15. Summary of Takeaways
16. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. State the Bradley-Terry preference model $P(y_w \succ y_l \mid x)$.
2. Write the RLHF objective with KL penalty to reference policy $\pi_{\text{ref}}$.
3. Derive the closed-form optimal policy $\pi^*$ in terms of reward $r$.
4. Eliminate the reward model to obtain the DPO loss directly on policy log-probabilities.
5. Explain why DPO avoids reinforcement learning (PPO) during alignment.

Prerequisites

• Logistic Regression — sigmoid, cross-entropy
• KL Divergence
• Scaling Laws — pretrained reference policy

Notation

• $x$ — Prompt
• $y_w, y_l$ — Preferred (winner) and dispreferred (loser) completions
• $\pi_\theta, \pi_{\mathrm{ref}}$ — Trainable and frozen reference policies
• $r(x, y)$ — Latent reward function
• $\beta$ — KL penalty coefficient
• $\sigma(\cdot)$ — Logistic sigmoid

Core Intuition

Alignment adjusts a pretrained LLM to produce helpful, harmless outputs. RLHF (Reinforcement Learning from Human Feedback) trains a reward model on human preferences, then optimizes the policy with PPO — unstable and complex.

DPO (Rafailov et al., 2023) shows the optimal RLHF policy has a closed form in terms of the reward. Substituting back into the preference model eliminates the reward entirely — alignment becomes supervised learning on preference pairs with a simple classification loss.

Series context. Volume III, Chapter 13 (Alignment).

Preference Data and Bradley-Terry Model

Definition 1 (Preference Dataset). Triples $(x, y_w, y_l)$ where $y_w$ is preferred (winner) over $y_l$ (loser) for prompt $x$.

Definition 2 (Bradley-Terry Model).

where $r(x, y)$ is a latent reward function and $\sigma$ is the logistic sigmoid from Logistic Regression.

The RLHF Objective

Definition 3 (KL-Constrained RLHF).

where $\pi_{\text{ref}}$ is the frozen pretrained policy and $\beta > 0$ controls deviation from reference.

Closed-Form Optimal Policy

Theorem 1 (Optimal Policy). The unique maximizer of (2) is:

where $Z(x) = \sum_y \pi_{\text{ref}}(y|x) \exp(r(x,y)/\beta)$ is the partition function.

Proof. Lagrange multipliers on probability simplex; Gibbs distribution form. $\blacksquare$

Corollary 1 (Reward Recovery).

Deriving the DPO Loss

Substitute (4) into Bradley-Terry (1). The partition function $Z(x)$ cancels:

Theorem 2 (DPO Loss).

Proof. From (1):

using (4) and cancellation of $\log Z(x)$. Maximum likelihood on preferences gives (5). $\blacksquare$

Important equation. DPO optimizes policy $\pi_\theta$ directly — no explicit reward model, no PPO.

Implicit Reward Model

Definition 4 (Implicit Reward).

DPO implicitly defines a reward through the policy ratio to reference.

DPO vs. RLHF Pipeline

• 1 — SFT on demonstrations — SFT (same)
• 2 — Train reward model — —
• 3 — PPO on reward — DPO loss (5)
• Stability — PPO hyperparameters — Standard SGD
• Compute — 3 models — 2 models ($\pi_\theta$, frozen $\pi_{\text{ref}}$)

Worked Examples

Example 1: Preference Probability

If $\pi_\theta(y_w|x)/\pi_{\text{ref}}(y_w|x) = 2$ and ratio for $y_l$ is 1, DPO logit $= \beta \log 2$. For $\beta = 0.1$, preference probability $\approx 0.52$.

Example 2: Identical to Reference

If $\pi_\theta = \pi_{\text{ref}}$, DPO logit $= 0$ — 50% preference probability (uninformative).

Connection to the Broader Curriculum

• Logistic Regression — sigmoid preference model
• KL Divergence — RLHF penalty
• LoRA — efficient DPO fine-tuning
• RAG — complementary alignment technique

Common Pitfalls and Misconceptions

Pitfall 1: DPO still requires good SFT initialization.

Pitfall 2: $\beta$ too small → ignore preferences; too large → collapse to reference.

Pitfall 3: Length bias in preferences affects DPO.

Pitfall 4: DPO assumes Bradley-Terry transitivity.

Research Perspective

DPO (Rafailov et al., 2023). Extensions: IPO, KTO, ORPO, SimPO. Active: online DPO, multi-objective alignment.

Summary of Takeaways

• Bradley-Terry — $\sigma(r_w - r_l)$
• Optimal policy — $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$
• DPO loss — (5)
• Implicit reward — $\beta \log(\pi_\theta / \pi_{\text{ref}})$

Next: LoRA

Exercises

Exercise 1. Derive (3) from (2).

Exercise 2. Prove $Z(x)$ cancels in (6).

Exercise 3. Derive gradient of (5) w.r.t. $\theta$.

Exercise 4. Effect of $\beta \to 0$ and $\beta \to \infty$.

Exercise 5. Connect Bradley-Terry to Logistic Regression.

Exercise 6. Why store $\pi_{\text{ref}}$ separately?

Exercise 7. DPO with LoRA on both policies?

Exercise 8. Limitations of preference transitivity.