ML

DPO, KTO & Direct Alignment Methods

Aligning LLMs without RL: Direct Preference Optimization, Kahneman-Tversky Optimization, IPO, ORPO, SimPO — the mathematics of preference-based training without reward models or PPO.

Advanced

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. DPO: Direct Preference Optimization
  5. DPO Derivation from RLHF
  6. IPO: Identity Preference Optimization
  7. KTO: Kahneman-Tversky Optimization
  8. ORPO: Odds Ratio Preference Optimization
  9. SimPO: Simple Preference Optimization
  10. Comparison & Selection Guide
  11. Common Pitfalls
  12. Summary
  13. Exercises

Learning Objectives

  1. Derive DPO from the RLHF objective (closed-form solution).
  2. Explain the implicit reward model in DPO.
  3. Compare DPO, IPO, KTO, ORPO, and SimPO objectives.
  4. Identify when each method is preferred (data types, scale, compute).
  5. Implement DPO training and diagnose common failure modes.

Notation

  • πθ\pi_\theta — policy being trained
  • πref\pi_{\text{ref}} — reference policy (frozen SFT model)
  • (x,yw,yl)(x, y_w, y_l) — prompt, preferred response, dispreferred response

Core Intuition

RLHF is expensive (4 models, PPO instability). DPO asks: can we directly optimize preferences WITHOUT explicitly training a reward model or running RL? The answer: yes. The optimal RLHF policy has a closed-form relationship to the reward — we can rearrange to get a loss that directly trains the policy from preference pairs, using only standard cross-entropy-style optimization.

DPO Training

Chosen ✓"Helpful, accurate response"Rejected ✗"Harmful, wrong response"Implicit reward margin: 0.80DPO loss: 0.513 | β=0.5 → reward scale 0.40
Beta β
0.50
ChosenRejected
Explore: DPO directly optimizes preferences without a reward model. Higher β increases the penalty for preferring rejected responses — sharper alignment but risk of overfitting.

DPO: Direct Preference Optimization

Rafailov et al. (2023): The DPO loss:

LDPO=−E(x,yw,yl)[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))].(1)\boxed{\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l)}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)\right].} \tag{1}

Interpretation: Increase the probability of preferred response (relative to reference) AND decrease the probability of dispreferred response.

The implicit reward:

r(x,y)=βlog⁡πθ(y∣x)πref(y∣x)+βlog⁡Z(x).(2)r(x, y) = \beta\log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x). \tag{2}

DPO doesn't need an explicit reward model — the policy itself defines an implicit one.

DPO Derivation from RLHF

Start with RLHF objective:

max⁡πEy∼π[r(x,y)]−βDKL(π∥πref).(3)\max_\pi \mathbb{E}_{y \sim \pi}[r(x, y)] - \beta D_{\text{KL}}(\pi \| \pi_{\text{ref}}). \tag{3}

Optimal policy (closed form):

π∗(y∣x)=1Z(x)πref(y∣x)exp⁡(r(x,y)β).(4)\pi^*(y|x) = \frac{1}{Z(x)}\pi_{\text{ref}}(y|x)\exp\left(\frac{r(x,y)}{\beta}\right). \tag{4}

Rearrange to express reward in terms of policy:

r(x,y)=βlog⁡π∗(y∣x)πref(y∣x)+βlog⁡Z(x).(5)r(x, y) = \beta\log\frac{\pi^*(y|x)}{\pi_{\text{ref}}(y|x)} + \beta\log Z(x). \tag{5}

Substitute into Bradley-Terry preference model:

P(yw≻yl)=σ(r(x,yw)−r(x,yl))=σ(βlog⁡π∗(yw∣x)πref(yw∣x)−βlog⁡π∗(yl∣x)πref(yl∣x)).(6)P(y_w \succ y_l) = \sigma(r(x, y_w) - r(x, y_l)) = \sigma\left(\beta\log\frac{\pi^*(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta\log\frac{\pi^*(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right). \tag{6}

The Z(x)Z(x) cancels. Replace π∗\pi^* with the learnable πθ\pi_\theta and minimize negative log-likelihood → equation (1).

IPO: Identity Preference Optimization

Azar et al. (2023): DPO can overfit because the sigmoid saturates. IPO uses a squared loss:

LIPO=E[(log⁡πθ(yw∣x)πref(yw∣x)−log⁡πθ(yl∣x)πref(yl∣x)−12β)2].(7)\mathcal{L}_{\text{IPO}} = \mathbb{E}\left[\left(\log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} - \frac{1}{2\beta}\right)^2\right]. \tag{7}

Advantage: Non-saturating gradient. Even when the model is "correct" (prefers ywy_w), it still gets gradient signal pushing toward the exact margin.

Disadvantage: Can be too aggressive (pushes past the preference boundary).

KTO: Kahneman-Tversky Optimization

Ethayarajh et al. (2024): Doesn't require PAIRS — only needs individual responses labeled as "good" or "bad":

LKTO=Ey good[1−σ(βr(y)−z0)]+λEy bad[1−σ(z0−βr(y))],(8)\mathcal{L}_{\text{KTO}} = \mathbb{E}_{y \text{ good}}\left[1 - \sigma(\beta r(y) - z_0)\right] + \lambda\mathbb{E}_{y \text{ bad}}\left[1 - \sigma(z_0 - \beta r(y))\right], \tag{8}

where r(y)=log⁡πθ(y∣x)/πref(y∣x)r(y) = \log\pi_\theta(y|x)/\pi_{\text{ref}}(y|x) and z0z_0 is a baseline.

Key advantage: No need for paired comparisons. Can use: thumbs-up/down data, user engagement signals, or binary quality labels.

When to use: When you have lots of binary feedback but not pairwise comparisons (e.g., user ratings on a chatbot).

ORPO: Odds Ratio Preference Optimization

Hong et al. (2024): No reference model needed. Combines SFT and alignment in one objective:

LORPO=LSFT(yw)−λlog⁡σ(log⁡odds(yw)odds(yl)),(9)\mathcal{L}_{\text{ORPO}} = \mathcal{L}_{\text{SFT}}(y_w) - \lambda\log\sigma\left(\log\frac{\text{odds}(y_w)}{\text{odds}(y_l)}\right), \tag{9}

where odds(y)=p(y)/(1−p(y))\text{odds}(y) = p(y) / (1 - p(y)).

Advantage: Single training stage (no separate SFT → DPO). No reference model in memory. Simpler implementation.

SimPO: Simple Preference Optimization

Meng et al. (2024): Use average log-probability instead of sum (length-normalized):

LSimPO=−log⁡σ(β∣yw∣log⁡πθ(yw∣x)−β∣yl∣log⁡πθ(yl∣x)−γ),(10)\mathcal{L}_{\text{SimPO}} = -\log\sigma\left(\frac{\beta}{|y_w|}\log\pi_\theta(y_w|x) - \frac{\beta}{|y_l|}\log\pi_\theta(y_l|x) - \gamma\right), \tag{10}

where γ\gamma is a target margin.

No reference model: Uses the policy's own length-normalized probability as the implicit reward.

Advantage: Removes length bias (common in DPO where longer responses get lower log-prob). No reference model needed in memory.

Comparison & Selection Guide

MethodNeeds PairsNeeds Ref ModelTraining StagesMemory
RLHFYesYes (+ RM + value)3 (SFT→RM→PPO)4x model
DPOYesYes2 (SFT→DPO)2x model
IPOYesYes22x model
KTONo (binary)Yes22x model
ORPOYesNo11x model
SimPOYesNo11x model

Recommendation:

  • Maximum quality: RLHF (if you can afford it).
  • Standard alignment: DPO (best balance of quality and simplicity).
  • Binary feedback only: KTO.
  • Minimum complexity: SimPO or ORPO (single stage, no reference).

Common Pitfalls

Pitfall 1. Using a stale reference model. The reference should be the SFT model used BEFORE DPO, not an arbitrary checkpoint. Using the wrong reference misaligns the implicit reward.

Pitfall 2. β\beta too low in DPO. The model diverges far from the reference, producing reward-hacked outputs (similar to RLHF overoptimization). Start with β=0.1\beta=0.1 and tune.

Pitfall 3. Preference data quality. DPO is VERY sensitive to label noise. If 20% of preferences are flipped (human disagreement), DPO quality degrades significantly. Clean data is essential.

Summary

  • DPO: Direct optimization from preferences; implicit reward model; 2x memory.
  • IPO: Non-saturating loss; more stable gradients; prevents overconfidence.
  • KTO: Works with binary (good/bad) labels; no pairs needed.
  • ORPO/SimPO: No reference model; single training stage; simplest implementation.
  • All methods avoid the complexity/instability of PPO.
  • DPO matches RLHF quality in most benchmarks at 2-3x lower cost.

Exercises

Exercise 1. Derive equation (1) step by step from the RLHF objective.

Exercise 2. For DPO with β=0.1\beta=0.1: if πθ(yw)/πref(yw)=2\pi_\theta(y_w)/\pi_{\text{ref}}(y_w) = 2 and πθ(yl)/πref(yl)=0.5\pi_\theta(y_l)/\pi_{\text{ref}}(y_l) = 0.5, compute the loss value and gradient direction.

Exercise 3. Compare the memory requirements of DPO vs RLHF for a 70B model. How much GPU memory is saved?

Exercise 4. Design a KTO training pipeline using only thumbs-up/down data from a deployed chatbot. Specify data collection, filtering, and training details.

Exercise 5. Prove that DPO and RLHF have the same optimal policy (the derivation shows they're equivalent at convergence).

Next

Transfer Learning & Domain Adaptation