KL Divergence

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Cross-Entropy and the Definition of KL Divergence
6. Step-by-Step Derivation
7. Gibbs' Inequality: Proof of Non-Negativity
8. Properties of KL Divergence
9. Forward KL vs Reverse KL
10. KL Divergence and Maximum Likelihood
11. Closed Form for Gaussian Distributions
12. Conditional and Mutual Information Connections
13. Connection to Machine Learning
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

---

Learning Objectives

After reading this chapter, you should be able to:

1. Derive KL divergence as $D_{\text{KL}}(P \| Q) = H(P, Q) - H(P)$ from the definitions of cross-entropy and entropy.
2. Prove non-negativity (Gibbs' inequality) via Jensen's inequality and identify when equality holds.
3. Explain why KL divergence is asymmetric and why it is not a metric.
4. Distinguish forward KL (mean-seeking) from reverse KL (mode-seeking) and predict optimization behavior.
5. Prove that minimizing cross-entropy is equivalent to minimizing KL divergence and to MLE.
6. Derive the closed-form KL between multivariate Gaussians.
7. Identify KL terms in VAEs, diffusion models, variational inference, and RLHF objectives.

---

Prerequisites

• Bayes' Theorem: evidence, posterior, likelihood ratios.
• Entropy & Information Content: entropy, cross-entropy, mutual information, Jensen's inequality.
• The Multivariate Gaussian: Gaussian density, trace and quadratic forms.

---

Notation

• **$D_{\mathrm{KL}}(P \** — Q)$ — Kullback–Leibler divergence
• $H(P)$ — Shannon entropy
• $H(P, Q)$ — Cross-entropy
• $p(x), q(x)$ — Probability densities or mass functions
• $\mathbb{E}_P[\cdot]$ — Expectation under distribution $P$

---

Core Intuition

How different are two probability distributions $P$ and $Q$? The Kullback–Leibler (KL) divergence answers: If data are generated from $P$, how many extra bits per sample do we waste by encoding them with a code optimized for $Q$?

KL divergence is not a distance in the geometric sense — it is asymmetric ($D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)$ in general) and does not satisfy the triangle inequality. Nevertheless, it is the canonical measure of distribution mismatch in machine learning:

• Training classifiers minimizes $D_{\text{KL}}(P_{\text{data}} \| P_{\text{model}})$ via cross-entropy.
• VAEs regularize the encoder with $D_{\text{KL}}(q(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z}))$.
• RLHF constrains policy updates with $D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$.
• Diffusion model variational bounds decompose into sums of Gaussian KL terms.

This chapter derives KL divergence rigorously, proves its key properties, and connects it to the loss functions used throughout modern AI.

Series context. This is Part IV of Chapter 3 in Volume I. It follows Entropy and precedes Maximum Likelihood Estimation.

---

Cross-Entropy and the Definition of KL Divergence

Definition 1 (Cross-Entropy — Discrete). For distributions $P$ and $Q$ on countable $\mathcal{X}$:

$$H(P, Q) = -\sum_{x \in \mathcal{X}} P(x) \log Q(x). \tag{1}$$

Definition 2 (Entropy — Discrete).

$$H(P) = -\sum_{x} P(x) \log P(x). \tag{2}$$

Definition 3 (KL Divergence). The Kullback–Leibler divergence from $Q$ to $P$ (equivalently, of $P$ from $Q$) is:

$$D_{\text{KL}}(P \| Q) = H(P, Q) - H(P) = \sum_{x \in \mathcal{X}} P(x) \log \frac{P(x)}{Q(x)}. \tag{3}$$

Convention. When $P(x) = 0$, the term $P(x) \log \frac{P(x)}{Q(x)} = 0$. When $P(x) > 0$ and $Q(x) = 0$, the term is $+\infty$ (divergence is infinite if $Q$ fails to cover the support of $P$).

Definition 4 (KL Divergence — Continuous). For PDFs $p$ and $q$:

$$D_{\text{KL}}(p \| q) = \int p(x) \log \frac{p(x)}{q(x)}\, dx. \tag{4}$$

Interpretation. $D_{\text{KL}}(P \| Q)$ is the expected log-likelihood ratio $\mathbb{E}_{x \sim P}[\log P(x) - \log Q(x)]$ — how much more (or less) log-probability $P$ assigns to typical samples from $P$ compared to $Q$.

---

Step-by-Step Derivation

We derive (3) from (1) and (2) with no gaps.

Step 1. Write cross-entropy:

$$H(P, Q) = -\sum_x P(x) \log Q(x). \tag{5}$$

Step 2. Write entropy:

$$H(P) = -\sum_x P(x) \log P(x). \tag{6}$$

Step 3. Subtract (6) from (5):

$$H(P, Q) - H(P) = -\sum_x P(x) \log Q(x) + \sum_x P(x) \log P(x). \tag{7}$$

Step 4. Combine the sums (same index set):

$$= \sum_x P(x) [\log P(x) - \log Q(x)] = \sum_x P(x) \log \frac{P(x)}{Q(x)}. \tag{8}$$

Step 5. By Definition 3, this equals $D_{\text{KL}}(P \| Q)$. $\blacksquare$

Remark. The derivation requires only algebra and the definitions — no additional assumptions.

---

Gibbs' Inequality: Proof of Non-Negativity

Theorem 1 (Gibbs' Inequality). For discrete $P$ and $Q$:

$$D_{\text{KL}}(P \| Q) \geq 0, \tag{9}$$

with equality if and only if $P(x) = Q(x)$ for all $x$ with $P(x) > 0$.

Proof. Consider $-D_{\text{KL}}(P \| Q)$:

$$-D_{\text{KL}}(P \| Q) = -\sum_x P(x) \log \frac{P(x)}{Q(x)} = \sum_x P(x) \log \frac{Q(x)}{P(x)} = \mathbb{E}_{x \sim P}\left[\log \frac{Q(x)}{P(x)}\right]. \tag{10}$$

The function $\log(\cdot)$ is concave. By Jensen's inequality for concave $f$:

$$\mathbb{E}[f(Z)] \leq f(\mathbb{E}[Z]), \tag{11}$$

with $Z = \frac{Q(x)}{P(x)}$ and expectation over $x \sim P$:

$$\sum_x P(x) \log \frac{Q(x)}{P(x)} \leq \log \left(\sum_x P(x) \cdot \frac{Q(x)}{P(x)}\right) = \log \left(\sum_x Q(x)\right) = \log 1 = 0. \tag{12}$$

Therefore $-D_{\text{KL}}(P \| Q) \leq 0$, hence $D_{\text{KL}}(P \| Q) \geq 0$.

Equality condition. Jensen's inequality is tight iff $Z = Q(x)/P(x)$ is constant for all $x$ in the support of $P$. Since both $P$ and $Q$ sum to 1, the constant must be 1, so $P(x) = Q(x)$ wherever $P(x) > 0$. $\blacksquare$

Theorem 2 (Continuous Case). For PDFs $p, q$ with $p$ absolutely continuous w.r.t. $q$:

$$D_{\text{KL}}(p \| q) = \int p(x) \log \frac{p(x)}{q(x)}\, dx \geq 0, \tag{13}$$

with the same equality condition (a.e.).

Proof. Same Jensen argument with the integral in place of the sum. $\blacksquare$

---

Properties of KL Divergence

Property 1 (Non-negativity). $D_{\text{KL}}(P \| Q) \geq 0$ (Theorem 1).

Property 2 (Identity of indiscernibles). $D_{\text{KL}}(P \| Q) = 0 \iff P = Q$ (on the support of $P$).

Property 3 (Asymmetry). In general, $D_{\text{KL}}(P \| Q) \neq D_{\text{KL}}(Q \| P)$.

Example. Let $P = (0.9, 0.1)$ and $Q = (0.5, 0.5)$ on $\{0, 1\}$:

$$D_{\text{KL}}(P \| Q) = 0.9 \log \frac{0.9}{0.5} + 0.1 \log \frac{0.1}{0.5} \approx 0.252, \tag{14}$$ $$D_{\text{KL}}(Q \| P) = 0.5 \log \frac{0.5}{0.9} + 0.5 \log \frac{0.5}{0.1} \approx 0.510. \tag{15}$$

Property 4 (Not a metric). KL violates the triangle inequality: there exist $P, Q, R$ with $D_{\text{KL}}(P \| R) > D_{\text{KL}}(P \| Q) + D_{\text{KL}}(Q \| R)$.

Property 5 (Chain rule). For joint distributions:

$$D_{\text{KL}}(P(X,Y) \| Q(X,Y)) = D_{\text{KL}}(P(X) \| Q(X)) + D_{\text{KL}}(P(Y|X) \| Q(Y|X)), \tag{16}$$

where the second term is $\mathbb{E}_{x \sim P(X)}[D_{\text{KL}}(P(Y|X=x) \| Q(Y|X=x))]$.

Property 6 (Data processing / Invariance). If $Y = f(X)$ deterministically, then $D_{\text{KL}}(P(Y) \| Q(Y)) \leq D_{\text{KL}}(P(X) \| Q(X))$ — processing cannot increase divergence (information processing inequality).

Property 7 (Additivity for independent products). If $P = P_1 \times P_2$ and $Q = Q_1 \times Q_2$, then:

$$D_{\text{KL}}(P \| Q) = D_{\text{KL}}(P_1 \| Q_1) + D_{\text{KL}}(P_2 \| Q_2). \tag{17}$$

---

Forward KL vs Reverse KL

The asymmetry of KL divergence leads to qualitatively different behavior when used as an optimization objective.

Forward KL: $D_{\text{KL}}(P \| Q)$ — Mean-Seeking

Minimize over $Q$ (approximation) with $P$ (target) fixed:

$$\min_Q D_{\text{KL}}(P \| Q) = \min_Q \sum_x P(x) \log \frac{P(x)}{Q(x)}. \tag{18}$$

Behavior.

• If $P(x) > 0$ and $Q(x) \to 0$, then $\log \frac{P(x)}{Q(x)} \to +\infty$. So $Q(x) > 0$ wherever $P(x) > 0$ — $Q$ must cover all modes of $P$.
• $Q$ tends to be broader than $P$ (overdispersed) to avoid missing any mass.
• Used in expectation propagation, variational inference (ELBO), and MLE (via cross-entropy).

Reverse KL: $D_{\text{KL}}(Q \| P)$ — Mode-Seeking

Minimize over $Q$ with $P$ fixed:

$$\min_Q D_{\text{KL}}(Q \| P) = \min_Q \sum_x Q(x) \log \frac{Q(x)}{P(x)}. \tag{19}$$

Behavior.

• If $Q(x) > 0$ and $P(x) = 0$, then $\log \frac{Q(x)}{P(x)} \to +\infty$. So $Q(x) = 0$ wherever $P(x) = 0$ — $Q$ is supported only where $P$ is positive.
• $Q$ can ignore modes of $P$ and concentrate on one.
• Used in RLHF (policy vs reference), some GAN objectives, and mode-seeking variational approximations.

Proposition 1. For multimodal $P$, the forward KL minimizer $\arg\min_Q D_{\text{KL}}(P \| Q)$ typically covers all modes; the reverse KL minimizer often selects a single mode.

---

KL Divergence and Maximum Likelihood

Theorem 3 (MLE as KL Minimization). Let $\hat{p}_n(x) = \frac{1}{n}\sum_{i=1}^n \delta(x - x_i)$ be the empirical distribution. Then:

$$\hat{\boldsymbol\theta}_{\text{MLE}} = \underset{\boldsymbol\theta}{\text{argmax}} \; \frac{1}{n}\sum_{i=1}^n \log p_\theta(x_i) = \underset{\boldsymbol\theta}{\text{argmin}} \; D_{\text{KL}}(\hat{p}_n \| p_\theta). \tag{20}$$

Proof. Expand the KL:

$$D_{\text{KL}}(\hat{p}_n \| p_\theta) = \sum_x \hat{p}_n(x) \log \frac{\hat{p}_n(x)}{p_\theta(x)} = -H(\hat{p}_n) - \frac{1}{n}\sum_{i=1}^n \log p_\theta(x_i). \tag{21}$$

$H(\hat{p}_n)$ is constant in $\boldsymbol\theta$, so minimizing KL $\equiv$ maximizing $\sum_i \log p_\theta(x_i)$ $\equiv$ MLE. $\blacksquare$

Corollary 1 (Cross-Entropy Equivalence). For fixed $P$:

$$\underset{Q}{\text{argmin}} \; H(P, Q) = \underset{Q}{\text{argmin}} \; D_{\text{KL}}(P \| Q). \tag{22}$$

Since $H(P, Q) = H(P) + D_{\text{KL}}(P \| Q)$ and $H(P)$ is constant w.r.t. $Q$.

---

Closed Form for Gaussian Distributions

Theorem 4 (Univariate Gaussian KL). For $P = \mathcal{N}(\mu_1, \sigma_1^2)$ and $Q = \mathcal{N}(\mu_2, \sigma_2^2)$:

$$D_{\text{KL}}(P \| Q) = \log \frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2\sigma_2^2} - \frac{1}{2}. \tag{23}$$

Proof. Starting from (4):

$$D_{\text{KL}} = \int p(x) [\log p(x) - \log q(x)]\, dx. \tag{24}$$

Substitute Gaussian log-densities:

$$\log p(x) = -\frac{1}{2}\log(2\pi\sigma_1^2) - \frac{(x-\mu_1)^2}{2\sigma_1^2}, \tag{25}$$ $$\log q(x) = -\frac{1}{2}\log(2\pi\sigma_2^2) - \frac{(x-\mu_2)^2}{2\sigma_2^2}. \tag{26}$$

The difference $\log p(x) - \log q(x)$ is quadratic in $x$. Taking expectation under $P$:

$$\mathbb{E}_P[(x-\mu_1)^2] = \sigma_1^2, \quad \mathbb{E}_P[(x-\mu_2)^2] = \sigma_1^2 + (\mu_1 - \mu_2)^2. \tag{27}$$

Collecting constant and quadratic terms yields (23). $\blacksquare$

Theorem 5 (Multivariate Gaussian KL). For $p = \mathcal{N}(\boldsymbol\mu_1, \boldsymbol\Sigma_1)$ and $q = \mathcal{N}(\boldsymbol\mu_2, \boldsymbol\Sigma_2)$ in $\mathbb{R}^d$:

$$D_{\text{KL}}(p \| q) = \frac{1}{2}\left[\log\frac{|\boldsymbol\Sigma_2|}{|\boldsymbol\Sigma_1|} - d + \text{tr}(\boldsymbol\Sigma_2^{-1}\boldsymbol\Sigma_1) + (\boldsymbol\mu_2-\boldsymbol\mu_1)^T\boldsymbol\Sigma_2^{-1}(\boldsymbol\mu_2-\boldsymbol\mu_1)\right]. \tag{28}$$

Proof sketch. Use $\log p(\mathbf{x}) = -\frac{d}{2}\log(2\pi) - \frac{1}{2}\log|\boldsymbol\Sigma_1| - \frac{1}{2}(\mathbf{x}-\boldsymbol\mu_1)^T\boldsymbol\Sigma_1^{-1}(\mathbf{x}-\boldsymbol\mu_1)$ and take expectation under $p$, using $\mathbb{E}[(\mathbf{x}-\boldsymbol\mu_1)(\mathbf{x}-\boldsymbol\mu_1)^T] = \boldsymbol\Sigma_1$ and the trace trick for cross-terms. See The Multivariate Gaussian, equation (33).

Special case (Standard Normal Prior). For $p = \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ and $q = \mathcal{N}(\mathbf{0}, \mathbf{I})$:

$$D_{\text{KL}}(p \| q) = \frac{1}{2}\left[-\log|\boldsymbol\Sigma| - d + \text{tr}(\boldsymbol\Sigma) + \boldsymbol\mu^T\boldsymbol\mu\right]. \tag{29}$$

This is the VAE regularization term when the prior is standard normal.

---

Conditional and Mutual Information Connections

Definition 5 (Conditional KL).

$$D_{\text{KL}}(P(Y|X) \| Q(Y|X)) = \mathbb{E}_{x \sim P(X)}\left[D_{\text{KL}}(P(Y|X=x) \| Q(Y|X=x))\right]. \tag{30}$$

Theorem 6 (Mutual Information as KL).

$$I(X; Y) = D_{\text{KL}}(P(X,Y) \| P(X)P(Y)). \tag{31}$$

Proof. Expand (3) for the joint vs product of marginals; terms telescope to $H(X) + H(Y) - H(X,Y) = I(X;Y)$. See Entropy. $\blacksquare$

Theorem 7 (ELBO Decomposition). For latent variable model with $p(\mathbf{x}) = \int p(\mathbf{x}|\mathbf{z})p(\mathbf{z})\, d\mathbf{z}$ and variational $q(\mathbf{z}|\mathbf{x})$:

$$\log p(\mathbf{x}) = \mathcal{L}_{\text{ELBO}} + D_{\text{KL}}(q(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z}|\mathbf{x})), \tag{32}$$

where $\mathcal{L}_{\text{ELBO}} = \mathbb{E}_{q}[\log p(\mathbf{x}|\mathbf{z})] - D_{\text{KL}}(q(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z}))$. Maximizing ELBO minimizes forward KL to the true posterior.

---

Connection to Machine Learning

Variational Autoencoders

$$\mathcal{L}_{\text{VAE}} = \mathbb{E}_{q_\phi(\mathbf{z}|\mathbf{x})}[\log p_\theta(\mathbf{x}|\mathbf{z})] - D_{\text{KL}}(q_\phi(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z})). \tag{33}$$

The KL term uses (29) when $p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ and $q$ is diagonal Gaussian.

Diffusion Models

The variational bound decomposes:

$$L = D_{\text{KL}}(q(\mathbf{x}_T|\mathbf{x}_0) \| p(\mathbf{x}_T)) + \sum_{t=2}^{T} D_{\text{KL}}(q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0) \| p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)) + \text{const}. \tag{34}$$

Each term is a Gaussian KL (Theorem 5) — tractable closed form.

RLHF and PPO

$$\mathcal{L} = -\mathbb{E}_{x \sim \pi_\theta}[R(x)] + \beta \cdot D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}}). \tag{35}$$

Reverse KL (or symmetrized variants) prevents the policy from drifting far from the reference LLM, avoiding reward hacking.

Variational Inference

Approximating posterior $p(\boldsymbol\theta|\mathcal{D})$ with $q(\boldsymbol\theta)$ by minimizing $D_{\text{KL}}(q \| p(\boldsymbol\theta|\mathcal{D}))$ (reverse KL, mode-seeking) or maximizing ELBO equivalent to minimizing $D_{\text{KL}}(q \| p(\boldsymbol\theta|\mathcal{D}))$ — see Variational Inference.

---

Common Pitfalls and Misconceptions

Pitfall 1: Calling KL a "distance." It is a divergence or premetric, not a metric — asymmetry and triangle inequality failure matter for interpretation.

Pitfall 2: Swapping argument order. $D_{\text{KL}}(P \| Q)$ and $D_{\text{KL}}(Q \| P)$ optimize differently (mean-seeking vs mode-seeking). VAEs use forward KL in ELBO; RLHF often uses reverse KL.

Pitfall 3: Infinite KL from support mismatch. If $Q(x) = 0$ where $P(x) > 0$, forward KL is $+\infty$. Ensure model support covers data support (softmax over all classes, positive variance in Gaussians).

Pitfall 4: Confusing KL with JS divergence. Jensen–Shannon divergence $\text{JS}(P \| Q) = \frac{1}{2}D_{\text{KL}}(P \| M) + \frac{1}{2}D_{\text{KL}}(Q \| M)$ with $M = \frac{1}{2}(P+Q)$ is symmetric and bounded — different tradeoffs.

Pitfall 5: Ignoring that cross-entropy offset differs from KL scale. $H(P,Q) = H(P) + D_{\text{KL}}(P \| Q)$; training logs cross-entropy, but generalization depends on KL to the true $P$, not $H(P)$ alone.

---

Research Perspective

Kullback and Leibler (1951) introduced $D_{\text{KL}}$ in the context of sufficient statistics and hypothesis testing. It became central to information theory (I-divergence, Csiszár) and statistics (Bahadur efficiency, large deviations).

In machine learning, KL appears in the EM algorithm (E-step as KL minimization), variational Bayes (Jordan et al., 1999), VAEs (Kingma & Welling, 2014), and modern alignment (RLHF, Ouyang et al., 2022). Research on $f$-divergences (generalizing KL), Wasserstein distances (true metrics for generative modeling), and rate-distortion theory continues to refine how we measure distribution mismatch.

Open questions include: optimal divergence choice for generative models, tight KL-based generalization bounds for deep networks, and whether reverse KL penalties in RLHF optimally balance exploration and safety.

---

Summary of Takeaways

• KL definition — $\sum P(x) \log \frac{P(x)}{Q(x)}$ — Distribution mismatch
• Cross-entropy relation — $D_{\text{KL}}(P \ — Q) = H(P,Q) - H(P)$ — Training loss link
• Non-negativity — Gibbs' inequality (Jensen) — Valid divergence
• Forward KL — Mean-seeking, covers modes — MLE, ELBO
• Reverse KL — Mode-seeking, zero-forcing — RLHF, some VI
• Gaussian KL — (28), (29) — VAE, diffusion
• MLE equivalence — $\text{argmin}_\theta D_{\text{KL}}(\hat{p}_n \ — p_\theta)$ — Foundation of learning

Next article: Maximum Likelihood Estimation → — estimating parameters by minimizing KL to the empirical distribution.

---

Exercises

Exercise 1 (Derivation). Reproduce the step-by-step derivation (5)–(8) and verify for $P = (0.5, 0.5)$, $Q = (0.25, 0.75)$.

Exercise 2 (Gibbs). Write out Jensen's inequality step for the continuous case (13).

Exercise 3 (Asymmetry). Compute both $D_{\text{KL}}(P \| Q)$ and $D_{\text{KL}}(Q \| P)$ for $P = \text{Bernoulli}(0.1)$, $Q = \text{Bernoulli}(0.5)$.

Exercise 4 (Gaussian). Derive (23) completely, showing all integral steps.

Exercise 5 (Multivariate). Derive (29) from (28) with $\boldsymbol\mu_2 = \mathbf{0}$, $\boldsymbol\Sigma_2 = \mathbf{I}$.

Exercise 6 (Chain rule). Prove (16) by expanding the joint KL.

Exercise 7 (Conceptual). Explain why VAEs use forward KL in the ELBO but RLHF uses reverse KL for the policy penalty.

Exercise 8 (ELBO). Prove (32) starting from $\log p(\mathbf{x}) = \log \int q(\mathbf{z}|\mathbf{x}) \frac{p(\mathbf{x},\mathbf{z})}{q(\mathbf{z}|\mathbf{x})} d\mathbf{z}$ and Jensen's inequality.