The Multivariate Gaussian Distribution

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Definition and Basic Properties
6. The Characteristic Function and Derivation of the Density
7. Geometric Interpretation and the Mahalanobis Distance
8. Affine Transformations
9. Marginalization
10. Conditioning and the Schur Complement
11. The Information Form and Products of Gaussians
12. Maximum Entropy and Differential Entropy
13. Standard Identities and Special Cases
14. Connection to Machine Learning
15. Common Pitfalls and Misconceptions
16. Research Perspective
17. Summary of Takeaways
18. Exercises

---

Learning Objectives

After reading this chapter, you should be able to:

1. State the multivariate Gaussian PDF and verify that it integrates to one using the characteristic function.
2. Interpret the covariance matrix geometrically via ellipsoids and eigendecomposition.
3. Prove that affine transformations preserve Gaussianity and derive the transformed mean and covariance.
4. Derive marginal and conditional distributions by block matrix operations, including the Schur complement formulas.
5. Convert between moment parameterization $(\boldsymbol\mu, \boldsymbol\Sigma)$ and information form $(\boldsymbol\eta, \boldsymbol\Lambda)$.
6. Prove that the Gaussian maximizes differential entropy for fixed mean and covariance.
7. Apply Gaussian identities to Bayesian linear regression, Gaussian processes, VAEs, and diffusion models.

---

Prerequisites

• Bayes' Theorem: posterior computation, conjugate updating.
• Eigenvalues & Eigenvectors: eigendecomposition of symmetric matrices, principal axes.
• Positive Definite Matrices: quadratic forms, Schur complements, matrix inversion lemmas.

---

Notation

• $\mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ — Multivariate Gaussian distribution
• $\boldsymbol{\mu} \in \mathbb{R}^d$ — Mean vector
• $\boldsymbol{\Sigma} \in \mathbb{R}^{n \times n}$ — Symmetric covariance matrix ($n = d$)
• $\det(\boldsymbol{\Sigma}), \boldsymbol{\Sigma}^{-1}$ — Determinant and precision matrix
• $\mathbf{x} \sim \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ — Random vector notation

---

Core Intuition

The multivariate Gaussian (normal) distribution is the central continuous distribution in machine learning. When we assume Gaussian noise, Gaussian priors, or Gaussian approximate posteriors, inference becomes analytic — integrals that would be intractable for general distributions reduce to matrix algebra.

Why is the Gaussian so special? Three reasons:

1. Closure under linear operations. If $\mathbf{x}$ is Gaussian, then any linear transform $\mathbf{A}\mathbf{x} + \mathbf{b}$ is Gaussian. Marginals and conditionals of joint Gaussians are Gaussian.
2. Maximum entropy. Among all distributions with given mean and covariance, the Gaussian is the least informative — it makes no assumptions beyond first and second moments.
3. Central limit theorem. Sums of many independent random variables converge to Gaussian, explaining its ubiquity in natural data and noise.

Every chapter in probabilistic ML — from Bayesian Linear Regression to variational autoencoders to diffusion models — relies on the algebraic properties developed here.

Series context. This is Part II of Chapter 3 in Volume I. It follows Bayes' Theorem and precedes Entropy and KL Divergence, where the closed-form KL between Gaussians is derived.

---

Definition and Basic Properties

Definition 1 (Multivariate Gaussian). A random vector $\mathbf{x} \in \mathbb{R}^d$ follows a multivariate Gaussian distribution, written $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$, if its probability density function is:

$$p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\boldsymbol\Sigma|^{1/2}} \exp\left(-\frac{1}{2}(\mathbf{x} - \boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu)\right), \tag{1}$$

where $\boldsymbol\mu \in \mathbb{R}^d$ is the mean vector and $\boldsymbol\Sigma \in \mathbb{R}^{d \times d}$ is the covariance matrix, assumed symmetric positive definite (SPD).

Definition 2 (Mean and Covariance). For any random vector $\mathbf{x}$ with finite second moments:

$$\boldsymbol\mu = \mathbb{E}[\mathbf{x}], \qquad \boldsymbol\Sigma = \mathbb{E}[(\mathbf{x} - \boldsymbol\mu)(\mathbf{x} - \boldsymbol\mu)^T] = \text{Cov}(\mathbf{x}). \tag{2}$$

Proposition 1. For $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$:

(i) $\mathbb{E}[\mathbf{x}] = \boldsymbol\mu$.

(ii) $\text{Cov}(\mathbf{x}) = \boldsymbol\Sigma$.

(iii) The moment generating function (MGF) is $M_{\mathbf{x}}(\mathbf{t}) = \exp(\mathbf{t}^T \boldsymbol\mu + \frac{1}{2}\mathbf{t}^T \boldsymbol\Sigma \mathbf{t})$.

Proof of (i) and (ii). Differentiate the MGF: $\mathbb{E}[\mathbf{x}] = \nabla_{\mathbf{t}} M_{\mathbf{x}}(\mathbf{t})\big|_{\mathbf{t}=\mathbf{0}} = \boldsymbol\mu$. For covariance, $\text{Cov}(\mathbf{x}) = \nabla^2 M_{\mathbf{x}}(\mathbf{t})\big|_{\mathbf{t}=\mathbf{0}} - \boldsymbol\mu \boldsymbol\mu^T = \boldsymbol\Sigma$. $\blacksquare$

Definition 3 (Precision Matrix). The precision matrix is $\boldsymbol\Lambda = \boldsymbol\Sigma^{-1}$. It quantifies conditional dependencies: large $|\Lambda_{ij}|$ indicates strong partial correlation between components $i$ and $j$.

---

The Characteristic Function and Derivation of the Density

Definition 4 (Characteristic Function). The characteristic function of $\mathbf{x}$ is:

$$\varphi_{\mathbf{x}}(\mathbf{t}) = \mathbb{E}[e^{i \mathbf{t}^T \mathbf{x}}], \quad \mathbf{t} \in \mathbb{R}^d. \tag{3}$$

Theorem 1. The characteristic function of $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ is:

$$\varphi_{\mathbf{x}}(\mathbf{t}) = \exp\left(i \mathbf{t}^T \boldsymbol\mu - \frac{1}{2}\mathbf{t}^T \boldsymbol\Sigma \mathbf{t}\right). \tag{4}$$

Proof. By definition:

$$\varphi_{\mathbf{x}}(\mathbf{t}) = \int_{\mathbb{R}^d} e^{i \mathbf{t}^T \mathbf{x}} p(\mathbf{x})\, d\mathbf{x}. \tag{5}$$

Substitute $\mathbf{y} = \mathbf{x} - \boldsymbol\mu$ and complete the square in the exponent:

$$i\mathbf{t}^T\mathbf{x} - \frac{1}{2}(\mathbf{x}-\boldsymbol\mu)^T\boldsymbol\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu) = -\frac{1}{2}(\mathbf{y} - \boldsymbol\Sigma i\mathbf{t})^T\boldsymbol\Sigma^{-1}(\mathbf{y} - \boldsymbol\Sigma i\mathbf{t}) + i\mathbf{t}^T\boldsymbol\mu + \frac{1}{2}\mathbf{t}^T\boldsymbol\Sigma\mathbf{t}. \tag{6}$$

The integral of the Gaussian kernel over $\mathbf{y}$ yields $(2\pi)^{d/2}|\boldsymbol\Sigma|^{1/2}$, giving (4). $\blacksquare$

Corollary 1 (Normalization). The density (1) integrates to 1.

Proof. Setting $\mathbf{t} = \mathbf{0}$ in (4) gives $\varphi_{\mathbf{x}}(\mathbf{0}) = 1$. By the inversion formula for characteristic functions (Fourier transform), the density integrates to 1. Alternatively, the completing-the-square argument above directly evaluates the integral. $\blacksquare$

Theorem 2 (Uniqueness). A distribution is uniquely determined by its characteristic function. Two random vectors with the same characteristic function (4) are equal in distribution.

This justifies defining the Gaussian via its characteristic function — a common approach in theoretical probability.

---

Geometric Interpretation and the Mahalanobis Distance

Definition 5 (Mahalanobis Distance). The squared Mahalanobis distance of $\mathbf{x}$ from $\boldsymbol\mu$ is:

$$D_M^2(\mathbf{x}, \boldsymbol\mu) = (\mathbf{x} - \boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu). \tag{7}$$

The Gaussian density (1) can be written as:

$$p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\boldsymbol\Sigma|^{1/2}} \exp\left(-\frac{1}{2} D_M^2(\mathbf{x}, \boldsymbol\mu)\right). \tag{8}$$

Proposition 2 (Level Sets). The level sets $\{\mathbf{x} : p(\mathbf{x}) = c\}$ are ellipsoids centered at $\boldsymbol\mu$:

$$\{\mathbf{x} : D_M^2(\mathbf{x}, \boldsymbol\mu) = -2\log(c \cdot (2\pi)^{d/2}|\boldsymbol\Sigma|^{1/2})\}. \tag{9}$$

Theorem 3 (Spectral Decomposition). Let $\boldsymbol\Sigma = \mathbf{Q}\boldsymbol\Lambda\mathbf{Q}^T$ be the eigendecomposition with orthonormal $\mathbf{Q}$ and diagonal $\boldsymbol\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_d)$, $\lambda_i > 0$. Then:

(i) The eigenvectors $\mathbf{q}_i$ (columns of $\mathbf{Q}$) are the principal axes of the ellipsoid.

(ii) The semi-axis lengths in direction $\mathbf{q}_i$ are $\sqrt{\lambda_i}$.

(iii) $|\boldsymbol\Sigma| = \prod_{i=1}^d \lambda_i$ (generalized variance).

Proof. Transform to whitened coordinates $\mathbf{z} = \boldsymbol\Lambda^{-1/2}\mathbf{Q}^T(\mathbf{x} - \boldsymbol\mu)$. Then $D_M^2 = \|\mathbf{z}\|^2$ and level sets are spheres in $\mathbf{z}$-space, which are ellipsoids in $\mathbf{x}$-space with axes aligned to $\mathbf{q}_i$ and lengths $\sqrt{\lambda_i}$. $\blacksquare$

Interpretation. Uncorrelated components ($\boldsymbol\Sigma$ diagonal) yield axis-aligned ellipsoids. Strong correlation rotates the ellipsoid. Small eigenvalues indicate directions of low variance — the distribution is concentrated along those directions.

---

Affine Transformations

Theorem 4 (Affine Transformation). If $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ and $\mathbf{y} = \mathbf{A}\mathbf{x} + \mathbf{b}$ where $\mathbf{A} \in \mathbb{R}^{m \times d}$, $\mathbf{b} \in \mathbb{R}^m$, then:

$$\mathbf{y} \sim \mathcal{N}(\mathbf{A}\boldsymbol\mu + \mathbf{b},\; \mathbf{A}\boldsymbol\Sigma\mathbf{A}^T). \tag{10}$$

Proof (Characteristic Function).

$$\varphi_{\mathbf{y}}(\mathbf{t}) = \mathbb{E}[e^{i\mathbf{t}^T(\mathbf{A}\mathbf{x}+\mathbf{b})}] = e^{i\mathbf{t}^T\mathbf{b}} \mathbb{E}[e^{i(\mathbf{A}^T\mathbf{t})^T\mathbf{x}}] = e^{i\mathbf{t}^T\mathbf{b}} \varphi_{\mathbf{x}}(\mathbf{A}^T\mathbf{t}). \tag{11}$$

Substituting (4) with $\mathbf{A}^T\mathbf{t}$ in place of $\mathbf{t}$:

$$\varphi_{\mathbf{y}}(\mathbf{t}) = \exp\left(i\mathbf{t}^T(\mathbf{A}\boldsymbol\mu + \mathbf{b}) - \frac{1}{2}\mathbf{t}^T(\mathbf{A}\boldsymbol\Sigma\mathbf{A}^T)\mathbf{t}\right), \tag{12}$$

which is the characteristic function of $\mathcal{N}(\mathbf{A}\boldsymbol\mu + \mathbf{b}, \mathbf{A}\boldsymbol\Sigma\mathbf{A}^T)$. By Theorem 2, $\mathbf{y}$ has this distribution. $\blacksquare$

Proof (Direct — Mean and Covariance). $\mathbb{E}[\mathbf{y}] = \mathbf{A}\mathbb{E}[\mathbf{x}] + \mathbf{b} = \mathbf{A}\boldsymbol\mu + \mathbf{b}$.

$$\text{Cov}(\mathbf{y}) = \mathbb{E}[(\mathbf{y} - \mathbb{E}[\mathbf{y}])(\mathbf{y} - \mathbb{E}[\mathbf{y}])^T] = \mathbf{A}\mathbb{E}[(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T]\mathbf{A}^T = \mathbf{A}\boldsymbol\Sigma\mathbf{A}^T. \tag{13}$$

Corollary 2 (Marginalization via Projection). If $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ and $\mathbf{y} = \mathbf{C}\mathbf{x}$ for selection matrix $\mathbf{C} = [\mathbf{I}_k \mid \mathbf{0}]$, then $\mathbf{y} \sim \mathcal{N}(\mathbf{C}\boldsymbol\mu, \mathbf{C}\boldsymbol\Sigma\mathbf{C}^T)$ — the marginal over the first $k$ components.

Definition 6 (Reparameterization). For SPD $\boldsymbol\Sigma$, let $\mathbf{L}$ be the Cholesky factor with $\boldsymbol\Sigma = \mathbf{L}\mathbf{L}^T$. Then:

$$\mathbf{x} = \boldsymbol\mu + \mathbf{L}\boldsymbol\epsilon, \quad \boldsymbol\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_d), \tag{14}$$

is an equivalent sampling representation. This reparameterization trick is fundamental to VAE training.

---

Marginalization

Theorem 5 (Marginal Distribution). Partition $\mathbf{x} = [\mathbf{x}_a^T, \mathbf{x}_b^T]^T$ with:

$$\begin{bmatrix} \mathbf{x}_a \\ \mathbf{x}_b \end{bmatrix} \sim \mathcal{N}\left(\begin{bmatrix} \boldsymbol\mu_a \\ \boldsymbol\mu_b \end{bmatrix}, \begin{bmatrix} \boldsymbol\Sigma_{aa} & \boldsymbol\Sigma_{ab} \\ \boldsymbol\Sigma_{ba} & \boldsymbol\Sigma_{bb} \end{bmatrix}\right). \tag{15}$$

Then the marginal distribution is:

$$\mathbf{x}_a \sim \mathcal{N}(\boldsymbol\mu_a, \boldsymbol\Sigma_{aa}). \tag{16}$$

Proof. The marginal is obtained by integrating out $\mathbf{x}_b$ from the joint density. Equivalently, $\mathbf{x}_a = [\mathbf{I} \mid \mathbf{0}]\mathbf{x}$ is an affine transformation. By Theorem 4 with $\mathbf{A} = [\mathbf{I}_a \mid \mathbf{0}]$:

$$\mathbb{E}[\mathbf{x}_a] = \boldsymbol\mu_a, \quad \text{Cov}(\mathbf{x}_a) = [\mathbf{I} \mid \mathbf{0}] \begin{bmatrix} \boldsymbol\Sigma_{aa} & \boldsymbol\Sigma_{ab} \\ \boldsymbol\Sigma_{ba} & \boldsymbol\Sigma_{bb} \end{bmatrix} \begin{bmatrix} \mathbf{I} \\ \mathbf{0} \end{bmatrix} = \boldsymbol\Sigma_{aa}. \quad \blacksquare \tag{17}$$

Key insight. Marginalization of a Gaussian requires no integration — it is block extraction from the mean and covariance. This property is unique among common multivariate distributions and explains the tractability of Gaussian graphical models.

---

Conditioning and the Schur Complement

Theorem 6 (Conditional Distribution). With the partition (15), the conditional distribution is:

$$\mathbf{x}_a \mid \mathbf{x}_b = \mathbf{x}_b^* \sim \mathcal{N}(\boldsymbol\mu_{a|b}, \boldsymbol\Sigma_{a|b}), \tag{18}$$

where:

$$\boldsymbol\mu_{a|b} = \boldsymbol\mu_a + \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}(\mathbf{x}_b^* - \boldsymbol\mu_b), \tag{19}$$ $$\boldsymbol\Sigma_{a|b} = \boldsymbol\Sigma_{aa} - \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba}. \tag{20}$$

Proof. The conditional density satisfies $p(\mathbf{x}_a \mid \mathbf{x}_b) \propto p(\mathbf{x}_a, \mathbf{x}_b)$ as a function of $\mathbf{x}_a$. The joint exponent is the quadratic form:

$$Q(\mathbf{x}) = (\mathbf{x} - \boldsymbol\mu)^T \boldsymbol\Sigma^{-1}(\mathbf{x} - \boldsymbol\mu). \tag{21}$$

By the block inverse formula (see Positive Definite Matrices):

$$\boldsymbol\Sigma^{-1} = \begin{bmatrix} \boldsymbol\Lambda_{aa} & \boldsymbol\Lambda_{ab} \\ \boldsymbol\Lambda_{ba} & \boldsymbol\Lambda_{bb} \end{bmatrix}, \quad \boldsymbol\Lambda_{aa} = (\boldsymbol\Sigma_{aa} - \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba})^{-1} = \boldsymbol\Sigma_{a|b}^{-1}. \tag{22}$$

Expanding $Q$ and collecting terms in $\mathbf{x}_a$ (with $\mathbf{x}_b = \mathbf{x}_b^*$ fixed):

$$Q = (\mathbf{x}_a - \boldsymbol\mu_{a|b})^T \boldsymbol\Sigma_{a|b}^{-1}(\mathbf{x}_a - \boldsymbol\mu_{a|b}) + \text{const}(\mathbf{x}_b^*). \tag{23}$$

The constant depends only on $\mathbf{x}_b^*$ and cancels when normalizing. The result is a Gaussian in $\mathbf{x}_a$ with mean (19) and covariance (20). $\blacksquare$

Interpretation of (19). Start from the prior mean $\boldsymbol\mu_a$. Add a linear correction proportional to the deviation $(\mathbf{x}_b^* - \boldsymbol\mu_b)$. The matrix $\boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}$ is the optimal linear regression coefficient of $\mathbf{x}_a$ on $\mathbf{x}_b$.

Interpretation of (20). The conditional covariance $\boldsymbol\Sigma_{a|b}$ satisfies $\boldsymbol\Sigma_{a|b} \preceq \boldsymbol\Sigma_{aa}$ (in the Loewner order). Observing $\mathbf{x}_b$ never increases uncertainty about $\mathbf{x}_a$. The reduction $\boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}\boldsymbol\Sigma_{ba}$ is the Schur complement — it measures information gained.

Proposition 3. $\boldsymbol\Sigma_{a|b}$ is symmetric positive definite whenever $\boldsymbol\Sigma$ is SPD and $\boldsymbol\Sigma_{bb}$ is invertible.

---

The Information Form and Products of Gaussians

Definition 7 (Information Form). Parameterize the Gaussian via precision $\boldsymbol\Lambda = \boldsymbol\Sigma^{-1}$ and potential vector $\boldsymbol\eta = \boldsymbol\Lambda\boldsymbol\mu$:

$$p(\mathbf{x}) \propto \exp\left(-\frac{1}{2}\mathbf{x}^T\boldsymbol\Lambda\mathbf{x} + \boldsymbol\eta^T\mathbf{x}\right). \tag{24}$$

Proposition 4 (Conversion). Given $(\boldsymbol\mu, \boldsymbol\Sigma)$:

$$\boldsymbol\Lambda = \boldsymbol\Sigma^{-1}, \quad \boldsymbol\eta = \boldsymbol\Sigma^{-1}\boldsymbol\mu. \tag{25}$$

Given $(\boldsymbol\eta, \boldsymbol\Lambda)$:

$$\boldsymbol\Sigma = \boldsymbol\Lambda^{-1}, \quad \boldsymbol\mu = \boldsymbol\Lambda^{-1}\boldsymbol\eta. \tag{26}$$

Theorem 7 (Product of Gaussians). The product of two Gaussians in information form is proportional to a Gaussian:

$$\mathcal{N}(\mathbf{x}; \boldsymbol\mu_1, \boldsymbol\Sigma_1) \cdot \mathcal{N}(\mathbf{x}; \boldsymbol\mu_2, \boldsymbol\Sigma_2) \propto \mathcal{N}(\mathbf{x}; \boldsymbol\mu, \boldsymbol\Sigma), \tag{27}$$

with:

$$\boldsymbol\Lambda = \boldsymbol\Lambda_1 + \boldsymbol\Lambda_2, \quad \boldsymbol\eta = \boldsymbol\eta_1 + \boldsymbol\eta_2. \tag{28}$$

Proof. In information form, exponentials add: the product exponent is $-\frac{1}{2}\mathbf{x}^T(\boldsymbol\Lambda_1 + \boldsymbol\Lambda_2)\mathbf{x} + (\boldsymbol\eta_1 + \boldsymbol\eta_2)^T\mathbf{x}$, which is a Gaussian with precision $\boldsymbol\Lambda_1 + \boldsymbol\Lambda_2$ and potential $\boldsymbol\eta_1 + \boldsymbol\eta_2$. $\blacksquare$

Corollary 3 (Bayesian Updating). For Gaussian prior $\boldsymbol\phi \sim \mathcal{N}(\mathbf{0}, \boldsymbol\Sigma_p)$ and Gaussian likelihood $\mathbf{y} \mid \boldsymbol\phi \sim \mathcal{N}(\mathbf{X}\boldsymbol\phi, \sigma^2\mathbf{I})$, the posterior precision is:

$$\boldsymbol\Lambda_{\text{post}} = \boldsymbol\Lambda_p + \frac{1}{\sigma^2}\mathbf{X}^T\mathbf{X}, \quad \boldsymbol\eta_{\text{post}} = \boldsymbol\eta_p + \frac{1}{\sigma^2}\mathbf{X}^T\mathbf{y}. \tag{29}$$

Precisions add; this is the algebraic heart of Bayesian Linear Regression.

---

Maximum Entropy and Differential Entropy

Definition 8 (Differential Entropy). For continuous $\mathbf{x}$ with density $p$:

$$h(\mathbf{x}) = -\int p(\mathbf{x}) \log p(\mathbf{x})\, d\mathbf{x}. \tag{30}$$

Theorem 8 (Gaussian Differential Entropy). For $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$:

$$h(\mathbf{x}) = \frac{d}{2}\log(2\pi e) + \frac{1}{2}\log|\boldsymbol\Sigma|. \tag{31}$$

Proof. Substitute (1) into (30). The quadratic form in the exponent integrates via standard Gaussian moment formulas. $\blacksquare$

Theorem 9 (Maximum Entropy). Among all distributions on $\mathbb{R}^d$ with given mean $\boldsymbol\mu$ and covariance $\boldsymbol\Sigma$, the Gaussian $\mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$ maximizes differential entropy.

Proof sketch. Maximize $h(p) = -\int p \log p$ subject to $\int p = 1$, $\int \mathbf{x} p = \boldsymbol\mu$, $\int (\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T p = \boldsymbol\Sigma$ using Lagrange multipliers. The Euler–Lagrange equation yields $\log p(\mathbf{x}) = -1 - \lambda_0 - \boldsymbol\lambda_1^T\mathbf{x} - \text{tr}(\boldsymbol\Lambda_2(\mathbf{x}-\boldsymbol\mu)(\mathbf{x}-\boldsymbol\mu)^T)$, which is the log of a Gaussian. Full derivation in Entropy. $\blacksquare$

ML significance. Choosing a Gaussian prior is the maximum entropy choice given only mean and covariance — the least committal assumption. This justifies Gaussian noise models and Gaussian approximate posteriors in variational inference.

---

Standard Identities and Special Cases

Proposition 5 (Standard Normal). If $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}_d)$ and $\mathbf{x} = \boldsymbol\mu + \mathbf{L}\mathbf{z}$ with $\boldsymbol\Sigma = \mathbf{L}\mathbf{L}^T$, then $\mathbf{x} \sim \mathcal{N}(\boldsymbol\mu, \boldsymbol\Sigma)$.

Proposition 6 (Uncorrelated Implies Independent — Gaussian Only). For jointly Gaussian $(\mathbf{x}_a, \mathbf{x}_b)$, if $\boldsymbol\Sigma_{ab} = \mathbf{0}$, then $\mathbf{x}_a$ and $\mathbf{x}_b$ are independent. This is false for general distributions.

Proposition 7 (Linear Conditional Expectation). For jointly Gaussian variables:

$$\mathbb{E}[\mathbf{x}_a \mid \mathbf{x}_b] = \boldsymbol\mu_a + \boldsymbol\Sigma_{ab}\boldsymbol\Sigma_{bb}^{-1}(\mathbf{x}_b - \boldsymbol\mu_b), \tag{32}$$

which equals the conditional mean (19). For Gaussians, the conditional expectation is linear in $\mathbf{x}_b$ — the best linear predictor is also the best (possibly nonlinear) predictor.

Theorem 10 (KL Divergence Between Gaussians — Preview). For $p = \mathcal{N}(\boldsymbol\mu_1, \boldsymbol\Sigma_1)$ and $q = \mathcal{N}(\boldsymbol\mu_2, \boldsymbol\Sigma_2)$:

$$D_{\text{KL}}(p \| q) = \frac{1}{2}\left[\log\frac{|\boldsymbol\Sigma_2|}{|\boldsymbol\Sigma_1|} - d + \text{tr}(\boldsymbol\Sigma_2^{-1}\boldsymbol\Sigma_1) + (\boldsymbol\mu_2-\boldsymbol\mu_1)^T\boldsymbol\Sigma_2^{-1}(\boldsymbol\mu_2-\boldsymbol\mu_1)\right]. \tag{33}$$

Full derivation in KL Divergence.

---

Connection to Machine Learning

Bayesian Linear Regression

Prior $\boldsymbol\phi \sim \mathcal{N}(\mathbf{0}, \sigma_p^2\mathbf{I})$, likelihood $\mathbf{y} \mid \boldsymbol\phi \sim \mathcal{N}(\mathbf{X}\boldsymbol\phi, \sigma^2\mathbf{I})$. Posterior $\boldsymbol\phi \mid \mathcal{D} \sim \mathcal{N}(\boldsymbol\mu_{\text{post}}, \boldsymbol\Sigma_{\text{post}})$ via (29) or Theorem 6.

Gaussian Processes

A GP defines $f(\mathbf{x}) \sim \mathcal{N}(\mathbf{0}, \mathbf{K})$ over function values at finitely many points. Prediction at new inputs is conditioning (Theorem 6) on observed $(\mathbf{X}, \mathbf{y})$.

Variational Autoencoders

Encoder $q_\phi(\mathbf{z} \mid \mathbf{x}) = \mathcal{N}(\boldsymbol\mu_\phi(\mathbf{x}), \text{diag}(\boldsymbol\sigma_\phi^2(\mathbf{x})))$. Reparameterization (14) enables backpropagation. KL term $D_{\text{KL}}(q \| \mathcal{N}(\mathbf{0}, \mathbf{I}))$ uses (33).

Diffusion Models

Forward process $q(\mathbf{x}_t \mid \mathbf{x}_0) = \mathcal{N}(\sqrt{\bar\alpha_t}\mathbf{x}_0, (1-\bar\alpha_t)\mathbf{I})$ is affine (Theorem 4). Posterior $q(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0)$ is Gaussian via conditioning (Theorem 6). Products of Gaussians (Theorem 7) simplify the variational bound.

---

Common Pitfalls and Misconceptions

Pitfall 1: Confusing uncorrelated with independent. For non-Gaussian distributions, zero covariance does not imply independence. For Gaussians, it does (Proposition 6).

Pitfall 2: Using singular covariance matrices. The density (1) requires $\boldsymbol\Sigma$ to be invertible. Degenerate Gaussians (supported on a subspace) require care — use the characteristic function or restrict to the support subspace.

Pitfall 3: Conditioning on measure-zero events. $P(\mathbf{x}_a \mid \mathbf{x}_b = \mathbf{x}_b^*)$ is defined via densities, not limits of $P(\mathbf{x}_a \mid |\mathbf{x}_b - \mathbf{x}_b^*| < \epsilon)$. The conditional formulas (19)–(20) are the correct definitions.

Pitfall 4: Forgetting that marginalization drops cross-terms. $\boldsymbol\Sigma_{aa}$ alone determines the marginal; off-diagonal blocks $\boldsymbol\Sigma_{ab}$ affect marginals only through their contribution to the joint before integration.

Pitfall 5: Assuming all approximate posteriors are Gaussian. Variational inference often uses Gaussian approximations, but true posteriors in nonlinear models can be multimodal, skewed, or heavy-tailed.

---

Research Perspective

The multivariate Gaussian was studied systematically by Gauss, Laplace, and later multivariate statisticians (Hotelling, Wishart). Its closure properties were formalized in the mid-twentieth century and became the backbone of linear Gaussian state-space models (Kalman, 1960) and Gaussian processes (Rasmussen & Williams, 2006).

In modern deep learning, Gaussians remain central despite the nonlinearity of networks: diffusion models (Ho et al., 2020; Song et al., 2021) build on forward noising and reverse denoising of Gaussian perturbations; VAEs (Kingma & Welling, 2014) use Gaussian latents; and Laplace approximations to neural network posteriors (MacKay, 1992; Daxberger et al., 2021) exploit local Gaussian structure.

Research frontiers include: non-Gaussian diffusion (flow matching, Schrödinger bridges), structured covariances for scalable GPs, and understanding when Gaussian variational approximations fail for language model posteriors.

---

Summary of Takeaways

• Density — (1) with Mahalanobis distance (7) — Noise models, priors
• Affine transform — $\mathbf{Ax}+\mathbf{b} \sim \mathcal{N}(\mathbf{A}\boldsymbol\mu+\mathbf{b}, \mathbf{A}\boldsymbol\Sigma\mathbf{A}^T)$ — Layers, reparameterization
• Marginal — $\mathbf{x}_a \sim \mathcal{N}(\boldsymbol\mu_a, \boldsymbol\Sigma_{aa})$ — GP prediction, feature subsets
• Conditional — (19)–(20) via Schur complement — Bayesian updating, GP conditioning
• Information form — $\boldsymbol\Lambda = \boldsymbol\Lambda_1 + \boldsymbol\Lambda_2$ — Conjugate inference
• Max entropy — (31) maximal for given $(\boldsymbol\mu, \boldsymbol\Sigma)$ — Prior selection
• KL closed form — (33) — VAE regularization

Next article: Entropy & Information Content → — quantifying uncertainty and connecting to cross-entropy loss.

---

Exercises

Exercise 1 (Verification). Verify Proposition 1 (i) and (ii) by direct integration for $d = 1$.

Exercise 2 (Affine transform). If $\mathbf{x} \sim \mathcal{N}(\mathbf{0}, \boldsymbol\Sigma)$ and $\mathbf{y} = \mathbf{A}\mathbf{x}$, show $\text{Cov}(\mathbf{y}) = \mathbf{A}\boldsymbol\Sigma\mathbf{A}^T$ without using characteristic functions.

Exercise 3 (Conditioning). For bivariate Gaussian with correlation $\rho$, derive $X_1 \mid X_2 = x_2$ and verify $\boldsymbol\Sigma_{1|2} = \sigma_1^2(1-\rho^2)$.

Exercise 4 (Schur complement). Prove that $\boldsymbol\Sigma_{a|b} \preceq \boldsymbol\Sigma_{aa}$ using the block inverse formula.

Exercise 5 (Product). Derive (29) for Bayesian linear regression using Theorem 7.

Exercise 6 (Entropy). Compute $h(\mathbf{x})$ for $d = 2$ with $\boldsymbol\Sigma = \text{diag}(\sigma_1^2, \sigma_2^2)$.

Exercise 7 (Conceptual). Explain why Gaussian processes perform exact Bayesian inference without MCMC.

Exercise 8 (Proof). Complete the completing-the-square step in the proof of Theorem 6, explicitly identifying $\boldsymbol\mu_{a|b}$.