Bayesian Linear Regression

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. From MLE to MAP to Full Bayesian Inference
6. The Conjugate Gaussian Model
7. Posterior Derivation via Gaussian Identities
8. The Predictive Distribution
9. Epistemic and Aleatoric Uncertainty
10. Sequential Bayesian Updating
11. Connection to Ridge Regression
12. Computational Identities: Woodbury and Dual Form
13. Connection to Gaussian Processes
14. Worked Examples
15. Connection to the Broader Curriculum
16. Common Pitfalls and Misconceptions
17. Research Perspective
18. Summary of Takeaways
19. Exercises

---

Learning Objectives

After reading this chapter, you should be able to:

1. Distinguish MLE, MAP, and full Bayesian inference for linear regression and explain what each provides.
2. Specify a conjugate Gaussian prior–likelihood pair and derive the posterior $\Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y})$ in closed form.
3. Compute the posterior predictive distribution $\Pr(y^* \mid \mathbf{x}^*, \mathbf{X}, \mathbf{y})$ and decompose its variance into epistemic and aleatoric components.
4. Prove that MAP estimation with a Gaussian prior is equivalent to Ridge regression.
5. Perform sequential Bayesian updating via precision-matrix rank-1 additions.
6. Apply the Woodbury identity to convert between primal ($D \times D$) and dual ($N \times N$) formulations.
7. Recognize the Gaussian process prediction formula as the kernel dual of Bayesian linear regression.

---

Prerequisites

This chapter assumes familiarity with:

• Linear Regression — OLS, normal equations, Ridge penalty
• Bayes' Theorem — posterior $\propto$ likelihood $\times$ prior
• Multivariate Gaussian Distribution — density, conditioning, products
• Maximum Likelihood Estimation — likelihood maximization

---

Notation

• $\mathbf{w}$ — Regression weights (random vector)
• $\mathbf{m}_N, \mathbf{S}_N$ — Posterior mean and covariance
• $\mathbf{m}_0, \mathbf{S}_0$ — Prior mean and covariance
• $\alpha, \beta$ — Prior precision and noise precision
• $p(\mathbf{w} \mid \mathcal{D})$ — Posterior over weights

---

Core Intuition

Classical linear regression returns a point estimate $\hat{\boldsymbol{\theta}}$ — a single best guess for the parameters. Bayesian linear regression returns a distribution over parameters, encoding which settings are plausible given the data and prior beliefs.

Why does this matter?

1. Uncertainty quantification. Predictions come with confidence intervals that widen when extrapolating beyond training data.
2. Regularization with meaning. A prior $\Pr(\boldsymbol{\theta})$ is not an ad hoc penalty — it encodes genuine beliefs about parameter magnitude before seeing data.
3. Sequential learning. New data updates the posterior directly; the old posterior becomes the new prior.
4. Foundation for advanced methods. Variational inference, Gaussian processes, and Bayesian neural networks all extend the framework developed here.

For linear regression with Gaussian noise and a Gaussian prior, the posterior is exactly Gaussian — no approximation needed. This conjugacy makes Bayesian linear regression one of the few models where full Bayesian inference is analytically tractable.

Series context. This opens Chapter 5 (Bayesian Machine Learning) in Volume II. It is followed by Variational Inference, which handles models where the posterior is not conjugate.

---

From MLE to MAP to Full Bayesian Inference

Definition 1 (Likelihood). Given the Gaussian noise model $y_i = \boldsymbol{\theta}^T \tilde{\mathbf{x}}_i + \varepsilon_i$ with $\varepsilon_i \sim \mathcal{N}(0, \sigma_n^2)$, the likelihood of parameters $\boldsymbol{\theta}$ is

$$\Pr(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta}) = \mathcal{N}(\mathbf{y} \mid \mathbf{X}\boldsymbol{\theta}, \sigma_n^2 \mathbf{I}_N). \tag{1}$$

We use the standard design matrix convention $\mathbf{X} \in \mathbb{R}^{N \times (d+1)}$ with $\hat{y}_i = \boldsymbol{\theta}^T \tilde{\mathbf{x}}_i$, so $\mathbf{y} \sim \mathcal{N}(\mathbf{X}\boldsymbol{\theta}, \sigma_n^2 \mathbf{I})$.

Definition 2 (Three Inference Paradigms).

• MLE — $\hat{\boldsymbol{\theta}}_{\text{ML}} = \arg\max_{\boldsymbol{\theta}} \Pr(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta})$ — Point estimate
• MAP — $\hat{\boldsymbol{\theta}}_{\text{MAP}} = \arg\max_{\boldsymbol{\theta}} \Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y})$ — Point estimate (posterior mode)
• Bayesian — Compute $\Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y})$ — Full posterior distribution

Proposition 1 (MLE = OLS). The MLE is the ordinary least squares solution:

$$\hat{\boldsymbol{\theta}}_{\text{ML}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}. \tag{2}$$

Proof. Maximizing the Gaussian log-likelihood is equivalent to minimizing $\|\mathbf{X}\boldsymbol{\theta} - \mathbf{y}\|^2$. See Linear Regression. $\blacksquare$

Proposition 2 (Bayes' Rule for Parameters).

$$\Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) = \frac{\Pr(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta})\, \Pr(\boldsymbol{\theta})}{\Pr(\mathbf{y} \mid \mathbf{X})}. \tag{3}$$

The marginal likelihood $\Pr(\mathbf{y} \mid \mathbf{X}) = \int \Pr(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta})\Pr(\boldsymbol{\theta})\, d\boldsymbol{\theta}$ is the evidence. For conjugate models, it is analytically computable.

---

The Conjugate Gaussian Model

Definition 3 (Gaussian Prior). We place an isotropic Gaussian prior on parameters:

$$\Pr(\boldsymbol{\theta}) = \mathcal{N}(\boldsymbol{\theta} \mid \mathbf{0}, \sigma_p^2 \mathbf{I}_{d+1}). \tag{4}$$

The prior mean $\mathbf{0}$ reflects a belief that parameters should be small. The prior variance $\sigma_p^2$ controls strength: large $\sigma_p^2$ is a weak (vague) prior; small $\sigma_p^2$ strongly shrinks parameters toward zero.

Definition 4 (Conjugacy). A prior is conjugate to a likelihood if the posterior belongs to the same parametric family as the prior. The Gaussian prior is conjugate to the Gaussian likelihood — the posterior is Gaussian.

Theorem 1 (Posterior Distribution). Under the Gaussian likelihood (1) and Gaussian prior (4), the posterior is

$$\Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) = \mathcal{N}(\boldsymbol{\theta} \mid \boldsymbol{\mu}_{\text{post}}, \boldsymbol{\Sigma}_{\text{post}}), \tag{5}$$

with

$$\boldsymbol{\Sigma}_{\text{post}}^{-1} = \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{X} + \frac{1}{\sigma_p^2}\mathbf{I}, \tag{6}$$ $$\boldsymbol{\mu}_{\text{post}} = \boldsymbol{\Sigma}_{\text{post}} \cdot \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{y}. \tag{7}$$

Equivalently, in closed form:

$$\boldsymbol{\mu}_{\text{post}} = \left(\mathbf{X}^T\mathbf{X} + \frac{\sigma_n^2}{\sigma_p^2}\mathbf{I}\right)^{-1}\mathbf{X}^T\mathbf{y}, \quad \boldsymbol{\Sigma}_{\text{post}} = \sigma_n^2\left(\mathbf{X}^T\mathbf{X} + \frac{\sigma_n^2}{\sigma_p^2}\mathbf{I}\right)^{-1}. \tag{8}$$

---

Posterior Derivation via Gaussian Identities

We prove Theorem 1 using two fundamental identities for multivariate Gaussians.

Lemma 1 (Likelihood as Gaussian in $\boldsymbol{\theta}$). Rewriting the likelihood as a function of $\boldsymbol{\theta}$:

$$\mathcal{N}(\mathbf{y} \mid \mathbf{X}\boldsymbol{\theta}, \sigma_n^2 \mathbf{I}) \propto \mathcal{N}(\boldsymbol{\theta} \mid \boldsymbol{\mu}_\ell, \boldsymbol{\Sigma}_\ell), \tag{9}$$

where $\boldsymbol{\Sigma}_\ell^{-1} = \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{X}$ and $\boldsymbol{\mu}_\ell = \boldsymbol{\Sigma}_\ell \cdot \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{y}$.

Proof. Complete the square in $\boldsymbol{\theta}$:

$$\|\mathbf{y} - \mathbf{X}\boldsymbol{\theta}\|^2 = (\boldsymbol{\theta} - \boldsymbol{\mu}_\ell)^T \boldsymbol{\Sigma}_\ell^{-1} (\boldsymbol{\theta} - \boldsymbol{\mu}_\ell) + \text{const}(\mathbf{y}). \tag{10}$$

The exponent of the Gaussian density in $\boldsymbol{\theta}$ is $-\frac{1}{2}(\boldsymbol{\theta} - \boldsymbol{\mu}_\ell)^T \boldsymbol{\Sigma}_\ell^{-1}(\boldsymbol{\theta} - \boldsymbol{\mu}_\ell)$. $\blacksquare$

Lemma 2 (Product of Gaussians). For two Gaussians in the same variable:

$$\mathcal{N}(\boldsymbol{\theta} \mid \mathbf{a}, \mathbf{A}) \cdot \mathcal{N}(\boldsymbol{\theta} \mid \mathbf{b}, \mathbf{B}) \propto \mathcal{N}(\boldsymbol{\theta} \mid \mathbf{c}, \mathbf{C}), \tag{11}$$

where

$$\mathbf{C}^{-1} = \mathbf{A}^{-1} + \mathbf{B}^{-1}, \quad \mathbf{c} = \mathbf{C}(\mathbf{A}^{-1}\mathbf{a} + \mathbf{B}^{-1}\mathbf{b}). \tag{12}$$

Proof. Expanding both exponents and collecting quadratic and linear terms in $\boldsymbol{\theta}$:

$$-\frac{1}{2}\boldsymbol{\theta}^T(\mathbf{A}^{-1} + \mathbf{B}^{-1})\boldsymbol{\theta} + \boldsymbol{\theta}^T(\mathbf{A}^{-1}\mathbf{a} + \mathbf{B}^{-1}\mathbf{b}) + \text{const}. \tag{13}$$

Completing the square yields (12). $\blacksquare$

Proof of Theorem 1. The posterior is proportional to likelihood $\times$ prior:

$$\Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) \propto \mathcal{N}(\boldsymbol{\theta} \mid \boldsymbol{\mu}_\ell, \boldsymbol{\Sigma}_\ell) \cdot \mathcal{N}(\boldsymbol{\theta} \mid \mathbf{0}, \sigma_p^2 \mathbf{I}). \tag{14}$$

Applying Lemma 2 with $\mathbf{A} = \boldsymbol{\Sigma}_\ell$, $\mathbf{a} = \boldsymbol{\mu}_\ell$, $\mathbf{B} = \sigma_p^2 \mathbf{I}$, $\mathbf{b} = \mathbf{0}$:

$$\boldsymbol{\Sigma}_{\text{post}}^{-1} = \boldsymbol{\Sigma}_\ell^{-1} + \frac{1}{\sigma_p^2}\mathbf{I} = \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{X} + \frac{1}{\sigma_p^2}\mathbf{I}, \tag{15}$$ $$\boldsymbol{\mu}_{\text{post}} = \boldsymbol{\Sigma}_{\text{post}} \left(\boldsymbol{\Sigma}_\ell^{-1}\boldsymbol{\mu}_\ell\right) = \boldsymbol{\Sigma}_{\text{post}} \cdot \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{y}. \tag{16}$$

Normalizing gives the full posterior (5). $\blacksquare$

Important equation. Equation (8) shows that the posterior mean equals the Ridge regression solution with $\lambda = \sigma_n^2/\sigma_p^2$. Bayesian inference and regularized frequentist estimation coincide for point predictions — but the Bayesian framework additionally provides $\boldsymbol{\Sigma}_{\text{post}}$ for uncertainty.

---

The Predictive Distribution

Definition 5 (Posterior Predictive). For a new input $\mathbf{x}^*$, the posterior predictive distribution is

$$\Pr(y^* \mid \mathbf{x}^*, \mathbf{X}, \mathbf{y}) = \int \Pr(y^* \mid \mathbf{x}^*, \boldsymbol{\theta})\, \Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y})\, d\boldsymbol{\theta}. \tag{17}$$

Theorem 2 (Predictive Distribution). Under the conjugate Gaussian model:

$$\Pr(y^* \mid \mathbf{x}^*, \mathbf{X}, \mathbf{y}) = \mathcal{N}\left(y^* \;\Big|\; \boldsymbol{\mu}_{\text{post}}^T \tilde{\mathbf{x}}^*,\;\; \tilde{\mathbf{x}}^{*T}\boldsymbol{\Sigma}_{\text{post}}\tilde{\mathbf{x}}^* + \sigma_n^2\right). \tag{18}$$

Proof. Both factors in (17) are Gaussian:

$$\Pr(y^* \mid \mathbf{x}^*, \boldsymbol{\theta}) = \mathcal{N}(y^* \mid \boldsymbol{\theta}^T \tilde{\mathbf{x}}^*, \sigma_n^2), \quad \Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) = \mathcal{N}(\boldsymbol{\theta} \mid \boldsymbol{\mu}_{\text{post}}, \boldsymbol{\Sigma}_{\text{post}}). \tag{19}$$

The integral is a convolution of Gaussians. The predictive mean is $\mathbb{E}[\boldsymbol{\theta}^T \tilde{\mathbf{x}}^*] = \boldsymbol{\mu}_{\text{post}}^T \tilde{\mathbf{x}}^*$. The predictive variance is:

$$\text{Var}(y^*) = \underbrace{\tilde{\mathbf{x}}^{*T}\boldsymbol{\Sigma}_{\text{post}}\tilde{\mathbf{x}}^*}_{\text{epistemic}} + \underbrace{\sigma_n^2}_{\text{aleatoric}}. \tag{20}$$

The epistemic term arises from uncertainty in $\boldsymbol{\theta}$; the aleatoric term is irreducible observation noise. $\blacksquare$

---

Epistemic and Aleatoric Uncertainty

Definition 6 (Epistemic Uncertainty). Epistemic (model) uncertainty reflects incomplete knowledge about $\boldsymbol{\theta}$. It is reducible with more data — as $N \to \infty$, $\boldsymbol{\Sigma}_{\text{post}} \to \mathbf{0}$ and epistemic uncertainty vanishes.

Definition 7 (Aleatoric Uncertainty). Aleatoric (data) uncertainty reflects inherent noise in observations. It is irreducible regardless of sample size.

Proposition 3 (Extrapolation Behavior). As $\|\tilde{\mathbf{x}}^*\| \to \infty$ in directions poorly covered by training data, $\tilde{\mathbf{x}}^{*T}\boldsymbol{\Sigma}_{\text{post}}\tilde{\mathbf{x}}^* \to \infty$ and predictive uncertainty grows without bound.

Proof sketch. $\boldsymbol{\Sigma}_{\text{post}}$ is positive definite (for any $\lambda > 0$). For $\tilde{\mathbf{x}}^*$ orthogonal to the column space of $\mathbf{X}$, $\tilde{\mathbf{x}}^{*T}(\mathbf{X}^T\mathbf{X})\tilde{\mathbf{x}}^* = 0$, so the prior term dominates and $\tilde{\mathbf{x}}^{*T}\boldsymbol{\Sigma}_{\text{post}}\tilde{\mathbf{x}}^* \approx \sigma_p^2 \|\tilde{\mathbf{x}}^*\|^2$. $\blacksquare$

This is a principled mechanism for knowing what you don't know — something point estimates cannot provide.

---

Sequential Bayesian Updating

Theorem 3 (Online Update). After observing a new data point $(\tilde{\mathbf{x}}_{\text{new}}, y_{\text{new}})$, the updated posterior precision is

$$\boldsymbol{\Sigma}_{\text{new}}^{-1} = \boldsymbol{\Sigma}_{\text{old}}^{-1} + \frac{1}{\sigma_n^2}\tilde{\mathbf{x}}_{\text{new}}\tilde{\mathbf{x}}_{\text{new}}^T. \tag{21}$$

The updated mean satisfies

$$\boldsymbol{\mu}_{\text{new}} = \boldsymbol{\Sigma}_{\text{new}}\left(\boldsymbol{\Sigma}_{\text{old}}^{-1}\boldsymbol{\mu}_{\text{old}} + \frac{1}{\sigma_n^2}\tilde{\mathbf{x}}_{\text{new}}\, y_{\text{new}}\right). \tag{22}$$

Proof. The old posterior becomes the new prior. Applying Lemma 2 with the new likelihood (rank-1 update to precision) yields (21)–(22). $\blacksquare$

Each observation adds a rank-1 term to the precision matrix — the posterior concentrates incrementally. This is the Bayesian analog of recursive least squares.

---

Connection to Ridge Regression

Theorem 4 (MAP–Ridge Equivalence). The MAP estimator under prior (4) is

$$\hat{\boldsymbol{\theta}}_{\text{MAP}} = \arg\max_{\boldsymbol{\theta}} \Pr(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) = \left(\mathbf{X}^T\mathbf{X} + \frac{\sigma_n^2}{\sigma_p^2}\mathbf{I}\right)^{-1}\mathbf{X}^T\mathbf{y}. \tag{23}$$

This is Ridge regression with regularization parameter $\lambda = \sigma_n^2/\sigma_p^2$.

Proof. The MAP maximizes $\log \Pr(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta}) + \log \Pr(\boldsymbol{\theta})$, equivalent to minimizing:

$$\frac{1}{2\sigma_n^2}\|\mathbf{X}\boldsymbol{\theta} - \mathbf{y}\|^2 + \frac{1}{2\sigma_p^2}\|\boldsymbol{\theta}\|^2. \tag{24}$$

Setting the gradient to zero gives (23). $\blacksquare$

Interpretation. $\lambda = \sigma_n^2/\sigma_p^2$ is the ratio of observation noise to prior uncertainty. Noisy data ($\sigma_n^2$ large) $\implies$ weak regularization. Strong prior ($\sigma_p^2$ small) $\implies$ strong shrinkage.

---

Computational Identities: Woodbury and Dual Form

When $N \ll d$ (few samples, many features), inverting $\mathbf{X}^T\mathbf{X}$ is expensive. The Woodbury matrix identity converts between primal and dual forms.

Lemma 3 (Woodbury Identity). For conformable matrices:

$$(\mathbf{A}^{-1} + \mathbf{B}^T\mathbf{C}^{-1}\mathbf{B})^{-1}\mathbf{B}^T\mathbf{C}^{-1} = \mathbf{A}\mathbf{B}^T(\mathbf{B}\mathbf{A}\mathbf{B}^T + \mathbf{C})^{-1}. \tag{25}$$

Proposition 4 (Dual Predictive Mean). Applying (25) with $\mathbf{A} = \sigma_p^2 \mathbf{I}$, $\mathbf{B} = \mathbf{X}$, $\mathbf{C} = \sigma_n^2 \mathbf{I}$:

$$\tilde{\mathbf{x}}^{*T}\boldsymbol{\mu}_{\text{post}} = \sigma_p^2 \tilde{\mathbf{x}}^{*T}\mathbf{X}\left(\sigma_p^2 \mathbf{X}^T\mathbf{X} + \sigma_n^2 \mathbf{I}_N\right)^{-1}\mathbf{y}. \tag{26}$$

The inversion is now $N \times N$ instead of $(d+1) \times (d+1)$ — efficient when $N \ll d$.

---

Connection to Gaussian Processes

Define the linear kernel $k(\mathbf{x}, \mathbf{x}') = \sigma_p^2 \tilde{\mathbf{x}}^T \tilde{\mathbf{x}}'$. The dual predictive distribution (26) becomes:

$$\mathbb{E}[y^* \mid \mathbf{x}^*] = \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I}_N)^{-1} \mathbf{y}, \tag{27}$$ $$\text{Var}(y^* \mid \mathbf{x}^*) = k(\mathbf{x}^*, \mathbf{x}^*) - \mathbf{k}_*^T (\mathbf{K} + \sigma_n^2 \mathbf{I}_N)^{-1} \mathbf{k}_* + \sigma_n^2, \tag{28}$$

where $\mathbf{K}_{ij} = k(\tilde{\mathbf{x}}_i, \tilde{\mathbf{x}}_j)$ and $\mathbf{k}_* = [k(\tilde{\mathbf{x}}_1, \tilde{\mathbf{x}}^*), \ldots, k(\tilde{\mathbf{x}}_N, \tilde{\mathbf{x}}^*)]^T$.

This is the Gaussian process prediction formula with a linear kernel. Replacing the kernel with a nonlinear function (e.g., RBF) yields nonlinear Bayesian regression without explicit feature engineering — the function-space view of the same mathematics.

---

Worked Examples

Example 1: Single-Feature Model

Prior: $\sigma_p^2 = 1$, $\sigma_n^2 = 0.5$. One observation: $\tilde{\mathbf{x}} = (1, 2)^T$, $y = 3$.

$$\boldsymbol{\Sigma}_{\text{post}}^{-1} = \frac{1}{0.5}\begin{bmatrix}1\\2\end{bmatrix}\begin{bmatrix}1 & 2\end{bmatrix} + \mathbf{I} = \begin{bmatrix}3 & 4\\4 & 9\end{bmatrix}, \quad \boldsymbol{\mu}_{\text{post}} = \begin{bmatrix}3\\9\end{bmatrix}^{-1}\begin{bmatrix}6\\12\end{bmatrix}. \tag{29}$$

The posterior is tighter than the prior — data has reduced uncertainty.

Example 2: Predictive Uncertainty Near vs. Far from Data

For training points clustered near $\tilde{\mathbf{x}} = (1, 0)$, prediction at $\tilde{\mathbf{x}}^* = (1, 0)$ has small epistemic variance. At $\tilde{\mathbf{x}}^* = (1, 100)$, epistemic variance is large — we are extrapolating.

Example 3: Limit of Flat Prior

As $\sigma_p^2 \to \infty$: $\boldsymbol{\Sigma}_{\text{post}}^{-1} \to \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{X}$, and $\boldsymbol{\mu}_{\text{post}} \to \hat{\boldsymbol{\theta}}_{\text{ML}}$ (when $\mathbf{X}^T\mathbf{X}$ is invertible). The Bayesian posterior recovers the frequentist MLE.

---

Connection to the Broader Curriculum

• Gaussian posterior — Variational Inference for non-conjugate models
• Predictive uncertainty — Calibration in Logistic Regression and deep learning
• Kernel dual form — Gaussian processes, RBF kernels
• Sequential updating — Online learning, streaming data
• Prior = regularizer — Weight decay in Backpropagation

Variational Autoencoders apply variational inference to latent variable models where the posterior over latents is approximated rather than computed exactly.

---

Common Pitfalls and Misconceptions

Pitfall 1: Confusing posterior variance with prediction interval width. The posterior $\Pr(\boldsymbol{\theta} \mid \mathcal{D})$ describes uncertainty about parameters. Predictive intervals for $y^*$ include both epistemic and aleatoric terms (20).

Pitfall 2: Using an improper prior without care. The limit $\sigma_p^2 \to \infty$ yields an improper (non-normalizable) prior. Predictions may be valid, but the posterior may not exist when $N \leq d$.

Pitfall 3: Assuming Gaussian prior is always appropriate. Heavy-tailed priors (Student-$t$) provide robustness to outliers. The Gaussian prior penalizes large coefficients quadratically.

Pitfall 4: Ignoring hyperparameter selection. $\sigma_p^2$ and $\sigma_n^2$ are hyperparameters. Empirical Bayes or cross-validation selects them; treating them as fixed without justification is a form of misspecification.

Pitfall 5: Conflating MAP with full Bayesian inference. MAP gives a point estimate (posterior mode). It ignores posterior covariance and can be sensitive to reparameterization — unlike the full predictive distribution (18).

---

Research Perspective

Bayesian linear models date to Laplace and Gauss. The modern formulation through conjugate analysis connects to empirical Bayes (Robbins, 1955; Efron & Morris, 1973) and hierarchical modeling.

In machine learning:

• Gaussian Processes (Rasmussen & Williams, 2006) generalize Bayesian linear regression to infinite-dimensional function spaces via kernels.
• Bayesian deep learning attempts to place priors on neural network weights — but the posterior is intractable, motivating Variational Inference.
• Uncertainty quantification in safety-critical applications (medical diagnosis, autonomous driving) relies on principled epistemic uncertainty from Bayesian methods.
• Last-layer Bayesian inference — treating only the final layer as Bayesian while keeping feature extractors deterministic — provides efficient uncertainty at scale.

The conjugate Gaussian case remains the gold standard: when it applies, use it exactly rather than approximating.

---

Summary of Takeaways

• Posterior precision — $\boldsymbol{\Sigma}_{\text{post}}^{-1} = \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{X} + \frac{1}{\sigma_p^2}\mathbf{I}$ — Data + prior information
• Posterior mean — $\boldsymbol{\mu}_{\text{post}} = \boldsymbol{\Sigma}_{\text{post}} \frac{1}{\sigma_n^2}\mathbf{X}^T\mathbf{y}$ — = Ridge estimate
• Predictive variance — $\tilde{\mathbf{x}}^{*T}\boldsymbol{\Sigma}_{\text{post}}\tilde{\mathbf{x}}^* + \sigma_n^2$ — Epistemic + aleatoric
• Sequential update — $\boldsymbol{\Sigma}_{\text{new}}^{-1} = \boldsymbol{\Sigma}_{\text{old}}^{-1} + \frac{1}{\sigma_n^2}\tilde{\mathbf{x}}\tilde{\mathbf{x}}^T$ — Online learning
• MAP = Ridge — $\lambda = \sigma_n^2/\sigma_p^2$ — Bayesian interpretation of L2

Next article: Variational Inference → — for models where the posterior is not conjugate and must be approximated.

---

Exercises

Exercise 1 (Posterior derivation). Verify Theorem 1 by completing the square directly in $\boldsymbol{\theta}$ without using Lemma 2.

Exercise 2 (Predictive distribution). Derive (18) by evaluating the integral (17) using the formula for the convolution of two Gaussians.

Exercise 3 (Sequential update). Starting from prior $\mathcal{N}(\mathbf{0}, \sigma_p^2 \mathbf{I})$, derive the posterior after observing two data points sequentially. Show the result equals the posterior from observing both simultaneously.

Exercise 4 (MAP limit). Prove that as $\sigma_p^2 \to \infty$, $\boldsymbol{\mu}_{\text{post}} \to (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$ (when invertible).

Exercise 5 (Uncertainty decomposition). For a given dataset, compute the fraction of predictive variance that is epistemic vs. aleatoric at a test point near the training data centroid vs. far from it.

Exercise 6 (Woodbury). Derive (26) by applying the Woodbury identity to $\boldsymbol{\mu}_{\text{post}}$.

Exercise 7 (Kernel form). Show that the GP formula (27)–(28) with linear kernel reproduces the Bayesian linear regression predictive distribution.

Exercise 8 (Conceptual). Explain why the posterior covariance $\boldsymbol{\Sigma}_{\text{post}}$ does not depend on $\mathbf{y}$ — only on $\mathbf{X}$, $\sigma_n^2$, and $\sigma_p^2$. What does this imply about uncertainty before seeing targets?