Bayes' Theorem

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Probability Spaces and the Kolmogorov Axioms
6. Conditional Probability: Definition and Properties
7. The Product Rule and Bayes' Theorem
8. The Law of Total Probability
9. Bayes' Theorem in Continuous Settings
10. Sequential Bayesian Updating
11. The Bayesian Inference Framework
12. Worked Examples
13. Connection to Machine Learning
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

---

Learning Objectives

After reading this chapter, you should be able to:

1. State the Kolmogorov axioms and verify that conditional probability is a valid probability measure on restricted sample spaces.
2. Derive the product rule and Bayes' theorem from the definition of conditional probability, with no logical gaps.
3. Apply the law of total probability to compute evidence terms $P(B)$ when only conditional probabilities are known.
4. Write Bayes' theorem for continuous random variables using densities and explain when ratios of densities are meaningful.
5. Perform sequential Bayesian updating and explain why the posterior from one observation becomes the prior for the next.
6. Identify the roles of prior, likelihood, posterior, and evidence in Bayesian inference and connect them to Naive Bayes, Bayesian neural networks, and preference modeling.

---

Prerequisites

This is the opening chapter of Volume I, Chapter 3 (Probability & Statistical Learning). No prior chapters in this volume are required. Familiarity with basic set notation and elementary calculus (for the continuous case) is assumed.

Recommended background from earlier volumes:

• Vectors, Spans & Linear Independence: expectation as a linear operator appears throughout.
• Gradient Descent: optimization of log-posteriors connects to MAP estimation in later chapters.

---

Notation

• $(\Omega, \mathcal{F}, P)$ — Probability space
• $P(A \mid B)$ — Conditional probability
• $P(A, B)$ — Joint probability
• $\pi(\theta), p(\mathcal{D} \mid \theta)$ — Prior and likelihood
• $p(\theta \mid \mathcal{D})$ — Posterior
• $p(x), p(x \mid y)$ — Densities in continuous settings

---

Core Intuition

Machine learning is, at its core, learning under uncertainty. We observe data $\mathcal{D}$, hold beliefs about unknown quantities $\theta$ (parameters, latent variables, or hypotheses), and wish to update those beliefs in light of evidence. Bayes' theorem is the precise mathematical rule governing this update.

The central question is: Given that we observed evidence $B$, how should we revise our belief about hypothesis $A$? Before seeing $B$, our belief is captured by the prior $P(A)$. After seeing $B$, it becomes the posterior $P(A \mid B)$. Bayes' theorem tells us exactly how to compute this revision using the likelihood $P(B \mid A)$ — how probable the evidence is under each hypothesis — and the evidence (marginal) $P(B)$, which normalizes the result.

This is not a heuristic or approximation. Bayes' theorem is a theorem — a logical consequence of the definition of conditional probability. Every probabilistic learning algorithm, from Naive Bayes classifiers to variational autoencoders to RLHF reward models, ultimately relies on this identity or an approximation to it.

Series context. This is Part I of Chapter 3 in Volume I: Foundations. Subsequent parts cover the Multivariate Gaussian, Entropy, KL Divergence, Maximum Likelihood Estimation, and the Bias–Variance Tradeoff.

---

Probability Spaces and the Kolmogorov Axioms

Definition 1 (Probability Space). A probability space is a triple $(\Omega, \mathcal{F}, P)$ where:

• $\Omega$ is the sample space — the set of all possible outcomes.
• $\mathcal{F} \subseteq 2^\Omega$ is a $\sigma$-algebra of events (subsets of $\Omega$ closed under complement and countable union).
• $P: \mathcal{F} \to [0, 1]$ is a probability measure satisfying the Kolmogorov axioms.

Axiom 1 (Non-negativity). For every event $A \in \mathcal{F}$:

$$P(A) \geq 0. \tag{1}$$

Axiom 2 (Normalization). The entire sample space has probability one:

$$P(\Omega) = 1. \tag{2}$$

Axiom 3 (Countable Additivity). For any countable collection of mutually disjoint events $\{A_i\}_{i=1}^\infty$ with $A_i \cap A_j = \emptyset$ for $i \neq j$:

$$P\left(\bigcup_{i=1}^\infty A_i\right) = \sum_{i=1}^\infty P(A_i). \tag{3}$$

Definition 2 (Random Variable). A random variable $X$ is a measurable function $X: \Omega \to \mathbb{R}$ (or $\mathbb{R}^d$). For discrete $X$, the probability mass function (PMF) is $p(x) = P(X = x)$. For continuous $X$, the probability density function (PDF) $f(x)$ satisfies $P(X \in B) = \int_B f(x)\, dx$.

Proposition 1. From the axioms, the following hold:

(i) $P(\emptyset) = 0$.

(ii) $P(A^c) = 1 - P(A)$ where $A^c = \Omega \setminus A$.

(iii) If $A \subseteq B$, then $P(A) \leq P(B)$.

(iv) $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ (inclusion–exclusion).

Proof of (i). Let $A_1 = \emptyset$ and $A_i = \emptyset$ for $i \geq 2$. Then $\bigcup_i A_i = \emptyset$ and by Axiom 3, $P(\emptyset) = \sum_{i=1}^\infty P(\emptyset)$. The only non-negative solution is $P(\emptyset) = 0$. $\blacksquare$

These elementary consequences will be used repeatedly in what follows.

---

Conditional Probability: Definition and Properties

Definition 3 (Conditional Probability). Let $A, B \in \mathcal{F}$ with $P(B) > 0$. The conditional probability of $A$ given $B$ is:

$$P(A \mid B) = \frac{P(A \cap B)}{P(B)}. \tag{4}$$

Interpretation. Restrict attention to outcomes in $B$. Among these, what fraction also belong to $A$? The sample space is effectively rescaled so that $B$ has probability 1.

Theorem 1. Fix $B$ with $P(B) > 0$. The map $A \mapsto P(A \mid B)$ defines a valid probability measure on $(\Omega, \mathcal{F})$.

Proof. We verify the Kolmogorov axioms for $P(\cdot \mid B)$.

(Non-negativity.) $P(A \mid B) = P(A \cap B)/P(B) \geq 0$ since $P(A \cap B) \geq 0$ and $P(B) > 0$.

(Normalization.) $P(\Omega \mid B) = P(\Omega \cap B)/P(B) = P(B)/P(B) = 1$.

(Countable additivity.) Let $\{A_i\}$ be disjoint. Then $\{A_i \cap B\}$ are also disjoint, and

$$P\left(\bigcup_i A_i \;\middle|\; B\right) = \frac{P\left(\left(\bigcup_i A_i\right) \cap B\right)}{P(B)} = \frac{P\left(\bigcup_i (A_i \cap B)\right)}{P(B)} = \frac{\sum_i P(A_i \cap B)}{P(B)} = \sum_i P(A_i \mid B). \tag{5}$$

Therefore $P(\cdot \mid B)$ is a probability measure. $\blacksquare$

Proposition 2 (Conditional Probability Rules). For events with $P(B) > 0$:

(i) $P(\emptyset \mid B) = 0$.

(ii) $P(A^c \mid B) = 1 - P(A \mid B)$.

(iii) If $A \subseteq C$, then $P(A \mid B) \leq P(C \mid B)$.

(iv) $P(A \cup C \mid B) = P(A \mid B) + P(C \mid B) - P(A \cap C \mid B)$.

Definition 4 (Independence). Events $A$ and $B$ are independent if $P(A \cap B) = P(A)P(B)$, equivalently (when $P(B) > 0$) if $P(A \mid B) = P(A)$. Knowing $B$ does not change the probability of $A$.

Proposition 3. If $A$ and $B$ are independent, then $A$ and $B^c$, $A^c$ and $B$, and $A^c$ and $B^c$ are also independent.

---

The Product Rule and Bayes' Theorem

Theorem 2 (Product Rule). If $P(B) > 0$, then:

$$P(A \cap B) = P(A \mid B) \cdot P(B). \tag{6}$$

Proof. This is the definition (4) multiplied by $P(B)$ on both sides. $\blacksquare$

By symmetry of intersection ($A \cap B = B \cap A$), we also have:

$$P(A \cap B) = P(B \mid A) \cdot P(A), \tag{7}$$

provided $P(A) > 0$.

Theorem 3 (Bayes' Theorem — Event Form). If $P(A) > 0$ and $P(B) > 0$, then:

$$P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}. \tag{8}$$

Proof. Equations (6) and (7) both equal $P(A \cap B)$. Equating their right-hand sides:

$$P(A \mid B) \cdot P(B) = P(B \mid A) \cdot P(A). \tag{9}$$

Dividing both sides by $P(B) > 0$ yields (8). $\blacksquare$

Important equation. Equation (8) is the foundational identity of Bayesian reasoning. Each term has a standard name:

• $P(A \mid B)$ — Posterior — Updated belief about $A$ after observing $B$
• $P(B \mid A)$ — Likelihood — How probable the evidence is if $A$ is true
• $P(A)$ — Prior — Belief about $A$ before observing $B$
• $P(B)$ — Evidence (marginal) — Total probability of observing $B$

Remark. The derivation uses exactly three steps: write the product rule in two directions, equate, divide. There is no additional assumption beyond the Kolmogorov axioms and the definition of conditional probability.

Corollary 1 (Odds Form). The posterior odds equal the prior odds times the likelihood ratio:

$$\frac{P(A \mid B)}{P(A^c \mid B)} = \frac{P(B \mid A)}{P(B \mid A^c)} \cdot \frac{P(A)}{P(A^c)}. \tag{10}$$

Proof. Apply (8) to both $A$ and $A^c$, then divide. $\blacksquare$

The likelihood ratio $P(B \mid A)/P(B \mid A^c)$ is the Bayes factor — a measure of how strongly the evidence supports $A$ over its complement.

---

The Law of Total Probability

In applications, $P(B)$ is often not directly available. We compute it by marginalizing over a partition of the hypothesis space.

Definition 5 (Partition). Events $\{A_1, \ldots, A_n\}$ form a partition of $\Omega$ if they are mutually disjoint and $\bigcup_{i=1}^n A_i = \Omega$.

Theorem 4 (Law of Total Probability). If $\{A_1, \ldots, A_n\}$ is a partition with $P(A_i) > 0$ for all $i$, then for any event $B$:

$$P(B) = \sum_{i=1}^{n} P(B \mid A_i) \cdot P(A_i). \tag{11}$$

Proof. Since the $A_i$ partition $\Omega$, we can write $B = B \cap \Omega = B \cap \bigcup_i A_i = \bigcup_i (B \cap A_i)$, a disjoint union. By countable additivity:

$$P(B) = \sum_{i=1}^{n} P(B \cap A_i) = \sum_{i=1}^{n} P(B \mid A_i) P(A_i), \tag{12}$$

where the last step applies the product rule (6). $\blacksquare$

Corollary 2 (Binary Case). For hypothesis $A$ and its complement:

$$P(B) = P(B \mid A) P(A) + P(B \mid A^c) P(A^c). \tag{13}$$

Corollary 3 (Expanded Bayes' Theorem). Substituting (13) into (8):

$$P(A \mid B) = \frac{P(B \mid A) P(A)}{P(B \mid A) P(A) + P(B \mid A^c) P(A^c)}. \tag{14}$$

This is the form used for computation when only conditional probabilities and the prior are known.

Theorem 5 (Law of Total Probability — Continuous). If $X$ is a continuous random variable with PDF $f_X(x)$, then for any event $B$:

$$P(B) = \int_{-\infty}^{\infty} P(B \mid X = x) f_X(x)\, dx. \tag{15}$$

This continuous version is essential for integrating out nuisance parameters in Bayesian models.

---

Bayes' Theorem in Continuous Settings

For continuous random variables, we work with densities rather than point probabilities.

Definition 6 (Conditional Density). The conditional PDF of $X$ given $Y = y$ is:

$$f_{X \mid Y}(x \mid y) = \frac{f_{X,Y}(x, y)}{f_Y(y)}, \quad f_Y(y) > 0, \tag{16}$$

where $f_{X,Y}$ is the joint density and $f_Y$ is the marginal density.

Theorem 6 (Bayes' Theorem — Continuous Form). For parameters $\theta$ and data $x$:

$$f_{\Theta \mid X}(\theta \mid x) = \frac{f_{X \mid \Theta}(x \mid \theta) f_\Theta(\theta)}{f_X(x)} = \frac{f_{X \mid \Theta}(x \mid \theta) f_\Theta(\theta)}{\int f_{X \mid \Theta}(x \mid \theta') f_\Theta(\theta')\, d\theta'}. \tag{17}$$

Proof. By definition (16) applied to the joint $(\Theta, X)$:

$$f_{\Theta \mid X}(\theta \mid x) = \frac{f_{\Theta, X}(\theta, x)}{f_X(x)}. \tag{18}$$

The joint factors as $f_{\Theta, X}(\theta, x) = f_{X \mid \Theta}(x \mid \theta) f_\Theta(\theta)$ by the product rule for densities (analogous to (7)). The denominator $f_X(x) = \int f_{X \mid \Theta}(x \mid \theta') f_\Theta(\theta')\, d\theta'$ follows from (15). $\blacksquare$

Remark on notation. We write $f_{X \mid \Theta}(x \mid \theta)$ for the likelihood (density of data given parameters) and $f_\Theta(\theta)$ for the prior. The posterior $f_{\Theta \mid X}(\theta \mid x)$ is proportional to the product of likelihood and prior:

$$f_{\Theta \mid X}(\theta \mid x) \propto f_{X \mid \Theta}(x \mid \theta) f_\Theta(\theta). \tag{19}$$

The evidence $f_X(x)$ serves as the normalization constant. This proportionality is the computational workhorse of Bayesian inference.

Proposition 4. The continuous Bayes' theorem (17) is consistent with the discrete version (8): replacing sums by integrals and probabilities by densities yields the same logical structure.

---

Sequential Bayesian Updating

A powerful consequence of Bayes' theorem is that learning is additive in log-space and multiplicative in probability-space across independent observations.

Theorem 7 (Sequential Updating). Suppose we observe data in two stages: first $B_1$, then $B_2$. Then:

$$P(A \mid B_1, B_2) = \frac{P(B_2 \mid A, B_1) P(B_1 \mid A) P(A)}{P(B_1, B_2)}. \tag{20}$$

Proof. Apply Bayes' theorem (8) with evidence $B_1 \cap B_2$:

$$P(A \mid B_1, B_2) = \frac{P(B_1 \cap B_2 \mid A) P(A)}{P(B_1 \cap B_2)}. \tag{21}$$

If $B_2$ is conditionally independent of $B_1$ given $A$ (written $B_2 \perp B_1 \mid A$), then $P(B_1 \cap B_2 \mid A) = P(B_2 \mid A) P(B_1 \mid A)$, and (20) simplifies to:

$$P(A \mid B_1, B_2) = \frac{P(B_2 \mid A) P(B_1 \mid A) P(A)}{P(B_1, B_2)}. \tag{22}$$

Alternatively, update in two steps: compute $P(A \mid B_1)$ from (8), then use it as the prior for the second update:

$$P(A \mid B_1, B_2) = \frac{P(B_2 \mid A, B_1) P(A \mid B_1)}{P(B_2 \mid B_1)}. \tag{23}$$

When $B_2 \perp B_1 \mid A$, this reduces to (22). $\blacksquare$

Proposition 5 (Posterior as Prior). The posterior from one observation becomes the prior for the next:

$$\underbrace{P(A \mid B_1)}_{\text{interim posterior}} \xrightarrow{\text{observe } B_2} \underbrace{P(A \mid B_1, B_2)}_{\text{updated posterior}}. \tag{24}$$

This conjugate updating structure is the foundation of online learning, Kalman filtering, and streaming Bayesian inference.

Proposition 6 (Log-Posterior Additivity). For conditionally independent observations $x_1, \ldots, x_n$ given $\theta$:

$$\log f_{\Theta \mid X}(\theta \mid x_1, \ldots, x_n) = \sum_{i=1}^{n} \log f_{X \mid \Theta}(x_i \mid \theta) + \log f_\Theta(\theta) - \log f_X(x_1, \ldots, x_n). \tag{25}$$

The log-likelihood $\sum_i \log f_{X \mid \Theta}(x_i \mid \theta)$ accumulates evidence; this connects directly to Maximum Likelihood Estimation.

---

The Bayesian Inference Framework

We now assemble the components into the standard Bayesian inference pipeline.

Definition 7 (Bayesian Model). A Bayesian model consists of:

1. A parameter space $\Theta$ with prior $f_\Theta(\theta)$.
2. A likelihood $f_{X \mid \Theta}(x \mid \theta)$ specifying how data is generated given parameters.
3. The posterior $f_{\Theta \mid X}(\theta \mid x) \propto f_{X \mid \Theta}(x \mid \theta) f_\Theta(\theta)$ obtained via Bayes' theorem.

Definition 8 (Posterior Predictive). The posterior predictive distribution for new data $x_{\text{new}}$ is:

$$f_{X_{\text{new}} \mid X}(x_{\text{new}} \mid x) = \int f_{X_{\text{new}} \mid \Theta}(x_{\text{new}} \mid \theta) f_{\Theta \mid X}(\theta \mid x)\, d\theta. \tag{26}$$

This integrates over parameter uncertainty — a key advantage over point estimation.

Definition 9 (Maximum A Posteriori — Preview). The MAP estimator is the mode of the posterior:

$$\hat{\theta}_{\text{MAP}} = \underset{\theta}{\text{argmax}} \; f_{\Theta \mid X}(\theta \mid x) = \underset{\theta}{\text{argmax}} \; [\log f_{X \mid \Theta}(x \mid \theta) + \log f_\Theta(\theta)]. \tag{27}$$

When the prior is flat (uniform), MAP coincides with MLE. The log-prior term $\log f_\Theta(\theta)$ acts as regularization — developed fully in later chapters.

Theorem 8 (Exchangeability and de Finetti). If observations $x_1, \ldots, x_n$ are exchangeable (invariant under permutations) and the model is parameterized by $\theta$, then by de Finetti's representation theorem, there exists a prior $f_\Theta$ such that the data are conditionally i.i.d. given $\theta$. Bayesian updating via Bayes' theorem is then the coherent way to learn about $\theta$.

This theorem provides the philosophical foundation: if you believe data are exchangeable, you are (implicitly or explicitly) a Bayesian.

---

Worked Examples

Example 1: Medical Diagnosis (Discrete)

Problem. A disease affects 1% of the population. A diagnostic test has sensitivity $P(\text{Pos} \mid D) = 0.95$ and false positive rate $P(\text{Pos} \mid D^c) = 0.10$. A patient tests positive. What is $P(D \mid \text{Pos})$?

Step 1 — Identify quantities.

$$P(D) = 0.01, \quad P(D^c) = 0.99, \quad P(\text{Pos} \mid D) = 0.95, \quad P(\text{Pos} \mid D^c) = 0.10. \tag{28}$$

Step 2 — Compute evidence via (13).

$$P(\text{Pos}) = 0.95 \times 0.01 + 0.10 \times 0.99 = 0.0095 + 0.099 = 0.1085. \tag{29}$$

Step 3 — Apply Bayes' theorem (8).

$$P(D \mid \text{Pos}) = \frac{0.95 \times 0.01}{0.1085} = \frac{0.0095}{0.1085} \approx 0.0876. \tag{30}$$

Result. Despite 95% sensitivity, the posterior is only 8.76%. The low prior ($P(D) = 0.01$) means false positives from the 99% healthy population dominate.

Verification. $P(D^c \mid \text{Pos}) = (0.10 \times 0.99)/0.1085 \approx 0.9124$, and $0.0876 + 0.9124 = 1$.

Example 2: Sequential Updating

Using the posterior $P(D \mid \text{Pos}) \approx 0.0876$ as the new prior for a second independent positive test:

$$P(\text{Pos})_{\text{2nd}} = 0.95 \times 0.0876 + 0.10 \times 0.9124 = 0.1745, \tag{31}$$ $$P(D \mid \text{Pos}, \text{Pos}) = \frac{0.95 \times 0.0876}{0.1745} \approx 0.477. \tag{32}$$

Two consistent positive tests raise the posterior to roughly 48% — still not certainty, reflecting the imperfect specificity.

Example 3: Beta–Binomial Conjugacy (Continuous Prior)

Setup. $n$ trials with $k$ successes. Model: $X \mid p \sim \text{Binomial}(n, p)$, prior $p \sim \text{Beta}(\alpha, \beta)$.

Likelihood. $P(X = k \mid p) = \binom{n}{k} p^k (1-p)^{n-k}$.

Prior. $f(p) \propto p^{\alpha-1}(1-p)^{\beta-1}$.

Posterior. By (19):

$$f(p \mid k) \propto p^k (1-p)^{n-k} \cdot p^{\alpha-1}(1-p)^{\beta-1} = p^{k+\alpha-1}(1-p)^{n-k+\beta-1}. \tag{33}$$

This is $\text{Beta}(\alpha + k, \beta + n - k)$ — the same family as the prior. Conjugacy makes updating analytic.

---

Connection to Machine Learning

Naive Bayes Classification

Given features $\mathbf{x} = (x_1, \ldots, x_d)$ and class $C$, Bayes' theorem gives:

$$P(C \mid \mathbf{x}) = \frac{P(C) \prod_{j=1}^{d} P(x_j \mid C)}{P(\mathbf{x})}, \tag{34}$$

where the naive assumption $x_j \perp x_{k} \mid C$ for $j \neq k$ enables factorization via the product rule. Classification: $\hat{C} = \arg\max_C P(C \mid \mathbf{x})$; the evidence $P(\mathbf{x})$ cancels in the argmax.

Bayesian Neural Networks

For weights $\boldsymbol\theta$ and data $\mathcal{D}$:

$$P(\boldsymbol\theta \mid \mathcal{D}) = \frac{P(\mathcal{D} \mid \boldsymbol\theta) P(\boldsymbol\theta)}{P(\mathcal{D})}. \tag{35}$$

The evidence $P(\mathcal{D}) = \int P(\mathcal{D} \mid \boldsymbol\theta) P(\boldsymbol\theta)\, d\boldsymbol\theta$ is typically intractable for large networks, motivating Variational Inference and Laplace approximations.

Gaussian Processes and Regression

In Bayesian Linear Regression, the posterior over weights is computed via Bayes' theorem with a Gaussian prior and Gaussian likelihood — a direct application of (17) with conjugate structure developed in The Multivariate Gaussian.

RLHF and Preference Modeling

The Bradley–Terry model defines $P(y \succ y' \mid \theta) = \sigma(r_\theta(y) - r_\theta(y'))$. Training a reward model $r_\theta$ from human comparisons is maximum-likelihood (or Bayesian) inference over preferences — Bayes' theorem governs how comparison data updates beliefs about $\theta$.

---

Common Pitfalls and Misconceptions

Pitfall 1: Confusing $P(A \mid B)$ with $P(B \mid A)$. These are generally different. The prosecutor's fallacy equates $P(\text{evidence} \mid \text{innocent})$ with $P(\text{innocent} \mid \text{evidence})$. Bayes' theorem requires the prior and evidence term to convert between them.

Pitfall 2: Ignoring the base rate. Example 1 shows that high sensitivity does not imply high posterior when the prior is low. Always compute $P(B)$ via the law of total probability.

Pitfall 3: Treating the likelihood as a probability over parameters. $f_{X \mid \Theta}(x \mid \theta)$ as a function of $\theta$ with $x$ fixed is the likelihood, not a density over $\theta$. It need not integrate to 1 in $\theta$.

Pitfall 4: Assuming independence when it fails. Naive Bayes assumes conditional independence of features given the class. Violations degrade performance; structured models (graphical models) relax this assumption.

Pitfall 5: Double-counting evidence. In sequential updating, the prior for step 2 must be the posterior from step 1 — not the original prior combined with both observations simultaneously without accounting for dependence.

Pitfall 6: Confusing "Bayesian" with "subjective." Priors can encode genuine prior knowledge, regularization (as in MAP), or non-informative defaults. The mathematics of Bayes' theorem is agnostic to the prior's source.

---

Research Perspective

Thomas Bayes (1702–1761) formulated the theorem posthumously in "An Essay towards solving a Problem in the Doctrine of Chances" (1763). Pierre-Simon Laplace independently rediscovered and extended it, applying it to celestial mechanics and demographic estimation. For most of the twentieth century, frequentist methods dominated statistics, partly because computing the evidence integral was intractable.

The Bayesian revival began with conjugate analysis (Raiffa & Schlaifer, 1961), Markov chain Monte Carlo (Metropolis et al., 1953; Hastings, 1970; Geman & Geman, 1984), and variational methods (Jordan et al., 1999). Modern machine learning has made Bayesian reasoning central again: VAEs approximate posteriors via KL Divergence; Bayesian optimization guides hyperparameter search; RLHF uses KL penalties to constrain policy updates.

Open research questions include: scalable exact inference for deep networks, prior specification in high dimensions, and the relationship between SGD-trained overparameterized networks and implicit Bayesian inference (the "Bayesian deep learning" debate).

---

Summary of Takeaways

• Conditional probability — $P(A \mid B) = P(A \cap B)/P(B)$ — Restricted probability measure
• Product rule — $P(A \cap B) = P(A \mid B) P(B)$ — Joint from conditional
• Bayes' theorem — $P(A \mid B) = P(B \mid A) P(A) / P(B)$ — Belief updating
• Law of total probability — $P(B) = \sum_i P(B \mid A_i) P(A_i)$ — Compute evidence
• Continuous form — $f(\theta \mid x) \propto f(x \mid \theta) f(\theta)$ — Parameter inference
• Sequential updating — Posterior $\to$ prior for next observation — Online learning

Next article: The Multivariate Gaussian Distribution → — the distribution that makes Bayesian inference tractable in continuous settings.

---

Exercises

Exercise 1 (Verification). Prove Proposition 2 (conditional probability rules) directly from Definition 3.

Exercise 2 (Derivation). Derive the odds form (10) from Bayes' theorem applied to $A$ and $A^c$.

Exercise 3 (Partition). A factory has two machines: Machine 1 produces 60% of items with 2% defect rate; Machine 2 produces 40% with 5% defect rate. An item is defective. What is $P(\text{Machine 1} \mid \text{Defective})$? Use (14).

Exercise 4 (Sequential updating). For the medical diagnosis example, compute $P(D \mid \text{Pos}, \text{Pos}, \text{Pos})$ after three independent positive tests.

Exercise 5 (Continuous). For $X \mid \lambda \sim \text{Poisson}(\lambda)$ and $\lambda \sim \text{Gamma}(\alpha, \beta)$, derive the posterior $f(\lambda \mid x)$ and identify the conjugate family.

Exercise 6 (Conceptual). Explain why $P(\text{Data} \mid \text{Hypothesis})$ is not the probability that the hypothesis is true, and state what additional information Bayes' theorem requires.

Exercise 7 (Proof). Prove that if $A$ and $B$ are independent, then $P(A \mid B) = P(A)$ and $P(B \mid A) = P(B)$.

Exercise 8 (Bayes factor). Show that the posterior odds (10) can be written as prior odds times the Bayes factor, and interpret a Bayes factor of 10.