Maximum Likelihood Estimation (MLE)

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. The Likelihood Function and Likelihood Principle
6. Maximum Likelihood Estimation
7. Log-Likelihood and Computational Advantages
8. MLE for Exponential Family Distributions
9. Score Function and Fisher Information
10. Consistency of the MLE
11. Asymptotic Normality and the Cramér-Rao Bound
12. Invariance and Functional MLE
13. MLE as KL Minimization and Cross-Entropy
14. MLE for Neural Networks
15. Limitations and the Path to Bayesian Methods
16. Common Pitfalls and Misconceptions
17. Research Perspective
18. Summary of Takeaways
19. Exercises

---

Learning Objectives

After reading this chapter, you should be able to:

1. Define the likelihood function and distinguish it from a PDF over parameters.
2. Derive MLEs for Gaussian, Bernoulli, Poisson, and exponential family distributions.
3. State and interpret the score function and Fisher information matrix.
4. State consistency and asymptotic normality theorems for the MLE and sketch their proofs.
5. Explain the Cramér-Rao lower bound and asymptotic efficiency of the MLE.
6. Prove that MLE minimizes KL divergence to the empirical distribution and equates to cross-entropy minimization.
7. Identify MLE as the foundation of MSE loss, cross-entropy loss, and neural network training.

---

Prerequisites

• Bayes' Theorem: likelihood, posterior, prior; connection to MAP.
• KL Divergence: $D_{\text{KL}}(\hat{p}_n \| p_\theta)$, cross-entropy equivalence.
• Entropy & Information Content: entropy, expected log-probability.
• Gradient Descent: optimization of log-likelihood via gradients.

---

Notation

• $\mathcal{D} = \lbrace x_i \rbrace_{i=1}^N$ — Observed dataset
• $\theta, \boldsymbol{\theta}$ — Model parameters
• $L(\theta), \ell(\theta)$ — Likelihood and log-likelihood
• $\hat{\theta}_{\mathrm{MLE}}$ — Maximum likelihood estimator
• $I(\theta)$ — Fisher information
• $\mathcal{I}(\theta)$ — Observed information

---

Core Intuition

Given data $\mathcal{D} = \{x_1, \ldots, x_n\}$, we posit a parametric family $\{p(x \mid \boldsymbol\theta) : \boldsymbol\theta \in \Theta\}$ and ask: which parameter value makes the observed data most probable? This is Maximum Likelihood Estimation (MLE) — the dominant paradigm for parameter learning in statistics and machine learning.

When you train a neural network with cross-entropy loss, you are performing MLE under a categorical model. When you minimize MSE for regression, you are performing MLE under Gaussian noise. The loss functions of deep learning are not arbitrary — they are derived from probabilistic assumptions via the likelihood principle.

MLE has elegant asymptotic theory: under regularity conditions, $\hat{\boldsymbol\theta}_{\text{MLE}}$ is consistent (converges to the true parameter), asymptotically normal (with variance governed by Fisher information), and efficient (achieves the Cramér-Rao lower bound). These guarantees explain why MLE is the default — and why its limitations (overfitting, no uncertainty quantification) motivate Bayesian extensions.

Series context. This is Part V of Chapter 3 in Volume I. It precedes Bias–Variance Tradeoff and connects to Linear Regression and Logistic Regression in Volume II.

---

The Likelihood Function and Likelihood Principle

Setup. Observations $x_1, \ldots, x_n$ are i.i.d. from unknown distribution $p^*$, modeled by parametric family $\{p(x \mid \boldsymbol\theta) : \boldsymbol\theta \in \Theta \subseteq \mathbb{R}^d\}$.

Definition 1 (Likelihood Function). The likelihood function is the probability (or density) of the observed data viewed as a function of parameters:

$$\mathcal{L}(\boldsymbol\theta) = \mathcal{L}(\boldsymbol\theta; \mathcal{D}) = p(\mathcal{D} \mid \boldsymbol\theta) = \prod_{i=1}^n p(x_i \mid \boldsymbol\theta). \tag{1}$$

The i.i.d. factorization is essential.

Critical distinction.

• $p(x \mid \boldsymbol\theta)$ — $x$ (fixed $\boldsymbol\theta$) — $x$ — it is a PMF/PDF
• $\mathcal{L}(\boldsymbol\theta)$ — $\boldsymbol\theta$ (fixed data) — Not necessarily $\boldsymbol\theta$

Example. For $n$ Bernoulli trials with $k$ successes: $\mathcal{L}(p) = p^k(1-p)^{n-k}$. As a function of $p \in [0,1]$, this integrates to 1 over $p$ only for specific $n, k$ — it is not a density on $p$ without a prior (Bayesian view).

Likelihood Principle (Informal). All evidence about $\boldsymbol\theta$ from the experiment is contained in $\mathcal{L}(\boldsymbol\theta)$. Two experiments with proportional likelihoods ($\mathcal{L}_1(\boldsymbol\theta) = c \cdot \mathcal{L}_2(\boldsymbol\theta)$) yield identical inference about $\boldsymbol\theta$.

---

Maximum Likelihood Estimation

Definition 2 (Maximum Likelihood Estimator). The MLE is:

$$\hat{\boldsymbol\theta}_{\text{MLE}} = \underset{\boldsymbol\theta \in \Theta}{\text{argmax}} \; \mathcal{L}(\boldsymbol\theta) = \underset{\boldsymbol\theta \in \Theta}{\text{argmax}} \; \prod_{i=1}^n p(x_i \mid \boldsymbol\theta). \tag{2}$$

Definition 3 (Log-Likelihood). The log-likelihood is:

$$\ell(\boldsymbol\theta) = \log \mathcal{L}(\boldsymbol\theta) = \sum_{i=1}^n \log p(x_i \mid \boldsymbol\theta). \tag{3}$$

Since $\log$ is strictly increasing, $\hat{\boldsymbol\theta}_{\text{MLE}} = \arg\max_{\boldsymbol\theta} \ell(\boldsymbol\theta)$.

Proposition 1 (First-Order Condition). If $\hat{\boldsymbol\theta}_{\text{MLE}}$ is in the interior of $\Theta$ and $\ell$ is differentiable, then:

$$\nabla_{\boldsymbol\theta} \ell(\hat{\boldsymbol\theta}_{\text{MLE}}) = \mathbf{0}. \tag{4}$$

Proof. Necessary condition for an interior maximum of a differentiable function. $\blacksquare$

Proposition 2 (Second-Order Condition). If additionally $-\nabla^2 \ell(\hat{\boldsymbol\theta}_{\text{MLE}}) \succ 0$ (negative Hessian positive definite), then $\hat{\boldsymbol\theta}_{\text{MLE}}$ is a local maximum.

---

Log-Likelihood and Computational Advantages

Why log?

1. Products become sums: $\ell(\boldsymbol\theta) = \sum_i \log p(x_i \mid \boldsymbol\theta)$ — numerically stable, avoids underflow.
2. Exponential family: Log-likelihood is often concave, enabling closed-form or convex optimization.
3. Gradients decompose: $\nabla \ell(\boldsymbol\theta) = \sum_i \nabla \log p(x_i \mid \boldsymbol\theta)$ — supports stochastic gradients (SGD).

Proposition 3. For i.i.d. data, $\ell(\boldsymbol\theta) = n \cdot \frac{1}{n}\sum_{i=1}^n \log p(x_i \mid \boldsymbol\theta) = n \cdot \mathbb{E}_{\hat{p}_n}[\log p(X \mid \boldsymbol\theta)]$ where $\hat{p}_n$ is the empirical distribution.

---

MLE for Exponential Family Distributions

Definition 4 (Exponential Family). A distribution with density:

$$p(x \mid \boldsymbol\theta) = h(x) \exp\left(\boldsymbol\eta(\boldsymbol\theta)^T \mathbf{T}(x) - A(\boldsymbol\theta)\right), \tag{5}$$

where $\mathbf{T}(x)$ are sufficient statistics, $\boldsymbol\eta$ are natural parameters, and $A$ is the log-partition function.

Theorem 1 (MLE for Exponential Family). For i.i.d. samples from an exponential family, the MLE satisfies:

$$\mathbb{E}_{p_{\boldsymbol\theta}}[\mathbf{T}(X)] = \bar{\mathbf{T}}_n = \frac{1}{n}\sum_{i=1}^n \mathbf{T}(x_i), \tag{6}$$

i.e., match expected sufficient statistics to sample averages.

Gaussian — Known Variance

$x_i \sim \mathcal{N}(\mu, \sigma^2)$, $\sigma^2$ known:

$$\ell(\mu) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i - \mu)^2. \tag{7}$$ $$\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^n (x_i - \mu) = 0 \implies \hat{\mu}_{\text{MLE}} = \bar{x} = \frac{1}{n}\sum_{i=1}^n x_i. \tag{8}$$

Gaussian — Unknown Mean and Variance

$$\hat{\mu}_{\text{MLE}} = \bar{x}, \quad \hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2. \tag{9}$$

Remark. $\hat{\sigma}^2_{\text{MLE}}$ divides by $n$, not $n-1$ — it is biased. The unbiased estimator uses Bessel's correction ($n-1$). This bias–variance tradeoff is developed in Bias–Variance Tradeoff.

Bernoulli

$x_i \in \{0, 1\}$, $P(x_i = 1) = p$:

$$\ell(p) = \sum_{i=1}^n [x_i \log p + (1-x_i)\log(1-p)]. \tag{10}$$ $$\frac{\partial \ell}{\partial p} = \frac{\sum_i x_i}{p} - \frac{n - \sum_i x_i}{1-p} = 0 \implies \hat{p}_{\text{MLE}} = \frac{1}{n}\sum_{i=1}^n x_i. \tag{11}$$

Poisson

$x_i \sim \text{Poisson}(\lambda)$:

$$\ell(\lambda) = \sum_i [x_i \log \lambda - \lambda - \log(x_i!)] \implies \hat{\lambda}_{\text{MLE}} = \bar{x}. \tag{12}$$

---

Score Function and Fisher Information

Definition 5 (Score Function). The score is the gradient of the log-likelihood for a single observation:

$$\mathbf{s}(\boldsymbol\theta; x) = \nabla_{\boldsymbol\theta} \log p(x \mid \boldsymbol\theta). \tag{13}$$

For $n$ i.i.d. observations: $\nabla \ell(\boldsymbol\theta) = \sum_{i=1}^n \mathbf{s}(\boldsymbol\theta; x_i)$.

Proposition 4 (Score Has Zero Expectation). Under regularity conditions:

$$\mathbb{E}_{x \sim p(\cdot \mid \boldsymbol\theta)}[\mathbf{s}(\boldsymbol\theta; x)] = \mathbf{0}. \tag{14}$$

Proof.

$$\mathbb{E}[\mathbf{s}] = \int \nabla_{\boldsymbol\theta} p(x \mid \boldsymbol\theta)\, dx = \nabla_{\boldsymbol\theta} \int p(x \mid \boldsymbol\theta)\, dx = \nabla_{\boldsymbol\theta} 1 = \mathbf{0}. \tag{15}$$

(Exchange of derivative and integral justified by regularity.) $\blacksquare$

Definition 6 (Fisher Information). The Fisher information matrix is:

$$\mathbf{I}(\boldsymbol\theta) = \mathbb{E}\left[\mathbf{s}(\boldsymbol\theta; X) \mathbf{s}(\boldsymbol\theta; X)^T\right] = -\mathbb{E}\left[\nabla^2_{\boldsymbol\theta} \log p(X \mid \boldsymbol\theta)\right]. \tag{16}$$

The second equality follows from differentiating $\mathbb{E}[\mathbf{s}] = \mathbf{0}$ w.r.t. $\boldsymbol\theta$.

Interpretation. $\mathbf{I}(\boldsymbol\theta)$ measures the curvature of the log-likelihood — how sharply peaked it is around the true parameter. Large Fisher information $\implies$ data are highly informative about $\boldsymbol\theta$ $\implies$ lower variance of estimators.

Example (Bernoulli). $\mathbf{I}(p) = 1/(p(1-p))$. Information is lowest at $p = 1/2$ (hardest to distinguish from noise) and diverges at $p \in \{0, 1\}$.

Example (Gaussian, known $\sigma^2$). $\mathbf{I}(\mu) = n/\sigma^2$.

---

Consistency of the MLE

Definition 7 (Consistency). Estimator $\hat{\boldsymbol\theta}_n$ is consistent for $\boldsymbol\theta^*$ if $\hat{\boldsymbol\theta}_n \xrightarrow{p} \boldsymbol\theta^*$ as $n \to \infty$.

Theorem 2 (Consistency of MLE). Under regularity conditions (identifiability, compact $\Theta$, continuity, dominance for exchange of limit and integral):

$$\hat{\boldsymbol\theta}_{\text{MLE}} \xrightarrow{p} \boldsymbol\theta^*, \tag{17}$$

where $\boldsymbol\theta^* = \arg\max_{\boldsymbol\theta} \mathbb{E}_{p^*}[\log p(X \mid \boldsymbol\theta)]$ — the KL-closest parameter if $p^*$ is not in the model.

Proof sketch.

Step 1. By the Law of Large Numbers:

$$\frac{1}{n}\ell(\boldsymbol\theta) = \frac{1}{n}\sum_{i=1}^n \log p(x_i \mid \boldsymbol\theta) \xrightarrow{a.s.} \mathbb{E}_{p^*}[\log p(X \mid \boldsymbol\theta)] = -H(p^*, p_{\boldsymbol\theta}) + \text{const}. \tag{18}$$

Step 2. By identifiability, $\mathbb{E}_{p^*}[\log p(X \mid \boldsymbol\theta)]$ is uniquely maximized at $\boldsymbol\theta^*$ (true parameter if model is correct).

Step 3. By uniform convergence (regularity) and the argmax continuous mapping theorem, $\arg\max \frac{1}{n}\ell(\boldsymbol\theta) \xrightarrow{p} \arg\max \mathbb{E}[\log p(X \mid \boldsymbol\theta)] = \boldsymbol\theta^*$. $\blacksquare$

Interpretation. With enough data, MLE finds the best parameter in the model family — in the KL sense from KL Divergence, Theorem 3.

---

Asymptotic Normality and the Cramér-Rao Bound

Theorem 3 (Asymptotic Normality of MLE). Under regularity conditions, as $n \to \infty$:

$$\sqrt{n}(\hat{\boldsymbol\theta}_{\text{MLE}} - \boldsymbol\theta^*) \xrightarrow{d} \mathcal{N}(\mathbf{0}, \mathbf{I}(\boldsymbol\theta^*)^{-1}). \tag{19}$$

Proof sketch. Taylor expand the score at $\hat{\boldsymbol\theta}_{\text{MLE}}$ around $\boldsymbol\theta^*$:

$$\mathbf{0} = \nabla \ell(\hat{\boldsymbol\theta}_{\text{MLE}}) \approx \nabla \ell(\boldsymbol\theta^*) + \nabla^2 \ell(\boldsymbol\theta^*)(\hat{\boldsymbol\theta}_{\text{MLE}} - \boldsymbol\theta^*). \tag{20}$$

Rearranging:

$$\sqrt{n}(\hat{\boldsymbol\theta}_{\text{MLE}} - \boldsymbol\theta^*) \approx -[\nabla^2 \ell(\boldsymbol\theta^*)/n]^{-1} \cdot \nabla \ell(\boldsymbol\theta^*)/\sqrt{n}. \tag{21}$$

By LLN: $-\nabla^2 \ell(\boldsymbol\theta^*)/n \xrightarrow{p} \mathbf{I}(\boldsymbol\theta^*)$. By CLT: $\nabla \ell(\boldsymbol\theta^*)/\sqrt{n} \xrightarrow{d} \mathcal{N}(\mathbf{0}, \mathbf{I}(\boldsymbol\theta^*))$. Slutsky's theorem gives (19). $\blacksquare$

Theorem 4 (Cramér-Rao Lower Bound). For any unbiased estimator $\tilde{\boldsymbol\theta}$ of $\boldsymbol\theta^*$:

$$\text{Cov}(\tilde{\boldsymbol\theta}) \succeq \mathbf{I}(\boldsymbol\theta^*)^{-1}. \tag{22}$$

Proof sketch. Cauchy–Schwarz inequality on $\text{Cov}(\tilde{\boldsymbol\theta}, \mathbf{s})$ with $\mathbb{E}[\mathbf{s}] = \mathbf{0}$. $\blacksquare$

Corollary 2 (Asymptotic Efficiency). The MLE achieves the Cramér-Rao bound asymptotically — it is the most efficient regular estimator in the large-sample limit.

---

Invariance and Functional MLE

Theorem 5 (Invariance of MLE). If $\hat{\boldsymbol\theta}_{\text{MLE}}$ is the MLE of $\boldsymbol\theta$, then for any function $g$, the MLE of $\eta = g(\boldsymbol\theta)$ is:

$$\hat{\eta}_{\text{MLE}} = g(\hat{\boldsymbol\theta}_{\text{MLE}}). \tag{23}$$

Proof. The likelihood depends on $\boldsymbol\theta$ only through $g(\boldsymbol\theta)$ when reparameterizing; maximizing over $\eta$ is equivalent to maximizing over $\boldsymbol\theta$ and applying $g$. $\blacksquare$

Example. For Gaussian variance, $\hat{\sigma}^2_{\text{MLE}} = \frac{1}{n}\sum (x_i - \bar{x})^2$. The MLE of $\sigma$ is $\sqrt{\hat{\sigma}^2_{\text{MLE}}}$, not $\hat{\sigma}^2_{\text{MLE}}$ divided differently — invariance gives $\hat{\sigma}_{\text{MLE}} = \sqrt{\hat{\sigma}^2_{\text{MLE}}}$.

---

MLE as KL Minimization and Cross-Entropy

Theorem 6 (MLE $\Leftrightarrow$ KL Minimization). The MLE minimizes KL divergence from the empirical distribution:

$$\hat{\boldsymbol\theta}_{\text{MLE}} = \underset{\boldsymbol\theta}{\text{argmin}} \; D_{\text{KL}}(\hat{p}_n \| p_{\boldsymbol\theta}). \tag{24}$$

Proof. From KL Divergence, Theorem 3. $\blacksquare$

Theorem 7 (Cross-Entropy for Classification). For one-hot labels $y_i \in \{1, \ldots, C\}$ and model $p_{\boldsymbol\theta}(y \mid \mathbf{x}) = \text{softmax}(f_{\boldsymbol\theta}(\mathbf{x}))$:

$$\mathcal{L}_{\text{CE}} = -\sum_{i=1}^n \log p_{\boldsymbol\theta}(y_i \mid \mathbf{x}_i) \tag{25}$$

is the negative log-likelihood. Minimizing $\mathcal{L}_{\text{CE}}$ $\equiv$ MLE $\equiv$ minimizing $D_{\text{KL}}(\hat{p}_{\text{empirical}} \| p_{\boldsymbol\theta})$.

Theorem 8 (MSE for Regression). For $y_i \mid \mathbf{x}_i \sim \mathcal{N}(f_{\boldsymbol\theta}(\mathbf{x}_i), \sigma^2)$:

$$\ell(\boldsymbol\theta) = -\frac{n}{2}\log(2\pi\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n (y_i - f_{\boldsymbol\theta}(\mathbf{x}_i))^2. \tag{26}$$

Maximizing $\ell$ $\equiv$ minimizing $\sum_i (y_i - f_{\boldsymbol\theta}(\mathbf{x}_i))^2$ (MSE).

---

MLE for Neural Networks

A neural network defines a conditional distribution $p_{\boldsymbol\theta}(y \mid \mathbf{x})$.

Regression: Gaussian noise $\implies$ MSE loss (Theorem 8).

Classification: Categorical/softmax $\implies$ cross-entropy loss (Theorem 7).

Language modeling: Autoregressive factorization $p(x_1, \ldots, x_T) = \prod_t p(x_t \mid x_{<t})$ $\implies$ sum of log-probabilities (negative log-likelihood per token).

Optimization. MLE is solved via Gradient Descent on $-\ell(\boldsymbol\theta)$ — SGD on mini-batches approximates the full gradient.

Key insight. Loss function design is not separate from probabilistic modeling. Choosing a loss is choosing a likelihood.

---

Limitations and the Path to Bayesian Methods

1. Overfitting. With limited data, MLE can overfit (e.g., $\hat{p} = 1$ after 3/3 coin flips). Regularization and MAP estimation add prior terms.

2. No uncertainty quantification. MLE gives a point estimate; it does not provide credible intervals without asymptotic approximations.

3. Singular likelihoods. Mixture models can have unbounded likelihood (component collapses on a single point). Constraints or priors needed.

4. Model misspecification. If $p^* \notin \{p_{\boldsymbol\theta}\}$, MLE finds the KL-closest model, which may be misleading.

5. Finite-sample bias. MLE can be biased (variance estimation with divisor $n$). Unbiased alternatives exist but may have higher variance.

Progression: MLE $\to$ MAP (add prior) $\to$ Full Bayesian (posterior over $\boldsymbol\theta$) — see Bayes' Theorem and Bayesian Linear Regression.

---

Common Pitfalls and Misconceptions

Pitfall 1: Treating likelihood as a probability over parameters. $\mathcal{L}(\boldsymbol\theta)$ is not a posterior; it requires a prior and Bayes' theorem for that interpretation.

Pitfall 2: Assuming MLE is always unbiased. Gaussian variance MLE is biased; only asymptotic properties are guaranteed generally.

Pitfall 3: Confusing MLE with least squares in all settings. MSE is MLE only under Gaussian noise. Laplace noise $\implies$ L1 loss (median regression); robust losses correspond to different likelihoods.

Pitfall 4: Ignoring regularity conditions. Without identifiability, consistency fails. Mixture models without constraints have singular Fisher information.

Pitfall 5: Equating training log-likelihood with generalization. Minimizing $-\ell$ on training data does not guarantee low KL to $p^*$ on test data — see Bias–Variance Tradeoff.

---

Research Perspective

Maximum likelihood dates to Fisher (1912–1922), who formalized it and developed asymptotic theory. The connection to KL divergence and information geometry (Amari, 1985) unified statistics and information theory.

In deep learning, MLE remains the default training objective, but research explores: noise-contrastive estimation, score matching (avoiding normalization constants), adversarial losses (non-likelihood), and Bayesian deep learning (posterior over weights). The debate over whether SGD on overparameterized networks performs implicit Bayesian inference continues.

---

Summary of Takeaways

• Likelihood — $\mathcal{L}(\boldsymbol\theta) = \prod_i p(x_i \mid \boldsymbol\theta)$ — Data probability as function of $\boldsymbol\theta$
• Log-likelihood — $\ell(\boldsymbol\theta) = \sum_i \log p(x_i \mid \boldsymbol\theta)$ — Sum form, gradients
• MLE — $\hat{\boldsymbol\theta} = \arg\max \ell(\boldsymbol\theta)$ — Most probable parameters
• Score — $\mathbf{s} = \nabla \log p(x \mid \boldsymbol\theta)$, $\mathbb{E}[\mathbf{s}] = \mathbf{0}$ — First-order condition
• Fisher information — $\mathbf{I}(\boldsymbol\theta) = \mathbb{E}[\mathbf{s}\mathbf{s}^T]$ — Asymptotic variance
• Consistency — $\hat{\boldsymbol\theta} \xrightarrow{p} \boldsymbol\theta^*$ — Large-$n$ correctness
• Asymptotic normality — $\sqrt{n}(\hat{\boldsymbol\theta} - \boldsymbol\theta^*) \to \mathcal{N}(\mathbf{0}, \mathbf{I}^{-1})$ — Confidence intervals
• MLE $\Leftrightarrow$ min KL — $\arg\min D_{\text{KL}}(\hat{p}_n \ — p_{\boldsymbol\theta})$ — Closest model to data

Next article: The Bias–Variance Tradeoff → — decomposing prediction error beyond point estimation.

---

Exercises

Exercise 1 (Bernoulli). Derive (11) completely, verifying the second derivative is negative at the optimum.

Exercise 2 (Gaussian). Derive (9) for both $\mu$ and $\sigma^2$ simultaneously.

Exercise 3 (Fisher information). Compute $\mathbf{I}(p)$ for Bernoulli and verify the Cramér-Rao bound for $\hat{p}_{\text{MLE}}$.

Exercise 4 (Consistency). Explain why $\hat{p}_{\text{MLE}} = 3/3 = 1$ after three heads is consistent for the true $p = 1/2$.

Exercise 5 (Invariance). If $\hat{\lambda}_{\text{MLE}} = \bar{x}$ for Poisson, what is the MLE of $\eta = e^\lambda$?

Exercise 6 (KL). Prove Theorem 6 directly from the definition of $D_{\text{KL}}(\hat{p}_n \| p_{\boldsymbol\theta})$.

Exercise 7 (Conceptual). Why does MSE correspond to Gaussian noise but not Laplace noise?

Exercise 8 (Asymptotic). State the regularity conditions needed for Theorem 3 and give an example where they fail.