ML

Gaussian Processes

The function-space view of Bayesian learning: defining GPs via mean and covariance functions, deriving the predictive distribution in closed form, kernel design, marginal likelihood for hyperparameter optimization, and connections to neural networks.

Advanced

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. Definition of a Gaussian Process
  5. The Prior Over Functions
  6. GP Regression: Noise-Free Case
  7. GP Regression: Noisy Observations
  8. Kernel (Covariance) Functions
  9. Marginal Likelihood and Hyperparameter Optimization
  10. Connection to Bayesian Linear Regression
  11. Connection to Neural Networks
  12. Computational Considerations
  13. Common Pitfalls
  14. Summary
  15. Exercises

Learning Objectives

  1. Define a Gaussian process and understand it as a distribution over functions.
  2. Derive the GP predictive distribution using properties of multivariate Gaussians.
  3. Interpret the role of the kernel in encoding assumptions about function smoothness and periodicity.
  4. Derive the log marginal likelihood and its gradient for hyperparameter optimization.
  5. Show that GP regression with a linear kernel recovers Bayesian linear regression.
  6. Explain the connection between GPs and infinite-width neural networks.

Notation

  • f∼GP(m,k)f \sim \mathcal{GP}(m, k) — Gaussian process with mean function mm and kernel kk
  • m(x)=E[f(x)]m(\mathbf{x}) = \mathbb{E}[f(\mathbf{x})] — mean function
  • k(x,x′)=Cov(f(x),f(x′))k(\mathbf{x}, \mathbf{x}') = \text{Cov}(f(\mathbf{x}), f(\mathbf{x}')) — covariance (kernel) function
  • K∈Rn×n\mathbf{K} \in \mathbb{R}^{n \times n} — kernel matrix with Kij=k(xi,xj)K_{ij} = k(\mathbf{x}_i, \mathbf{x}_j)
  • k∗∈Rn\mathbf{k}_* \in \mathbb{R}^n — vector with k∗i=k(x∗,xi)k_{*i} = k(\mathbf{x}_*, \mathbf{x}_i)
  • σn2\sigma_n^2 — observation noise variance

Core Intuition

Instead of placing a prior over parameters (as in Bayesian linear regression), a GP places a prior directly over functions. Any finite collection of function values has a joint Gaussian distribution. Given observations, we condition this joint Gaussian to get the posterior — which is again a GP. This gives us both predictions and calibrated uncertainty, all in closed form.

Gaussian Process Regression

Length ℓ
0.15
Noise σ²
0.01
Obs pts
4
2σ bandGP mean
Explore: Small ℓ → wiggly interpolation through points. Large ℓ → smooth extrapolation. Uncertainty grows away from observations.

Definition of a Gaussian Process

Definition. A Gaussian process is a collection of random variables, any finite number of which have a joint Gaussian distribution.

A GP is fully specified by:

m(x)=E[f(x)],k(x,x′)=E[(f(x)−m(x))(f(x′)−m(x′))].(1)m(\mathbf{x}) = \mathbb{E}[f(\mathbf{x})], \qquad k(\mathbf{x}, \mathbf{x}') = \mathbb{E}[(f(\mathbf{x}) - m(\mathbf{x}))(f(\mathbf{x}') - m(\mathbf{x}'))]. \tag{1}

We write f∼GP(m,k)f \sim \mathcal{GP}(m, k). For any finite set {x1,…,xn}\{\mathbf{x}_1, \ldots, \mathbf{x}_n\}:

[f(x1)⋮f(xn)]∼N([m(x1)⋮m(xn)],K).(2)\begin{bmatrix} f(\mathbf{x}_1) \\ \vdots \\ f(\mathbf{x}_n) \end{bmatrix} \sim \mathcal{N}\left(\begin{bmatrix} m(\mathbf{x}_1) \\ \vdots \\ m(\mathbf{x}_n) \end{bmatrix}, \mathbf{K}\right). \tag{2}

The Prior Over Functions

With m(x)=0m(\mathbf{x}) = 0 (common default), the GP prior says: "I expect the function to be near zero, with covariance structure determined by kk." Sampling from the prior produces random smooth functions whose properties (smoothness, lengthscale, amplitude) are controlled by the kernel.

GP Regression: Noise-Free Case

Given noise-free observations f=[f(x1),…,f(xn)]T\mathbf{f} = [f(\mathbf{x}_1), \ldots, f(\mathbf{x}_n)]^T and a test point x∗\mathbf{x}_*, the joint prior is:

[ff∗]∼N(0,[Kk∗k∗Tk∗∗]),(3)\begin{bmatrix}\mathbf{f} \\ f_*\end{bmatrix} \sim \mathcal{N}\left(\mathbf{0}, \begin{bmatrix}\mathbf{K} & \mathbf{k}_* \\ \mathbf{k}_*^T & k_{**}\end{bmatrix}\right), \tag{3}

where k∗∗=k(x∗,x∗)k_{**} = k(\mathbf{x}_*, \mathbf{x}_*).

Conditioning on the observations:

f∗∣f∼N(k∗TK−1f,  k∗∗−k∗TK−1k∗).(4)f_* \mid \mathbf{f} \sim \mathcal{N}\bigl(\mathbf{k}_*^T\mathbf{K}^{-1}\mathbf{f}, \; k_{**} - \mathbf{k}_*^T\mathbf{K}^{-1}\mathbf{k}_*\bigr). \tag{4}

The predictive mean interpolates training points exactly; the variance is zero at observed locations and grows with distance.

GP Regression: Noisy Observations

With yi=f(xi)+ϵiy_i = f(\mathbf{x}_i) + \epsilon_i, ϵi∼N(0,σn2)\epsilon_i \sim \mathcal{N}(0, \sigma_n^2):

Cov(yi,yj)=k(xi,xj)+σn2δij.(5)\text{Cov}(y_i, y_j) = k(\mathbf{x}_i, \mathbf{x}_j) + \sigma_n^2\delta_{ij}. \tag{5}

The predictive distribution at x∗\mathbf{x}_*:

μ∗=k∗T(K+σn2I)−1y(6)\boxed{\mu_* = \mathbf{k}_*^T(\mathbf{K} + \sigma_n^2\mathbf{I})^{-1}\mathbf{y}} \tag{6} σ∗2=k∗∗−k∗T(K+σn2I)−1k∗(7)\boxed{\sigma_*^2 = k_{**} - \mathbf{k}_*^T(\mathbf{K} + \sigma_n^2\mathbf{I})^{-1}\mathbf{k}_*} \tag{7}

Interpretation:

  • μ∗\mu_* is a weighted average of training targets, with weights determined by kernel similarity.
  • σ∗2\sigma_*^2 captures epistemic uncertainty: small near training data, large far away.

Kernel (Covariance) Functions

The kernel encodes our prior beliefs about ff:

Squared Exponential (RBF):

k(x,x′)=σf2exp⁡(−∥x−x′∥22ℓ2).(8)k(\mathbf{x}, \mathbf{x}') = \sigma_f^2\exp\left(-\frac{\lVert\mathbf{x} - \mathbf{x}'\rVert^2}{2\ell^2}\right). \tag{8}

Produces infinitely differentiable (very smooth) functions. ℓ\ell = lengthscale, σf2\sigma_f^2 = signal variance.

Matérn family:

kν(r)=σf221−νΓ(ν)(2νrℓ)νKν(2νrℓ).(9)k_\nu(r) = \sigma_f^2\frac{2^{1-\nu}}{\Gamma(\nu)}\left(\frac{\sqrt{2\nu}r}{\ell}\right)^\nu K_\nu\left(\frac{\sqrt{2\nu}r}{\ell}\right). \tag{9}

ν\nu controls smoothness: ν=1/2\nu = 1/2 (Ornstein–Uhlenbeck, non-differentiable), ν=3/2\nu = 3/2 (once differentiable), ν→∞\nu \to \infty (recovers SE).

Periodic:

k(x,x′)=σf2exp⁡(−2sin⁡2(π∣x−x′∣/p)ℓ2).(10)k(\mathbf{x}, \mathbf{x}') = \sigma_f^2\exp\left(-\frac{2\sin^2(\pi|\mathbf{x}-\mathbf{x}'|/p)}{\ell^2}\right). \tag{10}

Kernels can be composed: sums, products, and compositions of valid kernels are valid.

Marginal Likelihood and Hyperparameter Optimization

The log marginal likelihood (model evidence):

log⁡p(y∣X,θ)=−12yT(Kθ+σn2I)−1y−12log⁡∣Kθ+σn2I∣−n2log⁡2π.(11)\log p(\mathbf{y} \mid \mathbf{X}, \boldsymbol\theta) = -\frac{1}{2}\mathbf{y}^T(\mathbf{K}_\theta + \sigma_n^2\mathbf{I})^{-1}\mathbf{y} - \frac{1}{2}\log|\mathbf{K}_\theta + \sigma_n^2\mathbf{I}| - \frac{n}{2}\log 2\pi. \tag{11}

Three terms:

  1. Data fit: −12yTKy−1y-\frac{1}{2}\mathbf{y}^T\mathbf{K}_y^{-1}\mathbf{y} — penalizes poor predictions.
  2. Complexity penalty: −12log⁡∣Ky∣-\frac{1}{2}\log|\mathbf{K}_y| — penalizes overly flexible models.
  3. Normalization: constant.

Hyperparameters θ=(ℓ,σf,σn)\boldsymbol\theta = (\ell, \sigma_f, \sigma_n) are optimized by maximizing (11) via gradient ascent.

Connection to Bayesian Linear Regression

Consider the linear kernel k(x,x′)=σp2xTx′k(\mathbf{x}, \mathbf{x}') = \sigma_p^2\mathbf{x}^T\mathbf{x}'. Then:

K=σp2XTX.\mathbf{K} = \sigma_p^2\mathbf{X}^T\mathbf{X}.

The GP predictive mean becomes:

μ∗=σp2x∗TX(σp2XTX+σn2I)−1y=x∗Tw^MAP,\mu_* = \sigma_p^2\mathbf{x}_*^T\mathbf{X}(\sigma_p^2\mathbf{X}^T\mathbf{X} + \sigma_n^2\mathbf{I})^{-1}\mathbf{y} = \mathbf{x}_*^T\hat{\mathbf{w}}_{\text{MAP}},

which is exactly the posterior mean of Bayesian linear regression with Gaussian prior w∼N(0,σp2I)\mathbf{w} \sim \mathcal{N}(\mathbf{0}, \sigma_p^2\mathbf{I}).

Connection to Neural Networks

Theorem (Neal, 1996; Lee et al., 2018). A single hidden-layer neural network with random weights and infinite width converges to a GP. The kernel is determined by the activation function and weight priors.

This connects to the Neural Tangent Kernel: the output of a trained (infinite-width) neural network under gradient descent is described by a GP with the NTK as the covariance function.

Computational Considerations

  • Training: Computing K−1\mathbf{K}^{-1} costs O(n3)O(n^3); storage is O(n2)O(n^2).
  • Prediction: Each prediction costs O(n)O(n) (after precomputing K−1y\mathbf{K}^{-1}\mathbf{y}).
  • Scalability: For large nn, use inducing-point methods (sparse GPs), which reduce cost to O(nm2)O(nm^2) where m≪nm \ll n is the number of inducing points.

Common Pitfalls

Pitfall 1. Using the SE kernel when the function is non-smooth. The SE kernel assumes infinite differentiability; use Matérn ν=3/2\nu = 3/2 or 5/25/2 for finite smoothness.

Pitfall 2. Confusing predictive variance with noise. The GP variance σ∗2\sigma_*^2 is epistemic (reducible with more data); the noise σn2\sigma_n^2 is aleatoric (irreducible).

Pitfall 3. Over-optimizing the marginal likelihood with too many hyperparameters can lead to overfitting the model selection criterion itself.

Summary

  • A GP defines a distribution over functions via a mean and kernel.
  • Conditioning on data gives exact posterior predictions: mean and variance in closed form.
  • The kernel encodes smoothness, periodicity, and lengthscale assumptions.
  • Marginal likelihood enables principled hyperparameter selection.
  • GPs are the infinite-width limit of Bayesian neural networks.

Exercises

Exercise 1. Derive the GP predictive equations (6)–(7) from the conditional Gaussian formula.

Exercise 2. Show that the GP predictive mean at training points equals the observations (for the noise-free case).

Exercise 3. Prove that the SE kernel k(x,x′)=exp⁡(−∥x−x′∥2/2ℓ2)k(\mathbf{x}, \mathbf{x}') = \exp(-\lVert\mathbf{x}-\mathbf{x}'\rVert^2/2\ell^2) corresponds to an infinite-dimensional feature space.

Exercise 4. Compute the gradient of the log marginal likelihood (11) with respect to a kernel hyperparameter θj\theta_j.

Exercise 5. Show that a GP with linear kernel k(x,x′)=xTx′k(\mathbf{x}, \mathbf{x}') = \mathbf{x}^T\mathbf{x}' recovers Bayesian linear regression.

Next

The Bias-Variance Tradeoff