Machine Learning

Loss Functions & Risk Minimization

The mathematical framework connecting loss functions, empirical risk, and statistical risk — deriving squared loss from Gaussian noise, cross-entropy from MLE, and establishing PAC-style generalization bounds.

Overfitting, Underfitting & Regularization

A rigorous treatment of model complexity control: why overparameterized models memorize, how regularization constrains the hypothesis class, Tikhonov regularization as MAP estimation, and the double descent phenomenon.

The Bias-Variance Tradeoff

Volume I, Chapter 3 — Part VI. Rigorous derivation of the bias-variance decomposition, analysis for linear and k-NN estimators, model complexity, regularization, double descent in overparameterized models, and implications for deep learning generalization.

Linear Regression

Volume II, Chapter 4 — Part I. A complete theoretical treatment of linear regression: the probabilistic model, least-squares derivation, normal equations, geometric interpretation as orthogonal projection, regularization theory, and the bias–variance connection.

Logistic Regression

Volume II, Chapter 4 — Part II. Binary classification via the logistic link function: Bernoulli likelihood, cross-entropy loss derivation, gradient structure, multiclass softmax extension, and the connection to neural network output layers.

Ridge & Lasso Regression

Complete derivation of penalized least squares: the ridge closed-form solution, spectral shrinkage, Lasso sparsity via subgradient conditions, coordinate descent, solution path behavior, and the geometry of constrained optimization.

Support Vector Machines

Derivation of hard and soft margin SVMs from maximum-margin geometry, the dual formulation via Lagrangian optimization, KKT conditions, the kernel trick, and the representer theorem connecting SVMs to reproducing kernel Hilbert spaces.

Decision Trees

The mathematical framework of recursive partitioning: impurity measures (Gini, entropy, MSE), optimal split criteria, the greedy induction algorithm, pruning theory, and computational complexity analysis.

Bayesian Linear Regression

Volume II, Chapter 5 — Part I. Full Bayesian treatment of linear regression: conjugate Gaussian prior, closed-form posterior derivation, predictive distribution with epistemic and aleatoric uncertainty, MAP–Ridge equivalence, sequential updating, and the kernel dual form leading to Gaussian processes.

Gaussian Processes

The function-space view of Bayesian learning: defining GPs via mean and covariance functions, deriving the predictive distribution in closed form, kernel design, marginal likelihood for hyperparameter optimization, and connections to neural networks.

Markov Chain Monte Carlo

Rigorous derivation of MCMC methods for Bayesian inference: the Metropolis-Hastings algorithm, detailed balance, ergodicity, Gibbs sampling as a special case, Hamiltonian Monte Carlo, diagnostics, and convergence theory.

Variational Inference

Volume II, Chapter 5 — Part II. From intractable Bayesian posteriors to tractable optimization: ELBO derivation, KL gap, mean-field approximation, CAVI, reparameterization gradients, amortized inference, and the theoretical foundation of VAEs.

Bagging & Random Forests

The variance-reduction principle behind bootstrap aggregation, derivation of the out-of-bag error estimate, the random subspace method, and a formal analysis of how decorrelation between trees improves generalization.

Gradient Boosting

Derivation of boosting as functional gradient descent in function space: the additive model framework, residual fitting, gradient boosting for arbitrary differentiable losses, shrinkage, subsampling, and the XGBoost second-order formulation.

Cross-Validation & Model Selection

The theory behind model evaluation and selection: hold-out estimation, K-fold cross-validation bias-variance tradeoff, leave-one-out CV, nested CV for hyperparameter tuning, and information criteria (AIC, BIC) as asymptotic approximations.

Classification Metrics

A formal treatment of evaluation metrics for classification: precision, recall, F-scores, ROC curves, AUC, calibration, and the mathematical relationships between confusion matrix quantities and probabilistic thresholds.