Deep Learning

Activation Functions

A rigorous analysis of nonlinear activation functions: the necessity of nonlinearity, derivation of sigmoid/tanh saturation, the ReLU family, smoothness properties, Lipschitz constants, and the connection between activation choice and gradient flow.

Backpropagation

Volume II, Chapter 6 — Part I. The chain rule on computation graphs: forward and backward passes, general layer-wise gradient formulas, vector-Jacobian products, and the theoretical foundation of deep network training.

Universal Approximation Theorem

Volume II, Chapter 6 — Part II. Why neural networks can approximate any continuous function: Cybenko's theorem, constructive proofs, depth vs. width tradeoffs, and the theoretical limits of expressivity.

Weight Initialization

Deriving proper initialization schemes from variance propagation analysis: Xavier/Glorot for sigmoid/tanh, He/Kaiming for ReLU, orthogonal initialization, and the relationship between initialization and gradient flow stability.

Batch Normalization

Volume II, Chapter 7 — Part I. Derivation of batch normalization: forward pass, complete backward pass, scale invariance, internal covariate shift, inference with running statistics, and comparison with LayerNorm and RMSNorm.

Gradient Flow

Volume II, Chapter 8 — Part II. The continuous-time limit of gradient descent: derivation of gradient flow ODEs, closed-form solutions for linear models, trainability analysis via eigenvalues of the Gram matrix, parameter evolution, and the connection to the Neural Tangent Kernel.

Layer Normalization & RMSNorm

Derivation of layer normalization from the perspective of internal covariate shift, comparison with batch normalization, the RMSNorm simplification, and why transformers use LayerNorm while CNNs use BatchNorm.

Residual Connections & Skip Connections

Why identity shortcuts solve the degradation problem: formal analysis of gradient flow in deep networks, the unraveled view of ResNets as exponential ensembles, and the dynamical systems interpretation of residual blocks.

The Convolution Operation

Mathematical foundations of discrete convolution for neural networks: 1D and 2D convolutions as linear operators, equivariance to translation, the relationship between convolution and matrix multiplication, stride, padding, and receptive fields.

Long Short-Term Memory (LSTM)

Complete derivation of the LSTM architecture: the vanishing gradient problem in vanilla RNNs, gate mechanisms (forget, input, output), the cell state as a controlled memory highway, gradient flow analysis, and the GRU simplification.

Self-Attention Mechanism

Volume II, Chapter 8 — Part I (Attention). Scaled dot-product attention from content-based addressing: Q/K/V projections, variance analysis motivating the scale factor, multi-head attention, causal masking, and complexity.

Multi-Head Attention

Complete derivation of multi-head attention: why multiple heads increase expressiveness, the projection geometry, concatenation and linear mixing, computational complexity analysis, and the connection to ensemble kernel methods.

Positional Encoding

Volume II, Chapter 8 — Part II (Chapter 12). Sinusoidal positional encodings from relative-shift desiderata, learned embeddings, RoPE, ALiBi, and length extrapolation theory for Transformers.

The Full Transformer Architecture

Complete specification of the transformer: encoder and decoder stacks, the role of each sublayer (attention, FFN, normalization), residual streams, encoder-decoder cross-attention, and architectural variants (encoder-only, decoder-only).

Variational Autoencoders

Volume II, Chapter 9 — Part I. VAE objective from the ELBO, reparameterization trick proof, Gaussian KL closed form, posterior collapse, β-VAE, and connection to latent diffusion models.

Generative Adversarial Networks

The minimax game formulation: deriving the GAN objective, proving the optimal discriminator, showing that the generator minimizes Jensen-Shannon divergence, training dynamics, mode collapse, and Wasserstein distance improvements.

Dropout & Its Bayesian Interpretation

Rigorous derivation of dropout as a regularizer: the forward pass with random masks, inverted dropout scaling, the connection to model averaging over exponentially many subnetworks, and the MC Dropout interpretation as approximate Bayesian inference.

Generalization Theory & PAC-Bayes

Why do overparameterized networks generalize? Formal treatment of VC dimension, Rademacher complexity, PAC-Bayes bounds, the role of flatness/sharpness of minima, and the lottery ticket hypothesis.