Systems & Optimization

Flash Attention

Volume IV, Chapter 17 — Part I. IO-aware exact attention: SRAM vs. HBM memory hierarchy, tiling strategy, online softmax, and O(N) memory complexity derivation.

Gradient Checkpointing & Activation Recomputation

Trading compute for memory: the checkpointing algorithm, optimal checkpoint placement, selective recomputation, memory savings analysis, and integration with pipeline/tensor parallelism.

Mixed Precision Training

FP16, BF16, and FP8 training: the IEEE floating-point formats, loss scaling for FP16, why BF16 dominates modern training, the master weight copy, and upcoming FP8 training on Hopper GPUs.

Quantization Fundamentals

Volume IV, Chapter 18 — Part I. Number representation, uniform and non-uniform quantization, quantization error analysis, affine mapping INT8, and post-training vs. quantization-aware training theory.

Data Parallelism

Volume IV, Chapter 19 — Part I. Distributed training via data parallelism: gradient aggregation, AllReduce, synchronous SGD theory, communication complexity, and scaling efficiency analysis.

Tensor Parallelism

Splitting individual layers across GPUs: column-parallel and row-parallel linear layers, Megatron-LM style partitioning, communication patterns (all-reduce), and scaling efficiency analysis.

Pipeline Parallelism

Splitting model layers across GPUs: naive pipeline with bubble overhead, GPipe micro-batching, 1F1B schedule, interleaved stages, and the bubble fraction analysis for optimal pipeline configuration.

ZeRO: Zero Redundancy Optimizer

DeepSpeed ZeRO stages 1-3: partitioning optimizer states, gradients, and parameters across data-parallel ranks, communication analysis, ZeRO-Offload to CPU/NVMe, and FSDP as PyTorch's ZeRO-3.

LLM Serving Systems & Infrastructure

Production LLM serving: vLLM, TensorRT-LLM, TGI architectures, request routing, model sharding for serving, autoscaling, cost optimization, and SLA-aware scheduling.