Transformers & Attention

Scaled Dot-Product Attention

The foundational attention mechanism: derivation of query-key-value formulation, the scaling factor, softmax temperature, attention as soft dictionary lookup, and computational complexity analysis.

Types of Attention Mechanisms

A comprehensive taxonomy of attention: multi-head, multi-query, grouped-query, linear attention, sparse attention, sliding window, flash attention, and cross-attention — with complexity analysis and use cases for each.

Multi-Head Attention: Full Mechanics

Complete derivation of multi-head attention: why multiple heads, the projection matrices, concatenation, parameter count, and how heads specialize — positional, syntactic, and semantic attention patterns.

Sliding Window Attention

Fixed-window local attention: derivation, effective receptive field across layers, dilated variants, global+local hybrid patterns (Longformer), and memory-complexity analysis for long-context models.

Flash Attention

IO-aware exact attention: the tiling algorithm, online softmax trick, memory hierarchy exploitation, FlashAttention-2 improvements, and FlashAttention-3 with FP8 — achieving 2-4x speedup without approximation.

Linear Attention & Kernel Methods

Breaking the quadratic barrier: kernel decomposition of softmax, feature maps (elu, random Fourier), causal linear attention, connection to RNNs, RetNet, RWKV, and the expressiveness-efficiency tradeoff.

Sparse Attention Patterns

Sub-quadratic attention through sparsity: strided patterns, block-sparse, BigBird (random+window+global), Routing Transformer, hash-based attention (Reformer), and learned sparsity — complexity proofs for each.

Cross-Attention & Conditioning

Attention between two sequences: encoder-decoder cross-attention, text-to-image conditioning (Stable Diffusion), retrieval-augmented generation, adapter cross-attention, and the mathematical properties of cross vs self attention.

Types of Transformer Architectures

Encoder-only, decoder-only, and encoder-decoder: mathematical formulation of each variant, their training objectives, causal vs bidirectional attention, and when to use which architecture.

Positional Encoding & RoPE

Why transformers need positional information, derivation of sinusoidal encoding, learned embeddings, Rotary Position Embedding (RoPE), ALiBi, and how position representations enable length generalization.

Optimizing Transformer Inference & Training

Techniques for making transformers faster and more memory-efficient: operator fusion, mixed precision, gradient checkpointing, tensor parallelism, sequence parallelism, speculative decoding, and continuous batching.

Speculative Decoding

Using a small draft model to accelerate large model inference: the acceptance-rejection algorithm, theoretical speedup bounds, draft model selection, tree-based speculation, and Medusa multi-head decoding.

Continuous Batching & Serving

Iteration-level scheduling for LLM serving: static vs continuous batching, PagedAttention memory management, prefill-decode disaggregation, request scheduling policies, and throughput optimization for production systems.

Quantization for Transformer Inference

Reducing precision for faster inference: INT8/INT4 weight quantization, activation quantization, GPTQ, AWQ, SmoothQuant, per-channel vs per-group calibration, and the theory of quantization error bounds.

KV-Cache: Theory & Derivation

Why autoregressive transformers need a KV-cache: derivation from the attention computation, memory growth analysis, the relationship between KV-cache size and inference latency, and how cache enables O(Td) per-step generation.

KV-Cache Optimization Techniques

Advanced KV-cache management: PagedAttention, prefix caching, KV quantization (KV-INT4/INT8), token eviction strategies, sliding window cache, multi-query sharing, and cache compression via low-rank projections.