Model Optimization

Pruning & Structured Sparsity

Removing redundant parameters: magnitude pruning, movement pruning, structured vs unstructured, the lottery ticket hypothesis, SparseGPT, and Wanda — achieving 50-70% sparsity without quality loss.

Structured Pruning: Channels, Heads & Layers

Removing entire structural units from neural networks: channel pruning in CNNs, attention head pruning in transformers, layer removal, importance scoring (Taylor expansion, sensitivity), and structured sparsity with hardware acceleration.

Efficient Model Architectures

Designing models for efficiency: MobileNet depthwise separable convolutions, EfficientNet scaling, mixture of experts, early exit, dynamic computation, and the Pareto frontier of accuracy vs FLOPs.

Mixture of Experts (MoE): Sparse Scaling

Scaling model capacity without proportional compute: MoE routing, top-k gating, load balancing, expert specialization, Switch Transformer, Mixtral architecture, and training challenges (routing collapse, expert imbalance).

Speculative Decoding & Draft-Verify Inference

Accelerating autoregressive inference without quality loss: speculative decoding theory, draft model selection, acceptance-rejection sampling, Medusa multi-head prediction, EAGLE, lookahead decoding, and self-speculative methods.

Neural Architecture Search: Theory & Methods

Automating network design: search spaces, search strategies (RL, evolutionary, gradient-based), weight sharing (supernets, one-shot NAS), DARTS differentiable search, hardware-aware NAS, and EfficientNet/MobileNet design.

KV-Cache Optimization: Memory-Efficient Inference

Managing the KV-cache bottleneck: PagedAttention (vLLM), continuous batching, KV-cache compression (quantization, eviction policies), chunked prefill, multi-query/grouped-query attention, and cache-aware scheduling.

LLM Serving Systems: vLLM, TensorRT-LLM & SGLang

Production serving frameworks: request scheduling, memory management, batching strategies, prefix caching, constrained decoding, tensor parallelism for serving, and performance benchmarking (TTFT, TPS, throughput).

Compression via Distillation & Low-Rank Factorization

Combining compression techniques: task-specific distillation for deployment, low-rank factorization (SVD, LoRA at inference), weight tying, vocabulary reduction, and progressive compression strategies for minimal quality loss.

Activation Optimization: Checkpointing, Offloading & Recomputation

Reducing memory during training and inference: gradient/activation checkpointing, CPU/NVMe offloading, selective recomputation, Flash Attention memory savings, and memory-compute tradeoff analysis.

Hardware-Aware Optimization: Roofline, MFU & Kernel Fusion

Maximizing GPU utilization: the roofline model, Model FLOP Utilization (MFU), arithmetic intensity, kernel fusion strategies, operator scheduling, and bridging the gap between theoretical and achieved performance.

Edge Deployment: Mobile, Browser & Embedded Inference

Running models on resource-constrained devices: ONNX Runtime, TensorFlow Lite, CoreML, WebAssembly/WebGPU inference, on-device quantization, model architecture choices for edge, and latency/power/memory budget analysis.