Quantization

Number Formats: FP32, FP16, BF16, FP8, INT8, INT4

Understanding floating-point and integer representations used in neural networks: IEEE 754 formats, the exponent-mantissa tradeoff, dynamic range vs precision, BFloat16 for training, FP8 for inference, and when to use each format.

Quantization Theory & Fundamentals

Mathematical foundations of neural network quantization: uniform and non-uniform quantization, the rate-distortion tradeoff, calibration methods, per-tensor vs per-channel vs per-group scaling, and error analysis.

Round-to-Nearest & Basic Weight Quantization

The simplest quantization method: round-to-nearest (RTN), per-channel vs per-tensor scaling, absmax and zeropoint calibration, and understanding when naive quantization works and when it fails.

GPTQ & AWQ: Post-Training Quantization

State-of-the-art weight quantization methods: GPTQ's Hessian-based error compensation, AWQ's activation-aware scaling, GGUF format, and practical recipes for quantizing LLMs to INT4.

GGUF & Mixed-Precision Quantization

Practical quantization for deployment: the GGUF format, k-quant mixed precision, importance-based bit allocation, llama.cpp quantization types (Q4_K_M, Q5_K_S, Q6_K), and choosing the right quantization level.

SmoothQuant: W8A8 Quantization

Quantizing both weights AND activations to INT8: the activation outlier problem, SmoothQuant's migration strategy, per-channel scaling, and achieving 2x speedup via INT8 tensor core acceleration.

FP8 Training & Inference

Using 8-bit floating-point for both training and inference: FP8 tensor cores on H100, E4M3 for forward, E5M2 for backward, per-tensor scaling, delayed scaling, and achieving near-BF16 quality with 2x speedup.

Quantization-Aware Training (QAT)

Training with quantization in the loop: the straight-through estimator, fake quantization nodes, QAT vs PTQ quality comparison, LSQ (learned step size), and when QAT is worth the training cost.

Extreme Quantization: Binary, Ternary & 1-bit LLMs

Pushing quantization to the limit: binary neural networks (XNOR-Net), ternary weights, BitNet, 1.58-bit LLMs, the scaling laws for extreme quantization, and when sub-4-bit models are viable.

Sub-4-Bit Quantization: AQLM, QuIP#, & HQQ

State-of-the-art methods for 2-3 bit quantization of LLMs: additive quantization (AQLM), incoherence processing (QuIP#), half-quadratic quantization (HQQ), and the information-theoretic limits of compression.

Quantization Hardware: Tensor Cores & Custom Kernels

How quantization maps to hardware: NVIDIA tensor core data types, CUTLASS INT8 kernels, CPU SIMD for quantized inference, Apple Neural Engine, and the hardware-software co-design for efficient quantized computation.

Quantized LLM Serving: vLLM, TRT-LLM & llama.cpp

End-to-end quantized serving systems: vLLM with AWQ/GPTQ, TensorRT-LLM INT4/FP8 engines, llama.cpp CPU inference, quantization-aware batching, and achieving maximum tokens/second at production scale.