Quantization Fundamentals

Table of Contents

1. Learning Objectives
2. Prerequisites
3. Notation
4. Core Intuition
5. Number Representation
6. Uniform Quantization
7. Quantization Error Analysis
8. Affine Quantization Mapping
9. Per-Tensor vs. Per-Channel Quantization
10. Post-Training vs. Quantization-Aware Training
11. Weight vs. Activation Quantization
12. Worked Examples
13. Connection to the Broader Curriculum
14. Common Pitfalls and Misconceptions
15. Research Perspective
16. Summary of Takeaways
17. Exercises

Learning Objectives

After reading this chapter, you should be able to:

1. Define uniform quantization with scale $s$ and zero-point $z$.
2. Derive affine mapping $q = \text{round}(x/s) + z$ and dequantization $\hat{x} = s(q - z)$.
3. Bound quantization error $|x - \hat{x}| \leq s/2$ for uniform quantizers.
4. Compare PTQ vs. QAT theoretically.
5. Explain why per-channel quantization preserves accuracy for weight matrices.

Prerequisites

• Matrix Operations — linear layers
• Backpropagation — gradient flow in QAT

Notation

• $x, \hat{x}$ — Full-precision and quantized values
• $s, z$ — Scale and zero-point
• $b$ — Bit width (INT8, INT4, etc.)
• $\mathrm{round}(\cdot)$ — Quantization rounding operator
• $\epsilon_q$ — Quantization error

Core Intuition

Neural networks use FP32/FP16 weights — 32 or 16 bits per parameter. Quantization maps weights and activations to low-bit integers (INT8, INT4), reducing memory and enabling faster integer arithmetic.

A 70B model in FP16: 140 GB. INT4: ~35 GB — runnable on consumer GPUs. Combined with LoRA (QLoRA) and Flash Attention, quantization is essential for deployment.

Series context. Volume IV, Chapter 18 (Quantization).

Number Representation

Definition 1 (Floating Point). FP16: 1 sign + 5 exponent + 10 mantissa bits. Dynamic range via exponent.

Definition 2 (Fixed Point / Integer). INT8: values in $\{-128, \ldots, 127\}$. Fixed scale.

Proposition 1. INT8 has 4× fewer bits than FP32 → 4× memory reduction (ignoring overhead).

Uniform Quantization

Definition 3 (Uniform Quantizer). Map real $x \in [x_{\min}, x_{\max}]$ to integer $q \in \{0, \ldots, 2^b - 1\}$:

where $s$ is scale, $z$ is zero-point, $b$ is bit-width.

Definition 4 (Symmetric Quantization). $z = 0$, $s = \max(|x|) / (2^{b-1} - 1)$.

Quantization Error Analysis

Theorem 1 (Uniform Quantization Error). For round-to-nearest uniform quantization:

Proof. Rounding error bounded by half quantization bin width. $\blacksquare$

Proposition 2 (MSE Error). For uniformly distributed $x$ in bin: $\mathbb{E}[(x-\hat{x})^2] \leq s^2/12$.

Corollary 1. Smaller scale $s$ (more bits or narrower range) → lower error.

Affine Quantization Mapping

Definition 5 (Scale and Zero-Point). For asymmetric INT8:

Maps $x_{\min} \to 0$, $x_{\max} \to 255$ exactly.

Proposition 3 (Quantized Matmul). Integer matrix multiply with rescaling:

Enables INT8 GEMM on hardware accelerators.

Per-Tensor vs. Per-Channel Quantization

Definition 6 (Per-Channel). Separate $(s, z)$ per output channel of weight matrix $\mathbf{W} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$.

Proposition 4. Per-channel preserves accuracy for weights with heterogeneous magnitude distributions across channels — critical for Self-Attention projection matrices.

Post-Training vs. Quantization-Aware Training

Definition 7 (PTQ). Quantize after training using calibration data to estimate $x_{\min}, x_{\max}$.

Definition 8 (QAT). Simulate quantization during training with straight-through estimator (STE):

Proposition 5. QAT allows model to adapt to quantization noise; PTQ is faster but lower accuracy at low bit-widths.

Weight vs. Activation Quantization

• Weights — Static (known at load) — Outlier channels
• Activations — Dynamic (runtime) — Range varies per input

Definition 9 (Weight-Only Quantization). Quantize $\mathbf{W}$ to INT4; activations remain FP16 — used in GPTQ, AWQ for LLMs.

Worked Examples

Example 1: INT8 Mapping

$x \in [0, 10]$, $b=8$: $s = 10/255 \approx 0.039$, $x=5 \to q = 127$, $\hat{x} \approx 4.96$.

Example 2: Memory

7B params × 4 bits = 3.5 GB vs. 14 GB FP16.

Connection to the Broader Curriculum

• LoRA — QLoRA
• Flash Attention — composable
• Data Parallelism — distributed quantized inference
• Linear Regression — quantization error as noise

Common Pitfalls and Misconceptions

Pitfall 1: INT8 always 4× faster (depends on hardware/kernel).

Pitfall 2: Ignoring outliers in activation ranges.

Pitfall 3: PTQ sufficient for INT4 (often needs QAT or advanced PTQ).

Pitfall 4: Quantizing KV Cache separately from weights.

Research Perspective

GPTQ (Frantar et al., 2022). AWQ (Lin et al., 2023). FP8 training (NVIDIA H100). LLM.int8() mixed precision.

Summary of Takeaways

• Quantize — $q = \text{round}(x/s) + z$
• Dequantize — $\hat{x} = s(q - z)$
• Error bound — $— x - \hat{x} — \leq s/2$
• PTQ vs QAT — Calibration vs. STE training

Next: Data Parallelism

Exercises

Exercise 1. Derive (4) for asymmetric INT8.

Exercise 2. Prove error bound (3).

Exercise 3. Compute memory for 70B model at INT4 vs FP16.

Exercise 4. Why per-channel for weights?

Exercise 5. STE gradient justification.

Exercise 6. Quantized matmul (5) derivation.

Exercise 7. Outlier effect on scale $s$.

Exercise 8. QLoRA: compose LoRA + INT4.