Quantization for Transformer Inference

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Uniform Quantization Theory
5. Weight-Only Quantization (W4A16)
6. GPTQ: Optimal Brain Quantization
7. AWQ: Activation-Aware Weight Quantization
8. SmoothQuant (W8A8)
9. Per-Channel vs Per-Group Calibration
10. Error Analysis and Bounds
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Derive uniform quantization and compute the reconstruction error.
2. Explain why weight-only quantization is effective for memory-bound inference.
3. Derive the GPTQ algorithm from optimal brain surgery.
4. Explain how AWQ exploits activation magnitudes to protect salient weights.
5. Prove how SmoothQuant migrates quantization difficulty from activations to weights.

Notation

• $s$ — scale factor
• $z$ — zero-point
• $b$ — bit-width (4 or 8)
• $\mathbf{H}$ — Hessian of the layer-wise loss
• $g$ — group size for per-group quantization

Core Intuition

A 70B model in FP16 requires 140 GB — too large for most GPUs. Quantization represents the same weights with fewer bits (INT4 = 4 bits per weight → 35 GB). For memory-bound inference, this directly translates to speedup: less data to load from memory means faster generation. The challenge: minimize the accuracy loss from reduced precision.

Uniform Quantization Theory

Forward quantization:

Dequantization:

Scale and zero-point from range $[w_{\min}, w_{\max}]$:

Symmetric quantization (zero-point = 0): $s = \max(|w|) / (2^{b-1} - 1)$.

Quantization error for a single weight:

Weight-Only Quantization (W4A16)

Strategy: Quantize weights to INT4; keep activations in FP16.

Why it works for inference: Generation is memory-bound → loading weights is the bottleneck. INT4 weights are 4x smaller → 4x less memory bandwidth → ~4x speedup.

Computation: Dequantize weights on-the-fly during matrix multiply:

Modern GPU kernels fuse dequantization with GEMM — negligible overhead.

GPTQ: Optimal Brain Quantization

Problem: Quantize each weight to minimize the output error of the layer:

Algorithm (column-by-column):

1. For column $j$: quantize weight $w_j$, compute error $\delta_j = w_j - \hat{w}_j$.
2. Compensate: Distribute the error to remaining unquantized columns:

where $\mathbf{H} = \mathbf{X}\mathbf{X}^T$ is the Hessian of the layer loss.

Intuition: After quantizing one weight with error, adjust remaining weights to compensate, weighted by their sensitivity (Hessian).

Result: Near-lossless INT4 quantization for models up to 175B parameters.

AWQ: Activation-Aware Weight Quantization

Key observation: Not all weights are equally important. Weights that correspond to large activation channels have more impact on the output.

Salient weight identification: For each weight column $j$, compute the average activation magnitude:

Strategy: Protect important weights by per-channel scaling before quantization:

Choose $\alpha_j$ larger for salient channels → reduces their quantization error at the expense of less important channels.

Optimal scaling: $\alpha_j^* = \sqrt{\mathbb{E}[x_j^2] / \max(|w_j|)}$ (balances activation and weight quantization ranges).

SmoothQuant (W8A8)

Problem: Activations have outlier channels (values 100x larger than average), making activation quantization difficult. Weights are well-behaved.

Solution: Migrate the quantization difficulty from activations to weights:

where $s_j = \max(|x_j|)^\alpha / \max(|w_j|)^{1-\alpha}$ balances the ranges.

Effect: Smoothed activations $\hat{\mathbf{X}} = \mathbf{X}\text{diag}(\mathbf{s})^{-1}$ have smaller outliers (easier to quantize). Scaled weights $\hat{\mathbf{W}} = \text{diag}(\mathbf{s})\mathbf{W}$ absorb the outlier magnitude.

Result: Enables INT8 quantization for both weights AND activations (W8A8) → can use INT8 tensor cores for 2x compute speedup.

Per-Channel vs Per-Group Calibration

Per-tensor: One $(s, z)$ for the entire weight matrix. Coarse; high error.

Per-channel: One $(s, z)$ per output channel. Better accuracy; standard for INT8.

Per-group: One $(s, z)$ per group of $g$ weights (e.g., $g=128$). Fine-grained; standard for INT4.

For $g=128, b=4$: overhead = $4/(128 \times 0.5) = 6.25\%$ additional storage.

Tradeoff: Smaller $g$ → better accuracy, more overhead. $g=128$ is the standard for INT4; $g=32$ for INT3.

Error Analysis and Bounds

Layer-wise error: The output error from quantizing layer $l$:

Error accumulation across layers: Errors compound:

Empirical observation: INT8 quantization typically adds $< 0.1\%$ perplexity increase. INT4 with GPTQ/AWQ: $< 1\%$ perplexity increase for models $\geq 7B$.

Smaller models are harder to quantize: The relative error $\|\Delta\mathbf{W}\|/\|\mathbf{W}\|$ is larger when there are fewer weights to distribute the error across.

Common Pitfalls

Pitfall 1. Quantizing small models ($< 3B$) to INT4. The quality degradation is substantial — INT8 is safer for small models.

Pitfall 2. Ignoring calibration data quality. GPTQ and AWQ require representative calibration data. Using random data or a mismatched domain leads to poor quantization.

Pitfall 3. Assuming W4A16 provides 4x compute speedup. It provides 4x memory bandwidth reduction (faster weight loading) but activations are still FP16, so tensor core utilization doesn't improve. W8A8 (SmoothQuant) provides actual compute speedup via INT8 tensor cores.

Summary

• W4A16 (GPTQ, AWQ): 4x memory reduction → 4x bandwidth speedup. Near-lossless for large models.
• W8A8 (SmoothQuant): 2x memory + 2x compute speedup via INT8 tensor cores.
• GPTQ: Hessian-aware error compensation; optimal per-column quantization.
• AWQ: Protect salient weights via activation-aware scaling.
• SmoothQuant: Migrate quantization difficulty from activations to weights.
• Per-group quantization ($g=128$) is standard for INT4 quality.

Exercises

Exercise 1. Compute the memory size of LLaMA-70B in FP16, INT8, and INT4 (with per-group $g=128$ overhead).

Exercise 2. Derive the optimal scale factor $s$ that minimizes MSE for symmetric quantization of a uniform distribution $w \sim U[-a, a]$.

Exercise 3. Explain why GPTQ processes columns in order and compensates remaining columns, rather than quantizing all independently.

Exercise 4. For SmoothQuant with $\alpha = 0.5$: compute the smoothing factor $s_j$ given $\max(|x_j|) = 100$ and $\max(|w_j|) = 0.5$.

Exercise 5. Compute the effective bits-per-weight for INT4 quantization with group size 128, accounting for the FP16 scale and zero-point overhead.