Quantization Theory & Fundamentals

Learning Objectives
Notation
Core Intuition
Uniform Quantization
Symmetric vs Asymmetric
Granularity: Per-Tensor, Per-Channel, Per-Group
Calibration Methods
Quantization Error Bounds
Non-Uniform Quantization
Common Pitfalls
Summary
Exercises

Learning Objectives

Derive uniform quantization and compute reconstruction error.
Compare symmetric vs asymmetric quantization tradeoffs.
Explain per-group quantization and its overhead calculation.
Derive optimal calibration ranges (MinMax, percentile, MSE-optimal).
Analyze quantization error propagation through neural network layers.

Notation

$b$ — bit-width
$s$ — scale factor
$z$ — zero-point
$g$ — group size
$\Delta$ — quantization step size

Core Intuition

Quantization maps continuous-valued weights/activations to a discrete set of levels (e.g., 256 levels for INT8, 16 for INT4). Fewer bits = less memory = faster inference (less data to load). The mathematical challenge: minimize the information lost by this discretization while maintaining model accuracy.

Interactive: Weight Quantization

Bit width: 8-bit (256 levels)

FP32 originalQuantizedError

MSE

7.52e-6

Max Error

0.0046

Compression

4.0×

Explore: Reduce bits to see quantization error grow. At 4-bit, you get 8× compression but visible error. At 1-bit (binary), only two values exist. The red lines show individual weight errors — notice outlier weights suffer most.

Uniform Quantization

Quantize (float → int):

x_q = \text{clamp}\left(\text{round}\left(\frac{x}{s}\right) + z, \; 0, \; 2^b - 1\right). \tag{1}

Dequantize (int → float):

\hat{x} = s \cdot (x_q - z). \tag{2}

Parameters:

s = \frac{x_{\max} - x_{\min}}{2^b - 1}, \quad z = \text{round}\left(-\frac{x_{\min}}{s}\right). \tag{3}

Quantization error: $|x - \hat{x}| \leq s/2 = \frac{x_{\max} - x_{\min}}{2(2^b-1)}$ .

Symmetric vs Asymmetric

Symmetric (zero-point $z = 0$ ):

s = \frac{\max(|x_{\max}|, |x_{\min}|)}{2^{b-1} - 1}, \quad x_q = \text{round}(x/s). \tag{4}

Range is symmetric around zero. Simpler computation (no zero-point offset).

Asymmetric (non-zero $z$ ): Uses full quantization range for asymmetric distributions. Better precision when data isn't centered at zero (e.g., ReLU activations are always non-negative).

Rule of thumb: Symmetric for weights (roughly zero-centered). Asymmetric for activations (often non-negative after ReLU).

Granularity: Per-Tensor, Per-Channel, Per-Group

Per-tensor: One $(s, z)$ for entire weight matrix. Coarsest; highest error.

Per-channel (per-output-channel): One $(s, z)$ per row of the weight matrix. Standard for INT8.

Per-group: One $(s, z)$ per group of $g$ consecutive weights:

\text{Overhead} = \frac{2 \times \text{(scale + zero bytes)}}{g \times b/8} = \frac{4}{g \times b/8}. \tag{5}

For INT4 with $g=128$ : overhead = $4/(128 \times 0.5) = 6.25\%$ .

Tradeoff: Finer granularity → better accuracy, more metadata overhead, more complex kernel.

Calibration Methods

Finding optimal $(x_{\min}, x_{\max})$ for quantization range:

MinMax: Use actual min/max values. Simple but sensitive to outliers.

Percentile: Use $p$ -th and $(100-p)$ -th percentile (e.g., $p=0.01$ ). Clips outliers.

MSE-optimal: Find range that minimizes $\mathbb{E}[(x - \hat{x})^2]$ :

(x_{\min}^*, x_{\max}^*) = \arg\min_{a,b} \int_a^b (x - Q(x))^2 p(x)dx + \int_{-\infty}^a (x-a)^2 p(x)dx + \int_b^{\infty}(x-b)^2 p(x)dx. \tag{6}

Cross-entropy optimal: Minimize KL divergence between original and quantized output distributions.

Quantization Error Bounds

Single-layer error: For $\mathbf{y} = \mathbf{Wx}$ , quantization error in output:

\|\Delta\mathbf{y}\| = \|(\hat{\mathbf{W}} - \mathbf{W})\mathbf{x}\| \leq \|\Delta\mathbf{W}\|_F \cdot \|\mathbf{x}\|. \tag{7}

For uniform quantization: $\|\Delta\mathbf{W}\|_F \leq \frac{s}{2}\sqrt{n_{\text{elements}}}$ .

Multi-layer accumulation: Errors compound multiplicatively through layers. Total error grows roughly as $O(\sqrt{L})$ for well-conditioned networks.

Non-Uniform Quantization

Idea: Place quantization levels non-uniformly, with more levels where the weight distribution is dense.

k-means quantization: Find optimal levels $\{c_1, \ldots, c_{2^b}\}$ by minimizing:

\min_{c_i} \sum_j \min_i (w_j - c_i)^2. \tag{8}

Log-scale quantization: Levels spaced logarithmically. Good for distributions with many small values and few large ones (common for weights).

NormalFloat (NF4): Quantization levels chosen for the normal distribution $\mathcal{N}(0, \sigma^2)$ . Used in QLoRA.

Common Pitfalls

Pitfall 1. Using MinMax calibration with outlier-heavy activations. A single outlier 100x larger than typical values wastes most of the quantization range on empty space.

Pitfall 2. Quantizing small models (under 3B) to INT4. Larger models are more robust to quantization error because they have more redundancy.

Pitfall 3. Ignoring activation quantization. Even with INT4 weights, if activations remain in FP16, the memory bandwidth improvement is limited to weight loading (not activation communication).

Summary

Uniform quantization: Linear mapping with scale and zero-point.
Granularity determines accuracy-overhead tradeoff: per-group ( $g=128$ ) is standard for INT4.
Calibration (MinMax, percentile, MSE) determines quantization range.
Error scales as $O(s \cdot \sqrt{n})$ per layer; compounds across layers.
Non-uniform (NF4, k-means) adapts to weight distribution for better precision.

Exercises

Exercise 1. Compute the quantization step size $s$ for INT4 symmetric quantization of weights in range $[-0.5, 0.5]$ .

Exercise 2. Derive the storage overhead of per-group quantization with FP16 scales for $g=64$ and $b=4$ .

Exercise 3. For a Gaussian distribution $\mathcal{N}(0, 1)$ : compute the optimal percentile clipping for INT8 that minimizes MSE.

Exercise 4. Show that per-channel quantization is equivalent to per-group with $g =$ number of input features.

Exercise 5. Compute the NF4 quantization levels for $\mathcal{N}(0, 1)$ (16 levels placed at quantiles).