Round-to-Nearest & Basic Weight Quantization

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Round-to-Nearest (RTN)
5. Per-Tensor vs Per-Channel
6. Absmax (Symmetric) Quantization
7. Zero-Point (Asymmetric) Quantization
8. When RTN Works
9. When RTN Fails: The Outlier Problem
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Implement symmetric and asymmetric round-to-nearest quantization.
2. Compute scale and zero-point from calibration data.
3. Explain the quality degradation from per-tensor to per-channel to per-group granularity.
4. Identify when RTN is sufficient (INT8 for large models) vs insufficient (INT4).
5. Diagnose the activation outlier problem that breaks naive quantization.

Notation

• $q$ — quantized integer value
• $s$ — scale factor
• $z$ — zero-point offset
• $[q_{\min}, q_{\max}]$ — quantization range (e.g., $[-128, 127]$ for INT8)

Core Intuition

Round-to-nearest is the simplest approach: find a linear mapping from the weight range to the integer range, round each weight to its nearest integer level, and store the integer + scale factor. It works surprisingly well for INT8 on large models because (1) INT8 has 256 levels (fine-grained enough) and (2) individual rounding errors are independent and roughly cancel across a matrix-vector product.

Round-to-Nearest (RTN)

Quantize:

Dequantize:

Error per weight: $|w - \hat{w}| \leq s/2$.

Total error for a layer $\mathbf{y} = \mathbf{Wx}$:

where $n$ is the number of weights per output neuron.

Per-Tensor vs Per-Channel

Per-tensor: One scale $s$ for the entire weight matrix.

Problem: If one row has weights in $[-0.01, 0.01]$ and another in $[-1, 1]$, the first row gets quantized with 100x less precision than it needs.

Per-channel (per-output-channel): One scale per row of $\mathbf{W}$:

Each row uses its full quantization range independently.

Per-group: One scale per $g$ consecutive elements within a row:

Quality ranking: Per-group ($g=32$) > Per-group ($g=128$) > Per-channel > Per-tensor.

Storage overhead: Each scale is FP16 (2 bytes). Per-group with $g=128$ and INT4 weights: overhead = $2/(128 \times 0.5) = 3.1\%$.

Absmax (Symmetric) Quantization

Symmetric: Zero-point $z = 0$, range centered at 0.

For INT8: $s = \max(|w|) / 127$.

Advantage: Simpler computation (no zero-point offset). Kernel implementation is faster.

Disadvantage: Wastes half the range if weights are all positive (e.g., after ReLU). The $[-128, -1]$ range is unused.

Best for: Weights (roughly symmetric around zero in well-trained models).

Zero-Point (Asymmetric) Quantization

Asymmetric: Use full range $[q_{\min}, q_{\max}]$ for the actual weight range $[w_{\min}, w_{\max}]$:

Advantage: Uses all quantization levels — better precision for asymmetric distributions.

Disadvantage: Extra computation (subtract zero-point in dequantization). Slightly more complex kernels.

Best for: Activations (often non-negative after ReLU or asymmetrically distributed).

When RTN Works

INT8 per-channel on models above 1B parameters: Typically less than 0.5% accuracy loss.

Why it works:

1. 256 levels is sufficient to represent most weight distributions.
2. Per-channel scaling handles cross-channel variance.
3. Large models have redundancy — small per-weight errors are absorbed.
4. Rounding errors are approximately unbiased and independent.

Error scaling: For a matmul $\mathbf{y} = \mathbf{Wx}$ with $n$ input features, random quantization errors average out: $\|\Delta\mathbf{y}\| \propto s\sqrt{n}/2$ (central limit theorem).

When RTN Fails: The Outlier Problem

INT4 RTN on any model: Significant quality degradation (5-20% accuracy loss).

Why it fails:

1. Only 16 levels — too coarse for most weight distributions.
2. Outlier weights force large scale → poor precision for majority of weights.

Activation outliers (Dettmers et al., 2022):

• In LLMs, certain channels develop activation magnitudes 100x larger than others.
• Per-tensor quantization of activations: outlier channels dominate the scale.
• Remaining channels get quantized to near-zero precision.

Example: If 99% of activations are in $[-1, 1]$ but one channel hits 100:

• Scale = $100/127 \approx 0.79$.
• Values in $[-1, 1]$ only use levels $[-1, 0, +1]$ — catastrophic precision loss.

Common Pitfalls

Pitfall 1. Applying per-tensor RTN to INT4. At 4 bits, per-tensor is far too coarse. Always use per-group ($g \leq 128$) for INT4.

Pitfall 2. Using symmetric quantization for post-ReLU activations. All values are non-negative; symmetric wastes half the range. Use asymmetric.

Pitfall 3. Assuming RTN quality from INT8 papers transfers to INT4. The jump from 256 to 16 levels is qualitatively different — advanced methods (GPTQ, AWQ) become essential.

Summary

• RTN: Round each weight to nearest quantization level. Simplest method.
• Symmetric (absmax): Centered at zero; simpler; best for weights.
• Asymmetric (zeropoint): Uses full range; better for non-symmetric distributions.
• Granularity: Per-tensor → per-channel → per-group (increasing quality and overhead).
• Works well: INT8 per-channel on models above 1B.
• Fails: INT4 per-tensor; activations with outliers.
• Advanced methods (GPTQ, AWQ) are needed when RTN quality is insufficient.

Exercises

Exercise 1. Quantize the weight vector $[0.23, -0.15, 0.42, -0.31, 0.08]$ to INT4 symmetric. Compute the reconstruction error.

Exercise 2. Compare per-tensor vs per-channel INT8 quantization error for a $4096 \times 4096$ weight matrix where row magnitudes vary by 10x.

Exercise 3. For INT4 with group size 128: compute total storage (weights + scales) for a 7B model with $d=4096$, and compare to FP16.

Exercise 4. Given activation values $[0.1, 0.3, 0.2, 50.0, 0.4, 0.1]$: show how one outlier destroys per-tensor INT8 quantization of the remaining values.

Exercise 5. Derive the expected MSE of RTN quantization for weights drawn from $\mathcal{N}(0, \sigma^2)$ at $b$ bits with optimal symmetric range.