SmoothQuant: W8A8 Quantization

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. Why W8A8 Matters
5. The Activation Outlier Problem
6. SmoothQuant: Migrating Difficulty
7. Choosing the Migration Strength
8. Per-Token vs Per-Tensor Activation Quantization
9. Implementation & Speedup
10. LLM.int8(): Mixed-Precision Decomposition
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Explain why activation quantization is harder than weight quantization.
2. Derive SmoothQuant's channel-wise scaling transformation.
3. Analyze the migration strength parameter $\alpha$ and its optimal value.
4. Compare W8A8 with W4A16 in terms of actual inference speedup.
5. Describe LLM.int8()'s mixed-precision approach for outlier channels.

Notation

• $\mathbf{s} \in \mathbb{R}^{c}$ — per-channel smoothing factors
• $\alpha \in [0, 1]$ — migration strength
• W$n$A$m$ — $n$-bit weights, $m$-bit activations

Core Intuition

Weight-only quantization (W4A16, W8A16) reduces memory but doesn't speed up computation — activations are still in FP16, and the matmul itself uses FP16 arithmetic. For actual COMPUTE speedup, both weights AND activations must be quantized to INT8, enabling INT8 tensor cores (2x throughput on A100/H100). The challenge: activations have outlier channels that make direct quantization catastrophic.

Why W8A8 Matters

Memory bandwidth vs compute:

• W4A16: Reduces weight loading (memory-bound regime). No compute speedup.
• W8A8: Both operands in INT8 → use INT8 tensor cores → 2x compute throughput.

When compute-bound (large batch, long sequence): W8A8 provides actual speedup. When memory-bound (batch=1, short prompt): W4A16 is sufficient.

Hardware support:

• NVIDIA A100: INT8 tensor cores (624 TOPS vs 312 TFLOPS FP16).
• NVIDIA H100: INT8 at 1979 TOPS vs 989 TFLOPS FP16.
• 2x peak throughput for quantized operations.

The Activation Outlier Problem

Discovery (Dettmers et al., 2022): In LLMs above 6B parameters, specific hidden dimensions develop extreme outlier activations.

Properties:

• Systematic: same channels are outliers across ALL tokens and layers.
• Magnitude: 10-100x larger than other channels.
• Sparse: only 1-5% of channels are outliers (but they're critical).

Impact on quantization: One outlier channel at magnitude 100 with other channels at magnitude 1:

• Per-tensor scale: $s = 100/127 \approx 0.79$.
• Non-outlier channels quantized to: $1/0.79 \approx 1.3$ → rounded to 1.
• Effective precision: 2-3 levels for 99% of channels.

SmoothQuant: Migrating Difficulty

Key insight (Xiao et al., 2023): Activations are hard to quantize (outliers); weights are easy (smooth distribution). Migrate the quantization difficulty from activations to weights via per-channel scaling.

Transformation:

Effect:

• $\hat{\mathbf{X}} = \mathbf{X}\text{diag}(\mathbf{s})^{-1}$: Outlier channels divided by large $s_j$ → smoothed.
• $\hat{\mathbf{W}} = \text{diag}(\mathbf{s})\mathbf{W}$: Corresponding weight columns scaled up → slightly harder to quantize, but weights can handle it.

Result: Both $\hat{\mathbf{X}}$ and $\hat{\mathbf{W}}$ are easy to quantize to INT8.

Choosing the Migration Strength

Smoothing factor:

where $X_j$ is column $j$ of activations and $W_j$ is row $j$ of weights.

$\alpha$ controls migration:

• $\alpha = 0$: No smoothing (original problem).
• $\alpha = 0.5$: Equal difficulty for both (geometric mean).
• $\alpha = 1$: All difficulty on weights.

Optimal: $\alpha = 0.5$ for most LLMs. For very outlier-heavy models (OPT-175B): $\alpha = 0.75$.

Calibration: Compute $\max(|X_j|)$ over a small calibration dataset (128 samples typical).

Per-Token vs Per-Tensor Activation Quantization

Per-tensor: One scale for entire activation matrix. Cheapest but most error.

Per-token (per-row): One scale per token (row of activation matrix).

Why per-token works: Different tokens have different magnitudes (e.g., special tokens vs content tokens). Per-token handles this without extra kernel overhead.

Combined with SmoothQuant: Per-token activation scaling + per-channel weight scaling = good W8A8 quality.

Implementation & Speedup

Inference pipeline:

1. Offline: Compute smoothing factors from calibration data. Pre-multiply weights by $\text{diag}(\mathbf{s})$.
2. Online: Quantize activation (per-token scale). INT8 matmul. Dequantize output.

Actual speedups (A100, OPT-175B):

• W8A8 SmoothQuant: 1.56x speedup over FP16.
• Memory: 2x reduction (both weights and activations in INT8).
• Quality: less than 0.5% accuracy loss on downstream tasks.

Supported frameworks: TensorRT-LLM, vLLM (with CUTLASS INT8 kernels), FasterTransformer.

LLM.int8(): Mixed-Precision Decomposition

Alternative approach (Dettmers et al., 2022): Instead of smoothing, handle outliers separately:

1. Identify outlier dimensions (magnitude exceeds threshold 6.0).
2. Extract outlier columns → compute in FP16.
3. Remaining columns → compute in INT8.
4. Sum the two results.

Tradeoff vs SmoothQuant:

• LLM.int8(): No calibration needed; slightly slower (two matmuls).
• SmoothQuant: Needs calibration; faster (single INT8 matmul).

Common Pitfalls

Pitfall 1. Assuming W8A8 always gives 2x speedup. Speedup depends on being compute-bound. At batch=1 (memory-bound), W4A16 may be faster than W8A8.

Pitfall 2. Using per-tensor activation quantization without SmoothQuant. Outliers will dominate the scale and destroy quality. Always smooth first.

Pitfall 3. Calibrating smoothing factors on unrepresentative data. The outlier pattern depends on the input domain. Use data similar to deployment inputs.

Summary

• W8A8 enables INT8 tensor cores → 2x compute speedup (when compute-bound).
• Activation outlier problem: 1-5% of channels have 100x magnitude → destroys naive quantization.
• SmoothQuant: Migrate difficulty from activations to weights via per-channel scaling.
• $\alpha = 0.5$: Equal difficulty split (optimal for most models).
• LLM.int8(): Handle outliers separately in FP16 (no calibration needed).
• Choose based on: compute-bound vs memory-bound, calibration feasibility, framework support.

Exercises

Exercise 1. For activations with channel magnitudes $[1, 1, 1, 100, 1]$ and weights with column magnitudes $[0.5, 0.5, 0.5, 0.5, 0.5]$: compute the SmoothQuant factors with $\alpha=0.5$.

Exercise 2. After smoothing with the factors from Exercise 1: compute the new activation and weight magnitudes. Verify both are now quantizable.

Exercise 3. Compare the wall-clock speedup of W8A8 vs W4A16 on an A100 for batch sizes 1, 16, and 128 (use roofline model).

Exercise 4. For LLM.int8(): if 3% of channels are outliers, what fraction of compute remains in FP16? Is this significant?

Exercise 5. Design an experiment to find the optimal $\alpha$ for a specific model. What metric should you optimize?