GPTQ & AWQ: Post-Training Quantization

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. GPTQ: Optimal Brain Quantization
5. The GPTQ Algorithm
6. AWQ: Activation-Aware Quantization
7. SmoothQuant for W8A8
8. Practical Recipes
9. Quality Comparison
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Derive GPTQ from the optimal brain surgeon framework.
2. Explain the column-wise quantization with Hessian compensation.
3. Derive AWQ's salient weight protection via activation-aware scaling.
4. Compare GPTQ vs AWQ vs round-to-nearest in terms of quality and speed.
5. Choose the appropriate quantization method for a given deployment scenario.

Notation

• $\mathbf{H} = \mathbf{X}\mathbf{X}^T$ — Hessian (input correlation matrix)
• $\delta_j$ — quantization error for column $j$
• $\alpha_j$ — per-channel scaling factor (AWQ)

Core Intuition

Naive round-to-nearest quantization independently rounds each weight, ignoring correlations. GPTQ recognizes that quantizing one weight creates error that can be partially compensated by adjusting neighboring weights. AWQ takes a different approach: protect the most important weights (those multiplied by large activations) by scaling them before quantization.

GPTQ: Optimal Brain Quantization

Objective: Minimize layer output error:

Key insight (from OBS): When quantizing weight $w_j$ with error $\delta_j = w_j - \hat{w}_j$, the optimal compensation for remaining weights is:

The GPTQ Algorithm

Process columns left-to-right:

1. For column $j$: quantize $w_j \to \hat{w}_j$ (round to nearest INT4 level).
2. Compute error: $\delta_j = w_j - \hat{w}_j$.
3. Compensate remaining columns: $\mathbf{w}_{j+1:} \leftarrow \mathbf{w}_{j+1:} - \frac{\delta_j}{[\mathbf{H}^{-1}]_{jj}} \cdot \mathbf{H}^{-1}_{j, j+1:}$.
4. Repeat for all columns.

Batch processing: Process 128 columns at a time for GPU efficiency.

Calibration data: 128 random samples from the training set to compute $\mathbf{H}$.

Speed: Quantizes a 70B model in about 4 hours on a single GPU.

AWQ: Activation-Aware Quantization

Observation: Not all weights matter equally. A weight multiplied by a large activation has more impact on the output than one multiplied by a small activation.

Salient channels: Identify channels with large average activation:

Per-channel scaling: Multiply salient weights by $\alpha_j > 1$ before quantization:

Optimal $\alpha$: Search for the scaling that minimizes output error on calibration data.

Advantage over GPTQ: Faster (no sequential column processing), comparable quality, smaller calibration set needed.

SmoothQuant for W8A8

When quantizing both weights AND activations to INT8:

Problem: Activation outliers make activation quantization difficult.

Solution: Migrate difficulty from activations to weights via per-channel scaling:

Smoothed activations are easier to quantize; weights absorb the outlier magnitude.

Practical Recipes

For inference serving (latency-sensitive):

• Use AWQ INT4 with group size 128.
• Fast to quantize, good quality, works with vLLM/TRT-LLM.

For maximum quality:

• Use GPTQ INT4 with group size 128 and act-order.
• Slightly better quality than AWQ at the cost of slower quantization.

For both weights and activations (W8A8):

• Use SmoothQuant.
• Enables INT8 tensor core acceleration (actual compute speedup).

For edge deployment:

• Use GGUF format (llama.cpp) with Q4_K_M or Q5_K_M.
• Mixed precision: important layers in higher precision.

Quality Comparison

For LLaMA-2 70B on standard benchmarks:

• FP16 (baseline): 100% quality
• GPTQ INT4 (g=128): approximately 99% quality
• AWQ INT4 (g=128): approximately 98.5% quality
• Round-to-nearest INT4: approximately 95% quality
• INT8 per-channel: approximately 99.5% quality

The gap between GPTQ/AWQ and naive rounding demonstrates the value of intelligent quantization.

Common Pitfalls

Pitfall 1. Using random calibration data unrelated to the deployment domain. GPTQ/AWQ calibration should use representative data from the target use case.

Pitfall 2. Expecting INT4 to provide 4x compute speedup. W4A16 only speeds up weight loading (memory-bound workloads). For compute speedup, need W8A8 (SmoothQuant) or W4A4.

Pitfall 3. Quantizing the embedding and LM-head layers. These layers are disproportionately sensitive to quantization; keep them in higher precision (FP16 or INT8).

Summary

• GPTQ: Column-wise quantization with Hessian-based error compensation. Best quality.
• AWQ: Protect salient weights via activation-aware scaling. Fast and effective.
• SmoothQuant: Enables W8A8 by migrating outliers from activations to weights.
• All methods achieve near-lossless INT4 for large models (above 7B).
• Choice depends on: quality needs, quantization speed, and inference framework.

Exercises

Exercise 1. Derive the GPTQ compensation formula (equation 2) from minimizing the quadratic objective.

Exercise 2. For AWQ with $\alpha=2$ on a channel: compute the effective quantization error compared to $\alpha=1$.

Exercise 3. Compute the wall-clock speedup from W4A16 vs FP16 for batch-1 inference on an A100 (2 TB/s bandwidth, 312 TFLOPS).

Exercise 4. Design a calibration dataset for quantizing a code-generation model. What properties should it have?

Exercise 5. Explain why larger models (70B) are more robust to INT4 quantization than smaller models (7B).