quantizationoptimizationdeployment

When and How to Quantize Your Models

A practical mathematical guide to model quantization — understanding the trade-offs between precision, performance, and accuracy.

ML for Everyone TeamMay 10, 202610 min read

The Precision-Performance Trade-off

Modern LLMs contain billions of parameters stored in 16-bit or 32-bit floating point. Quantization reduces this to 8-bit, 4-bit, or even lower — dramatically reducing memory and improving inference speed.

The Mathematics of Quantization

At its core, quantization maps continuous values to discrete ones:

x_q = \text{round}\left(\frac{x}{s}\right) + z

where s is the scale factor and z is the zero-point. The key challenge is choosing s and z to minimize the quantization error.

Types of Quantization

Post-Training Quantization (PTQ)

Apply quantization after training

No retraining needed

Works well for 8-bit, degradation at 4-bit

Quantization-Aware Training (QAT)

Simulate quantization during training

Model learns to be robust to precision loss

Better accuracy at very low bit-widths

GPTQ and AWQ

Advanced methods that solve for optimal quantization parameters

Use calibration data to minimize output error

Enable 4-bit and 3-bit quantization with minimal accuracy loss

When to Quantize

Scenario	Recommendation
Serving at scale	8-bit (INT8) — nearly lossless
Edge deployment	4-bit (INT4) with GPTQ/AWQ
Research/fine-tuning	Keep FP16/BF16
Memory-constrained	4-bit with careful validation

The Mathematical Insight

The key insight is that neural network weights are highly structured — they're not random numbers. This structure means that intelligent quantization (calibration-based, per-channel, mixed-precision) can preserve nearly all the model's capability at a fraction of the memory cost.

Understanding the mathematical properties of your weight distributions — kurtosis, outlier structure, layer sensitivity — is what separates naive quantization from state-of-the-art methods.