Number Formats: FP32, FP16, BF16, FP8, INT8, INT4

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. IEEE 754 Floating-Point
5. FP32: Full Precision
6. FP16 vs BF16: The Training Tradeoff
7. FP8: E4M3 vs E5M2
8. Integer Formats: INT8 and INT4
9. The Dynamic Range vs Precision Tradeoff
10. Choosing the Right Format
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Decode the bit layout of IEEE 754 floating-point numbers.
2. Explain why BF16 succeeded FP16 for training (dynamic range matters more).
3. Compare FP8 E4M3 vs E5M2 for forward vs backward passes.
4. Calculate the precision and range of each format.
5. Choose the appropriate format for training, fine-tuning, and inference.

Notation

• $e$ — exponent bits, $m$ — mantissa (significand) bits
• $\text{bias}$ — exponent bias ($2^{e-1}-1$)
• ULP — unit in last place (smallest representable difference)

Core Intuition

A number format is a tradeoff between three things: range (largest representable number), precision (smallest distinguishable difference), and storage (bits used). Training needs range (gradients can be huge or tiny); inference needs precision (weights are small, tightly distributed). Understanding this tradeoff is essential for choosing quantization strategies.

IEEE 754 Floating-Point

General form: $(-1)^s \times 2^{E - \text{bias}} \times (1 + M/2^m)$

Key formulas:

• Dynamic range: $[2^{-(2^{e-1}-2)}, \; 2^{2^{e-1}-1}]$
• Precision: $\sim 2^{-m}$ relative error (ULP at scale 1)
• Total values: $2^{1+e+m}$

FP32: Full Precision

Layout: 1 sign + 8 exponent + 23 mantissa = 32 bits.

Role: The "ground truth" reference. Used for master weights in mixed-precision training, loss computation, and optimizer states.

FP16 vs BF16: The Training Tradeoff

FP16 (IEEE half): 1 + 5 + 10 = 16 bits.

BF16 (Brain Float): 1 + 8 + 7 = 16 bits.

Why BF16 won for training:

Gradients and activations can exceed 65504 (causing FP16 overflow). BF16 matches FP32's range, avoiding overflow without loss scaling tricks.

FP16 advantage: More precision (10 vs 7 mantissa bits). Better for inference where values are bounded.

FP8: E4M3 vs E5M2

Two FP8 variants (NVIDIA Hopper/Ada):

E4M3 (1 + 4 + 3): More precision, less range.

• Range: $\pm 448$
• Precision: 3 mantissa bits (8 values per exponent)
• Best for: Forward pass (weights and activations)

E5M2 (1 + 5 + 2): More range, less precision.

• Range: $\pm 57344$
• Precision: 2 mantissa bits (4 values per exponent)
• Best for: Backward pass (gradients can spike)

FP8 recipe (NVIDIA):

• Forward: weights in E4M3, activations in E4M3.
• Backward: gradients in E5M2.
• Master weights: FP32 (as always).

Speedup: 2x over FP16 on H100 tensor cores.

Integer Formats: INT8 and INT4

INT8 (signed): Range $[-128, 127]$. Uniform spacing: every integer from -128 to 127.

INT8 (unsigned): Range $[0, 255]$. Used for activations after ReLU.

INT4 (signed): Range $[-8, 7]$. Only 16 distinct values.

Key difference from floating-point: Integers have UNIFORM spacing. Floats have LOG-spaced values (denser near zero). Neural network weights are roughly Gaussian → many values near zero → integers waste precision on the sparse tails.

Mitigation: Non-uniform quantization (NF4) places levels at Gaussian quantiles.

The Dynamic Range vs Precision Tradeoff

For $b$ total bits, every additional exponent bit doubles the range but halves the precision:

Implications:

• Training (needs range): prefer more exponent bits (BF16, E5M2).
• Inference (needs precision): prefer more mantissa bits (FP16, E4M3).
• Weights (static, known range): INT8/INT4 with good scaling.
• Activations (dynamic range): floating-point or careful calibration.

Choosing the Right Format

Common Pitfalls

Pitfall 1. Using FP16 for training without loss scaling. Values above 65504 overflow to infinity. Always use dynamic loss scaling or switch to BF16.

Pitfall 2. Assuming INT4 and FP4 are equivalent. INT4 has 16 uniformly-spaced values; FP4 (E2M1) has non-uniformly spaced values with more resolution near zero.

Pitfall 3. Comparing formats only by bit-width. BF16 and FP16 are both 16-bit but have very different characteristics. The bit allocation (exponent vs mantissa) matters as much as total bits.

Summary

• FP32: Ground truth; 4 bytes; used for master weights and optimizer states.
• BF16: Same range as FP32 with 2 bytes; dominant for training.
• FP16: Higher precision than BF16 but limited range; good for inference.
• FP8 E4M3/E5M2: 1 byte; 2x speedup on H100; precision vs range variants.
• INT8: 1 byte; uniform spacing; standard for deployment quantization.
• INT4: 0.5 bytes; 16 values; needs careful calibration or non-uniform formats.
• Choose format based on: training vs inference, range needs, precision needs.

Exercises

Exercise 1. Decode the FP16 bit pattern 0 10101 0110000000 to its decimal value.

Exercise 2. Compute the largest value representable in E4M3 (FP8) and compare to FP16.

Exercise 3. For a Gaussian-distributed weight with $\sigma=0.02$: compute the quantization error (MSE) for INT8 symmetric vs NF4 with optimal range.

Exercise 4. Explain why BF16 training doesn't need loss scaling but FP16 does (give a concrete gradient magnitude example).

Exercise 5. Design a custom 6-bit format for neural network weights. How many exponent and mantissa bits would you allocate? Justify.