GGUF & Mixed-Precision Quantization

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The GGUF Format
5. K-Quant Types
6. Mixed-Precision Strategy
7. Importance-Based Bit Allocation
8. Quality-Size Tradeoff
9. Choosing the Right Quantization
10. Common Pitfalls
11. Summary
12. Exercises

Learning Objectives

1. Describe the GGUF format structure and metadata.
2. Explain k-quant quantization types and their bit allocation.
3. Analyze mixed-precision strategies (important layers in higher precision).
4. Choose quantization level based on available RAM and quality needs.
5. Compare GGUF vs GPTQ vs AWQ for different deployment scenarios.

Notation

• Q$n$K$x$ — $n$-bit k-quant, size variant $x$ (S=small, M=medium, L=large)
• BPW — bits per weight (average across all layers)
• PPL — perplexity (lower = better quality)

Core Intuition

Real-world LLM deployment on consumer hardware (laptops, phones) requires aggressive quantization. GGUF (used by llama.cpp) takes a pragmatic approach: different layers get different bit-widths based on their sensitivity. Attention layers (especially Q, K projections) are more sensitive and get more bits; FFN layers tolerate more compression. This mixed-precision approach achieves better quality than uniform quantization at the same average bit-width.

The GGUF Format

Structure:

• Header: magic number, version, metadata (model architecture, vocab, context length).
• Tensor data: quantized weights in various formats.
• Self-contained: single file with everything needed for inference.

Key features:

• Multiple quantization types within one file (mixed precision).
• Memory-mapped loading (mmap): instant start, no deserialization.
• CPU-optimized: SIMD kernels for quantized matmul (AVX2, ARM NEON).
• Split files: for models exceeding single-file limits.

K-Quant Types

Super-blocks (256 elements) divided into sub-blocks (typically 32 elements):

Super-block structure (Q4_K example):

• 256 weights per super-block.
• 1 FP16 min + 1 FP16 scale for the super-block.
• 8 sub-blocks of 32 weights each, with 6-bit sub-scales.
• Weights stored as 4-bit integers.

Mixed-Precision Strategy

Q4_K_M ("medium" quality): Not all layers at 4 bits:

• Attention (Q, K, V, O): Q6_K (6 bits) — sensitive layers.
• FFN (gate, up, down): Q4_K (4 bits) — tolerant layers.
• Embeddings: Q6_K — first/last layers are critical.
• Output head: Q6_K.

Average: 4.85 BPW (more than pure Q4 due to higher-precision sensitive layers).

Q4_K_S ("small"): Same idea but less overhead:

• Fewer layers at higher precision.
• Average: 4.58 BPW.

Importance-Based Bit Allocation

Layer importance varies:

• Attention projections: High importance. Q, K determine what the model attends to. Errors here corrupt the attention pattern.
• FFN layers: Lower importance. Redundant computation; errors average out.
• First/last layers: High importance. Embedding and output head directly map tokens; errors here affect all predictions.
• Middle layers: Often most compressible.

Optimal allocation: Given a total budget of $B$ average bits, distribute more bits to sensitive layers:

where $\Delta_i > 0$ for important layers and $\Delta_i < 0$ for compressible ones.

Quality-Size Tradeoff

For LLaMA 2 7B (baseline FP16: 13.5 GB):

Rule of thumb: Q4_K_M is the sweet spot for most deployments (70% size reduction, minimal quality loss).

Choosing the Right Quantization

Decision tree:

1. RAM available: Choose the highest quality that fits.
2. Quality-critical (medical, legal): Q6_K or Q8_0.
3. General chatbot: Q4_K_M or Q5_K_M.
4. RAM-constrained (phone, 8GB laptop): Q3_K_M or Q4_K_S.
5. Extreme constraint: Q2_K (accept quality degradation).

GGUF vs GPTQ vs AWQ:

• GGUF: CPU inference (llama.cpp). Best for consumer hardware without GPU.
• GPTQ: GPU inference (vLLM, TGI). Best quality at INT4.
• AWQ: GPU inference. Fast quantization, good quality, vLLM compatible.

Common Pitfalls

Pitfall 1. Quantizing a 3B model to Q2_K and expecting useful outputs. Small models have less redundancy; aggressive quantization destroys them. Minimum recommended: Q4 for models under 7B.

Pitfall 2. Using Q8_0 when memory is the constraint. Q8 saves only 47% vs FP16; Q4_K_M saves 70%. If you're quantizing for memory, go to 4-5 bits.

Pitfall 3. Comparing perplexity across different quantization methods without controlling for calibration data. GPTQ, AWQ, and GGUF use different calibration; compare on the same eval suite.

Summary

• GGUF: Self-contained format for CPU inference with mixed-precision quantization.
• K-quants: Super-block structure with sub-block scales for fine granularity.
• Mixed precision: Important layers (attention, embeddings) get more bits.
• Sweet spot: Q4_K_M for most deployments (4.85 BPW, minimal quality loss).
• Size reduction: 70% at Q4, 80% at Q3, with varying quality impact.
• Choose format based on: hardware (CPU vs GPU), RAM budget, quality requirements.

Exercises

Exercise 1. Compute the exact file size for a 13B model quantized to Q4_K_M (estimate BPW = 4.85).

Exercise 2. Design a mixed-precision scheme for a 70B model that fits in 32GB RAM. Specify bit-width per layer type.

Exercise 3. For Q4_K: compute the overhead from 6-bit sub-scales in a super-block of 256 weights (as percentage of total storage).

Exercise 4. Explain why Q3_K_L (large) has better quality than Q3_K_S (small) despite both being "3-bit" — what's different?

Exercise 5. A user has 16GB RAM and wants to run LLaMA 2 70B. What quantization level is needed? Is the quality acceptable?