Quantized LLM Serving: vLLM, TRT-LLM & llama.cpp

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. vLLM with Quantized Models
5. TensorRT-LLM: Maximum GPU Throughput
6. llama.cpp: CPU & Consumer Hardware
7. Serving Tradeoffs: Latency vs Throughput
8. Production Deployment Recipes
9. Common Pitfalls
10. Summary
11. Exercises

Learning Objectives

1. Compare vLLM, TRT-LLM, and llama.cpp for quantized model serving.
2. Explain how quantization interacts with batching and KV-cache.
3. Choose the optimal framework for a given deployment scenario.
4. Estimate tokens/second for different quantization + hardware combinations.
5. Design a production serving stack for quantized LLMs.

Notation

• TTFT — time to first token
• TPOT — time per output token
• TPS — tokens per second (throughput)

Core Intuition

Quantization reduces model size, but the serving framework determines whether that translates to actual speed. Three dominant ecosystems exist: vLLM (PagedAttention + quantization plugins), TensorRT-LLM (NVIDIA-optimized compiled kernels), and llama.cpp (CPU-first with GGUF). Each excels in different scenarios: vLLM for high-batch GPU serving, TRT-LLM for maximum single-GPU throughput, llama.cpp for consumer/edge deployment.

vLLM with Quantized Models

Supported formats:

• AWQ (INT4, group=128): First-class support, Marlin kernel.
• GPTQ (INT4/INT8): Supported via ExLlama/Marlin kernels.
• FP8 (E4M3): Native on H100 with per-tensor scaling.
• SqueezeLLM, AQLM: Community support.

PagedAttention + quantization:

• KV-cache: Stored in FP16 (quantizing KV-cache separately).
• FP8 KV-cache: 2x more tokens in memory (vLLM supports this on H100).

Typical throughput (A100 80GB, LLaMA-2 70B):

• FP16: Doesn't fit (needs 140GB).
• AWQ INT4: Fits. ~1500 tokens/sec at batch=32.
• GPTQ INT4: Fits. ~1400 tokens/sec at batch=32.

TensorRT-LLM: Maximum GPU Throughput

NVIDIA's optimized inference engine:

1. Ahead-of-time compilation to optimized CUDA kernels.
2. Layer fusion (combine quantize + matmul + dequantize + bias + activation).
3. Native INT4/INT8/FP8 support with optimized kernels.
4. In-flight batching + paged KV-cache.

Key optimizations:

• Weight-only quantization (W4A16): Custom kernels dequantize INT4 → FP16 in shared memory, then FP16 tensor core matmul.
• W8A8 SmoothQuant: Pre-smoothed weights + INT8 tensor cores.
• FP8: Native E4M3 with per-tensor/per-channel scaling.

Speedup over vLLM: 1.5-2x for the same quantization method (TRT-LLM's compiled kernels are faster than vLLM's dynamic dispatch).

llama.cpp: CPU & Consumer Hardware

Philosophy: Run LLMs on any hardware, no GPU required.

Key features:

• GGUF format with k-quant types (Q4_K_M, Q5_K_S, etc.).
• CPU: Hand-tuned SIMD kernels (AVX2, AVX-512, ARM NEON).
• GPU offloading: Partial or full model on GPU (CUDA, Metal, Vulkan).
• Memory-mapped model loading (instant start).

Performance (M2 MacBook Pro, LLaMA-2 13B Q4_K_M):

• Prompt processing: ~80 tokens/sec (batched).
• Generation: ~15 tokens/sec (sequential).
• Memory: 7.5 GB.

Performance (RTX 4090, LLaMA-2 70B Q4_K_M, full GPU offload):

• Generation: ~25 tokens/sec.
• Memory: ~38 GB (24GB VRAM + system RAM for remaining layers).

Serving Tradeoffs: Latency vs Throughput

Latency-sensitive (chatbot, one user):

• Minimize TTFT and TPOT.
• Small batch size (1-4).
• W4A16: Memory-bandwidth bound → quantization helps most.
• Prefer llama.cpp or TRT-LLM (low-overhead).

Throughput-sensitive (batch processing, many users):

• Maximize total tokens/sec across all requests.
• Large batch size (32-256).
• W8A8: Compute-bound → tensor core utilization matters.
• Prefer vLLM or TRT-LLM (continuous batching).

The crossover: At batch size ~8-16, the workload transitions from memory-bound to compute-bound. Below this: W4A16 wins. Above: W8A8 wins.

Production Deployment Recipes

Recipe 1: Maximum quality, GPU serving (A100/H100)

• Model: AWQ INT4 or FP8.
• Framework: vLLM or TRT-LLM.
• Batch: Continuous batching with 32-64 max concurrent.
• KV-cache: FP16 (or FP8 on H100 for 2x more context).

Recipe 2: Cost-optimized, consumer GPU (RTX 4090)

• Model: GPTQ INT4 or AWQ INT4 (fits in 24GB VRAM for 13B).
• Framework: vLLM or text-generation-webui.
• Batch: 1-4 concurrent users.

Recipe 3: No GPU (CPU deployment, edge)

• Model: GGUF Q4_K_M (best quality/size tradeoff).
• Framework: llama.cpp or llamafile.
• Hardware: MacBook M2+, or server with AVX-512.

Recipe 4: Mobile/embedded

• Model: INT4 with CoreML (Apple) or ONNX Runtime (Android).
• Framework: MLX (Apple) or MediaPipe.
• Models: 1-3B only at acceptable speed.

Common Pitfalls

Pitfall 1. Using GPTQ/AWQ with llama.cpp. llama.cpp uses GGUF format; GPTQ/AWQ are for GPU frameworks (vLLM, TRT-LLM). Convert to GGUF for CPU inference.

Pitfall 2. Not enabling continuous batching in production. Without it, serving one user at a time wastes GPU utilization. Always use vLLM/TRT-LLM batching for multi-user scenarios.

Pitfall 3. Overprovisioning GPU for small batch. A single-user chatbot with a 7B model at INT4 runs perfectly on an RTX 3060 (12GB). Don't rent an A100 for this.

Summary

• vLLM: Best for multi-user GPU serving with continuous batching. AWQ/GPTQ/FP8.
• TRT-LLM: Maximum per-GPU throughput via compiled kernels. 1.5-2x over vLLM.
• llama.cpp: CPU inference for consumer hardware. GGUF format. No GPU needed.
• Latency: W4A16 at small batch (memory-bound regime).
• Throughput: W8A8 at large batch (compute-bound regime).
• Choose framework based on: hardware, user count, latency requirements, deployment complexity.

Exercises

Exercise 1. Compute the memory required to serve LLaMA-2 70B with AWQ INT4 on vLLM, including KV-cache for 4096 context length and batch size 32.

Exercise 2. Using the roofline model: at what batch size does an A100 transition from memory-bound to compute-bound for a 7B INT4 model?

Exercise 3. Compare tokens/sec/dollar for: (a) 8x A100 with TRT-LLM, (b) M2 Ultra Mac Studio with llama.cpp, (c) 4x RTX 4090 with vLLM. For a 70B model at INT4.

Exercise 4. Design a multi-tier serving system: fast tier (small model, low latency) and quality tier (large model, higher latency). Specify models, quantization, and routing logic.

Exercise 5. A startup needs to serve a 13B model to 100 concurrent users with TPOT less than 50ms. Specify the minimum hardware and quantization strategy.