ML

LLM Serving Systems: vLLM, TensorRT-LLM & SGLang

Production serving frameworks: request scheduling, memory management, batching strategies, prefix caching, constrained decoding, tensor parallelism for serving, and performance benchmarking (TTFT, TPS, throughput).

Advanced

Prerequisites

Table of Contents

  1. Learning Objectives
  2. Notation
  3. Core Intuition
  4. Performance Metrics
  5. vLLM Architecture
  6. TensorRT-LLM
  7. SGLang & RadixAttention
  8. Prefix Caching
  9. Constrained Decoding
  10. Serving at Scale
  11. Common Pitfalls
  12. Summary
  13. Exercises

Learning Objectives

  1. Define and measure TTFT, TPS, throughput, and SLO compliance.
  2. Compare vLLM, TensorRT-LLM, and SGLang architectures.
  3. Explain prefix caching and its impact on multi-turn conversations.
  4. Design constrained decoding for structured output.
  5. Configure tensor parallelism for serving (vs training).

Notation

  • TTFT — Time To First Token
  • TPS — Tokens Per Second (per request)
  • SLO — Service Level Objective

Core Intuition

An LLM in production serves thousands of concurrent users with strict latency requirements. The serving system must MAXIMIZE throughput (tokens/second across all users) while maintaining latency SLOs (TTFT less than 500ms, TPS greater than 30). This requires sophisticated scheduling (which request to process next), memory management (KV-cache allocation), and kernel optimization (efficient GPU utilization).

Serving Systems

Request QueueGPU Batchsize=4GPU util: 48%Throughput: 66 req/sLatency: 82msDynamic batching improves GPU utilization at higher request rates
Req/s
20
QueueGPU batch
Explore: Serving systems use continuous batching to group requests — higher request rates enable larger batches, improving GPU utilization and throughput at some latency cost.

Performance Metrics

Time To First Token (TTFT): Time from request arrival to first generated token.

  • Dominated by prefill computation.
  • Target: less than 500ms for interactive use.

Tokens Per Second (TPS): Generation speed per request.

  • Target: greater than 30 TPS for smooth streaming.
  • Bottleneck: memory bandwidth (loading weights per token).

Throughput: Total tokens generated across ALL requests per second.

  • Key for cost efficiency.
  • Throughput = batch_size × TPS_per_request.

SLO compliance: Fraction of requests meeting latency targets.

  • P50 (median): typical experience.
  • P99 (99th percentile): worst-case experience.
  • Target: P99 TTFT less than 2s; P99 TPS greater than 20.

vLLM Architecture

Key innovations:

  • PagedAttention: Non-contiguous KV-cache allocation (covered previously).
  • Continuous batching: Iteration-level scheduling.
  • Prefix sharing: Multiple requests with same prefix share KV pages.

Scheduler: Priority queue based on arrival time + SLO.

  • Preemption: If memory is full, preempt (swap to CPU) lowest-priority request.
  • Resume: When memory available, reload swapped KV-cache.

Performance: 2-24x throughput over naive HuggingFace serving.

Deployment: Single command: vllm serve model_name --tensor-parallel-size 4.

TensorRT-LLM

NVIDIA's optimized inference engine:

Kernel optimizations:

  • Custom CUDA kernels for attention (FlashAttention integrated).
  • Fused operations (LayerNorm + Linear, GeLU + Linear).
  • INT8/FP8 quantization with calibration.
  • In-flight batching (continuous batching in C++).

Compilation: Model compiled to optimized TensorRT engine (offline step).

  • Operator fusion.
  • Memory planning (pre-allocate all intermediate buffers).
  • Kernel selection (choose fastest kernel variant per layer).

Performance: Typically 20-50% faster than vLLM for same model due to kernel optimization. But less flexible.

SGLang & RadixAttention

Zheng et al. (2024): Optimized for PROGRAMS with multiple LLM calls:

RadixAttention: Store ALL KV-caches in a shared radix tree.

  • Common prefixes stored once (e.g., system prompt).
  • Automatic prefix matching for new requests.
  • Enables efficient multi-turn and branching conversations.

Frontend: Python DSL for LLM programs (fork, join, select):

Gen "What is 2+2?" → fork into multiple continuations
Select best continuation → continue generation

Optimizations:

  • Automatic prefix caching across requests.
  • Speculative execution of program branches.
  • Compressed finite state machines for constrained decoding.

Prefix Caching

Problem: Many requests share the same prefix (system prompt, few-shot examples).

Solution: Cache KV for common prefixes; reuse for new requests.

Impact on multi-turn chat:

  • Turn 1: Full prefill (system prompt + user message).
  • Turn 2: Only prefill the new user message (previous KV cached).
  • Turn 3: Only prefill newest message.

Savings: For a 2000-token system prompt: save 2000 × prefill_cost per request after the first.

Implementation: Hash prefix content → lookup in cache → hit = reuse KV, miss = compute.

Constrained Decoding

Force output to match a schema (JSON, regex, grammar):

Token masking: At each step, mask tokens that would violate the constraint:

p′(xt)=p(xt)⋅1[xt valid]∑x′p(x′)⋅1[x′ valid].(1)p'(x_t) = \frac{p(x_t) \cdot \mathbb{1}[x_t \text{ valid}]}{\sum_{x'} p(x') \cdot \mathbb{1}[x' \text{ valid}]}. \tag{1}

Finite state machine (FSM): Parse grammar into FSM; at each state, only allow valid transitions.

Outlines/Guidance: Libraries that implement efficient constrained decoding:

  • JSON schema → FSM → token masks.
  • Regex → DFA → token masks.
  • Context-free grammar → pushdown automaton.

Cost: Negligible compute overhead (FSM transitions are O(1)). But restricts the model's "freedom" — can sometimes reduce quality.

Serving at Scale

Multi-GPU serving:

  • Tensor parallelism (TP): Split model across GPUs for lower latency.
  • Pipeline parallelism (PP): Only for very large models (above 8 GPUs).
  • Data parallelism (replicas): Multiple model copies for throughput.

Typical configuration (70B model):

  • 8 GPUs per replica (TP=8).
  • Multiple replicas behind load balancer.
  • Auto-scaling based on queue depth.

Cost optimization:

  • Smaller models serve more requests per GPU.
  • Quantization (INT8/FP8): 2x more requests per GPU with minimal quality loss.
  • Longer batches: higher throughput but higher latency.

Common Pitfalls

Pitfall 1. Optimizing for throughput without latency SLOs. Maximum throughput means maximum batch size → high latency for individual requests. Always set P99 latency constraints.

Pitfall 2. Not using prefix caching for multi-turn chat. Without it, the system re-processes the entire conversation history on every turn. With a 4000-token history: wasting 4000 tokens of prefill compute per turn.

Pitfall 3. Using tensor parallelism across PCIe-connected GPUs. TP requires high-bandwidth interconnect (NVLink). Over PCIe: 70% of time spent in communication, not computation.

Summary

  • TTFT, TPS, throughput: The three metrics that define serving quality.
  • vLLM: PagedAttention + continuous batching; open-source standard.
  • TensorRT-LLM: NVIDIA-optimized kernels; 20-50% faster; less flexible.
  • SGLang: RadixAttention + LLM programs; best for multi-call workflows.
  • Prefix caching: 2-5x speedup for multi-turn conversations.
  • Constrained decoding: Force structured output with minimal overhead.

Exercises

Exercise 1. For LLaMA-70B on 8 H100s (TP=8): compute maximum throughput (tokens/s) given 3.35 TB/s memory bandwidth per GPU and 140GB model size.

Exercise 2. Compare TTFT for a 2000-token prompt with and without prefix caching (assuming 1500 tokens cached from system prompt).

Exercise 3. Design an auto-scaling policy for a chat service: specify metrics, scale-up/scale-down thresholds, and cooldown periods.

Exercise 4. Implement a constrained decoding FSM for JSON output with schema: {"name": string, "age": integer}. Specify valid tokens at each state.

Exercise 5. Benchmark vLLM vs TensorRT-LLM for: (a) batch-1 latency, (b) batch-64 throughput, (c) multi-turn conversation with prefix caching.