LLM Serving Systems & Infrastructure

Table of Contents

1. Learning Objectives
2. Notation
3. Core Intuition
4. The Serving Stack
5. vLLM Architecture
6. TensorRT-LLM
7. Model Sharding for Serving
8. Request Routing & Load Balancing
9. Autoscaling Strategies
10. Cost Optimization
11. Common Pitfalls
12. Summary
13. Exercises

Learning Objectives

1. Describe the key components of an LLM serving stack.
2. Explain vLLM's PagedAttention and continuous batching.
3. Compare vLLM, TensorRT-LLM, and TGI in terms of throughput and latency.
4. Design a serving configuration for different latency/throughput SLAs.
5. Analyze the cost structure of LLM serving and optimization strategies.

Notation

• $\text{TTFT}$ — time to first token
• $\text{TPOT}$ — time per output token
• $\text{TPS}$ — tokens per second (throughput)
• $\text{QPS}$ — queries per second

Core Intuition

Serving LLMs in production is fundamentally different from training: latency matters, requests arrive unpredictably, and cost per token determines business viability. A serving system must maximize throughput (tokens/second) while maintaining latency SLAs (time to first token < 1s, time per output token < 100ms), all while efficiently utilizing expensive GPU hardware.

The Serving Stack

Layer 1: Inference engine (vLLM, TRT-LLM): Manages GPU computation, KV-cache, batching.

Layer 2: Model server (Triton, Ray Serve): HTTP/gRPC API, request queuing, health checks.

Layer 3: Router/Gateway (nginx, Envoy): Load balancing across replicas, rate limiting, authentication.

Layer 4: Autoscaler (K8s HPA, custom): Scale replicas up/down based on demand.

vLLM Architecture

Core innovations:

• PagedAttention: Non-contiguous KV-cache management (as discussed in KV-cache optimization).
• Continuous batching: Insert/remove requests every iteration.
• Prefix caching: Reuse KV-cache for shared prefixes.

Architecture:

1. Scheduler: Decides which requests to admit/preempt each step.
2. Block manager: Allocates/frees KV-cache pages.
3. Worker: Runs the model (supports TP across GPUs).
4. Tokenizer: Encodes/decodes text.

Performance: 2-24x higher throughput than HuggingFace Transformers serving.

TensorRT-LLM

NVIDIA's optimized inference engine:

• Compiler-level optimization: Graph fusion, kernel auto-tuning, layout optimization.
• Custom kernels: Fused MHA, fused GeLU+bias, fused LayerNorm+residual.
• Quantization: INT8/INT4/FP8 with calibrated accuracy.
• In-flight batching: Their term for continuous batching.

Advantage over vLLM: Lower per-token latency (more optimized kernels). Better for latency-sensitive workloads.

Disadvantage: Less flexible, longer setup time, NVIDIA-only.

Model Sharding for Serving

Single-GPU serving (up to ~14B params in INT4): Simplest, lowest latency.

Tensor parallelism (within node): Split model across 2-8 GPUs. Reduces per-token latency by $N$ (parallel compute). Used for large models or when TTFT SLA is tight.

Pipeline parallelism (across nodes): Rarely used for serving (adds latency). Only when model doesn't fit in one node.

Expert parallelism (for MoE): Each GPU holds different experts. Route tokens to appropriate GPUs.

Recommendation:

• 7B model: 1 GPU (A100/H100)
• 70B model: 2-4 GPUs with TP (or 1 GPU in INT4)
• 405B model: 8 GPUs with TP

Request Routing & Load Balancing

Prefix-aware routing: Route requests with similar prefixes to the same replica (maximize prefix cache hits).

Load-based routing: Route to the replica with the shortest queue / lowest active requests.

SLA-tiered routing: Premium requests → dedicated low-utilization replicas. Standard requests → shared high-utilization replicas.

Geographic routing: Route to nearest datacenter for lowest network latency.

Autoscaling Strategies

Metric-based scaling:

• Scale on: GPU utilization > 80%, queue depth > threshold, P95 latency > SLA.
• Scale down on: GPU utilization < 30% for sustained period.

Predictive scaling: Use historical patterns (daily traffic cycles) to pre-scale before demand spikes.

Cost-aware scaling: Prefer spot/preemptible instances for burst traffic; reserved for baseline.

Cold start problem: New replicas take 30-60s to load model weights. Mitigation: keep warm standby replicas, use model caching on local SSDs.

Cost Optimization

Cost breakdown:

• GPU compute: 60-80% of total cost.
• Memory (KV-cache determines max concurrent requests): limits throughput.
• Networking: 5-10% (for TP communication).

Optimization strategies:

1. Quantization: INT4 → 4x more requests per GPU.
2. Speculative decoding: 2-3x faster generation → fewer GPU-seconds per request.
3. Prefix caching: Amortize system prompt processing across requests.
4. Batching: Higher batch sizes → better GPU utilization → lower cost per token.
5. Right-sizing: Match GPU to model size (don't use A100 for 7B models).

Cost per token (approximate, 2024):

• GPT-4: $30/M input,$60/M output tokens.
• Self-hosted LLaMA-70B on A100: ~$1-3/M tokens.
• Self-hosted 7B quantized: ~$0.1-0.3/M tokens.

Common Pitfalls

Pitfall 1. Optimizing only for throughput while ignoring latency. Users perceive time-to-first-token and streaming speed. A high-throughput system with 5s TTFT will feel unresponsive.

Pitfall 2. Not monitoring KV-cache utilization. If cache fills up, new requests are rejected or existing ones are preempted — causing spikes in latency.

Pitfall 3. Using the same configuration for all workloads. Short prompts + long outputs (chatbots) vs long prompts + short outputs (summarization) need different batch sizes, prefill priorities, and memory allocation.

Summary

• vLLM: PagedAttention + continuous batching → highest throughput open-source system.
• TensorRT-LLM: Compiler-optimized kernels → lowest per-token latency.
• Serving configuration: Choose TP degree based on model size and latency SLA.
• Autoscaling: Scale on queue depth/latency with predictive pre-scaling.
• Cost optimization: Quantization + batching + prefix caching + right-sizing.

Exercises

Exercise 1. For a 70B model serving 100 QPS with average 500 output tokens: compute the required GPU count assuming 50 tokens/s/GPU throughput.

Exercise 2. Compare the cost/token of serving LLaMA-70B on 4x A100 (TP=4) vs 8x A100 (TP=8). Which is more cost-efficient?

Exercise 3. Design an autoscaling policy for a chatbot service with predictable daily traffic patterns (peak at 2pm, trough at 4am).

Exercise 4. Compute the prefix cache hit rate needed to reduce TTFT by 50% for a service where 80% of requests share a 500-token system prompt.

Exercise 5. For a 7B model on a single A100: compute maximum concurrent requests given 80GB memory, 14GB model weights (INT4), and 160KB per-token KV-cache.