AI Engineering & LLMs (50 Qs)•Part 2: LLM Inference, Serving & Runtime Optimization
PagedAttention & vLLM Architecture: Eliminating KV Cache Memory Fragmentation
1. The Classical KV Cache Allocation Problem
In traditional serving frameworks (Hugging Face naive transformers, early TGI), memory for a request's KV cache had to be allocated contiguously based on the maximum potential sequence length (e.g., reserving 4,096 tokens even if the model outputs 20 tokens):
java
Traditional Contiguous Allocation:
┌──────────────────────────────────────────────┬───────────────────────────────┐
│ Actual Tokens Used (200 tokens) │ Reserved / Wasted (3,896 toks)│
└──────────────────────────────────────────────┴───────────────────────────────┘
▲ 60% - 80% VRAM Wasted!
This creates catastrophic internal fragmentation (reserved memory never used) and external fragmentation (memory chunks too fragmented to host a new request).
2. PagedAttention Virtual Memory Solution
Inspired by virtual memory paging in operating systems (OS), PagedAttention (developed by Kwon et al. at UC Berkeley) breaks the KV cache into fixed-size blocks (e.g., 16 or 32 tokens per block):
less
Logical KV Cache (Sequential):
[ Block 0 (Tokens 0-15) ] ──► [ Block 1 (Tokens 16-31) ] ──► [ Block 2 (Tokens 32-47) ]
│ │ │
▼ ▼ ▼
Physical Blocks in GPU Memory (Non-contiguous):
┌───────────────┬───────────────┬───────────────┬───────────────┐
│ Physical #42 │ Free Block │ Physical #11 │ Physical #89 │
│ (Block 0) │ │ (Block 2) │ (Block 1) │
└───────────────┴───────────────┴───────────────┴───────────────┘
3. Key Architectural Benefits of PagedAttention
| Metric | Traditional Contiguous Serving | vLLM with PagedAttention |
|---|---|---|
| VRAM Waste | 60% – 80% wasted memory | < 4% wasted (only in the final partial block) |
| Serving Throughput | 1x baseline | 2x to 4x higher throughput on identical GPUs |
| Parallel Sampling (Beam Search) | Duplicate full KV caches | Zero duplication: share parent blocks via copy-on-write |
| Prefix Caching | Requires expensive memory copies | Instant block-pointer reuse for identical system prompts |