AI Engineering & LLMs (50 Qs)•Part 2: LLM Inference, Serving & Runtime Optimization
Prompt Caching & Prefix Caching: Anthropic, Gemini & vLLM Internal Mechanics
1. The Redundant Prompt Problem
In typical enterprise AI workloads, 80% to 95% of prompt tokens are identical across sequential calls:
- Static System Prompts (1,000–5,000 tokens of rules and formatting constraints).
- Few-Shot Examples.
- Large Ingested Documents (e.g., 50-page financial report queried repeatedly by users).
- Multi-turn conversation chat history.
Without caching, the model must re-run the compute-heavy Prefill phase over the exact same prefix on every single turn!
2. Prefix Caching Mechanics
When a request enters the engine, the system hashes prefix token sequences into a Radix Tree or Trie:
sql
[ Root Node ]
│
hash("You are an enterprise support...")
│
▼
[ Block ID: 101-105 ] (Cached in VRAM)
│
┌─────────────────┴─────────────────┐
▼ ▼
User Turn 1 Chunks User Turn 2 Chunks
(Compute prefill only (Compute prefill only
for the new user delta) for the new user delta)
3. Economic & Latency Comparison
| Vector | Uncached Request | Cached Prefix Request (Anthropic / vLLM) | Improvement |
|---|---|---|---|
| Prefill Latency (TTFT) | 1,200 ms (re-computing 10k tokens) | 80 ms (fetching KV cache from VRAM) | 15x Faster TTFT |
| Compute Cost (Tokens) | 100% standard input token cost | 10% to 25% of standard token rate | 75–90% Cost Reduction |
| GPU Energy & Heat | Continuous GEMM matrix operations | Fast memory read | Substantially higher concurrency |
python
# Example: Anthropic Prompt Caching Header Configuration
import anthropic
client = anthropic.Anthropic()
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=[
{
"type": "text",
"text": "Detailed 20-page system documentation and corporate knowledge base...",
"cache_control": {"type": "ephemeral"} # Instructs engine to cache prefix
}
],
messages=[{"role": "user", "content": "What is the vacation policy?"}]
)