LLM SLA Metrics & Capacity Planning: TTFT, ITL, TPS & Concurrency Modeling

1. The Core LLM SLA Latency Metrics

Traditional web services measure P50/P95/P99 round-trip request latency. LLM systems decompose latency into four distinct operational metrics:

Client Sends Request ──► [ Network ] ──► Prefill Phase ──► [ First Token Emitted ] ──► Decode Tokens ... ──► [ <EOS> Received ]
│◄───────────────────────────── TTFT ─────────────────────────────►│
                                                                   │◄────── ITL ──────►│
│◄──────────────────────────────────────── Total Request E2E Latency ────────────────────────────────────────►│

1. Time-To-First-Token (TTFT): Elapsed time from request dispatch until the client receives token #1. Governed by prefill compute and queue waiting time.
2. Inter-Token Latency (ITL) / Time-Per-Output-Token (TPOT): Time between consecutive tokens during generation (target: < 30ms for human reading speed). Governed by memory bandwidth.
3. Tokens Per Second (TPS): Total throughput per GPU instance (target throughput for serving infrastructure).
4. Queue Time: Time request sits waiting before entering an active continuous batch.

2. GPU Capacity Planning & Sizing Formula

To size an enterprise deployment serving $U$ concurrent users generating $T_{\text{out}}$ tokens with average input $T_{\text{in}}$:

$$\text{Required Throughput (Tokens/sec)} = \frac{U \times (T_{\text{in}} + T_{\text{out}})}{\text{Target Latency Window (s)}}$$

Example Scenario:

• 200 concurrent active users.
• Average prompt = 1,000 tokens, output = 200 tokens. Total = 1,200 tokens.
• Max acceptable round-trip response time = 5 seconds.
• $\text{Target TPS} = \frac{200 \times 1,200}{5} = \mathbf{48,000\text{ tokens/sec}}$.
• A single 8x H100 node running vLLM delivers ~3,000–5,000 TPS on 70B models.
• Infrastructure required: 10–12 nodes of 8x H100 GPUs behind an AI load balancer.