AI Engineering & LLMs (50 Qs)•Part 2: LLM Inference, Serving & Runtime Optimization
Continuous Batching (Iteration-Level Scheduling) vs. Static Batching
1. Static Batching Bottleneck in Autoregressive Serving
In traditional deep learning (computer vision, classification), inputs have fixed dimensions and static batching works well. In LLM generation, however, requests have unpredictable generation lengths:
less
Static Batching: GPU Underutilization
Request A: [ Gen 10 tokens ] ──► Idle (Waiting for B & C) ──────────────────────────┐
Request B: [ Gen 40 tokens ] ──────────────► Idle (Waiting for C) ───────────────────┼─► Batch Finishes
Request C: [ Gen 120 tokens ] ───────────────────────────────────────────────────────┘
▲ Massive wasted GPU cycles during decoding
2. Continuous Batching (Iteration-Level Scheduling)
Pioneered by Orca (Yu et al.) and standard in vLLM / TensorRT-LLM, Continuous Batching operates at the iteration level (every individual decoding step):
- As soon as Request A outputs an
<EOS>token, it is immediately evicted from the active batch. - A waiting request from the queue (Request D) enters the batch on the very next step without pausing or resetting existing requests:
less
Continuous Iteration-Level Batching:
Step: 1 2 ... 10 11 12 ... 40 41 42 ...
Req A: [#1] [#2] ... [EOS] ◄── Evicted!
Req B: [#1] [#2] ... [#10][#11][#12] ... [EOS]
Req C: [#1] [#2] ... [#10][#11][#12] ... [#40][#41][#42] ...
Req D: ▲ Newly scheduled on Step 11!
3. Production Impact on Key Latency Metrics
- Throughput: Improves by 3x to 5x over static batching.
- Time to First Token (TTFT): New requests do not wait for earlier long-generation requests to complete.
- GPU Tensor Core Saturation: Keeps matrix multiply units continuously loaded.