Speculative Decoding: Small Draft Models, Verification & Latency Reduction

1. The Fundamental Decode Latency Wall

In modern GPUs, memory bandwidth—not compute capability—is the primary bottleneck for autoregressive token decoding. Reading all 70 billion parameters from GPU HBM memory to generate a single token utilizes only ~5% of GPU tensor core compute.

Speculative Decoding (Leviathan et al.) breaks this sequential barrier:

  1. Small Draft Model (e.g. Llama 3 8B)
     Generates K draft tokens sequentially (Fast & Cheap):
     [ "The", "capital", "of", "France", "is" ]
                         │
                         ▼
  2. Large Target Model (e.g. Llama 3 70B)
     Evaluates all K tokens concurrently in a SINGLE parallel prefill step!
     [ Validated? -> Yes, Yes, Yes, Yes, Correct ]
                         │
                         ▼
  Result: 4 to 5 tokens generated in the time of a single large-model forward pass!

2. Mathematical Guarantee: Zero Degradation in Output Quality

Speculative decoding uses Speculative Sampling (rejection sampling). If the target model rejects token $k$:

• All tokens after $k$ are discarded.
• A modified sampling distribution guarantees that the final output distribution is mathematically identical to sampling directly from the large target model!

3. Performance Trade-Off Matrix