GPU Memory Sizing & Sizing Formulas: Weights, KV Cache, Activations & AdamW States

1. Complete VRAM Breakdown for Training & Inference

GPU out-of-memory errors occur because developers calculate memory for model weights alone, ignoring dynamic runtime allocations:

  Inference Memory Breakdown:
  ┌─────────────────────────────────┬─────────────────────────────────┬───────────┐
  │ Model Weights (Parameters)      │ KV Cache (Context & Concurrency)│ Overhead  │
  └─────────────────────────────────┴─────────────────────────────────┴───────────┘
  
  Training Memory Breakdown (AdamW):
  ┌───────────────┬───────────────────────────────┬───────────────┬───────────────┐
  │ Model Weights │ Optimizer States (12 bytes/p) │ Gradients     │ Activations   │
  └───────────────┴───────────────────────────────┴───────────────┴───────────────┘

2. Comprehensive Memory Formulas

A. Inference Memory

$$\text{VRAM}{\text{inference}} = (P \times b{\text{weight}}) + \text{KV_Cache_Memory} + \text{Activation_Buffer}$$

• A 70B parameter model in FP16 ($b = 2$ bytes) requires:
  • Weights: $70 \times 2 = 140\text{ GB}$.
  • KV Cache (batch 16, 4k tokens): $\sim 21.5\text{ GB}$.
  • Activation overhead: $\sim 5\text{ GB}$.
  • Total: $\sim 166.5\text{ GB}$ VRAM (Requires $4 \times 80\text{GB}$ or $2 \times 80\text{GB}$ with FP8).

B. Full Training Memory with AdamW (16-bit Mixed Precision)

For full fine-tuning with 16-bit mixed precision and standard AdamW:

• Model Weights: $2 \times P$ bytes.
• Gradients: $2 \times P$ bytes.
• AdamW Optimizer States: Master weights (FP32, 4 bytes) + Momentum (FP32, 4 bytes) + Variance (FP32, 4 bytes) = $12 \times P$ bytes.
• Total Static Training Memory: $16 \times P$ bytes!

Training a 70B model requires $70 \times 16 = \mathbf{1,120\text{ GB}}$ VRAM just for static states before activations!