AI Engineering & LLMs (50 Qs)•Part 2: LLM Inference, Serving & Runtime Optimization
Model Quantization Techniques: PTQ vs. QAT, FP16, INT8, INT4, AWQ, GPTQ & GGUF
1. The VRAM Problem & Quantization Goals
A 70-billion parameter model stored in FP16 (16-bit floating point) requires $70 \times 2 = \mathbf{140\text{ GB}}$ of VRAM simply to hold model weights—demanding two expensive 80 GB A100/H100 GPUs. Quantization maps continuous 16-bit float weights into lower-precision discrete integers (INT8, INT4):
arduino
Continuous FP16 (-3.4e38 to +3.4e38) ──► Quantize: w_int = round(w_fp16 / Scale) + ZeroPoint ──► Discrete INT4 (-8 to +7)
2. State-of-the-Art Quantization Methods
| Quantization Method | Target Precision | Calibrated On | Hardware Target | Best Use Case |
|---|---|---|---|---|
| GPTQ | INT4 / INT8 | One-shot Hessian calibration dataset | GPU (CUDA) | High-throughput GPU inference (vLLM, TensorRT) |
| AWQ (Activation-aware) | INT4 | Identifies & protects top 1% salient weights from quantization | GPU (Tensor Cores) | Best accuracy retention among 4-bit GPU methods |
| GGUF (llama.cpp) | 2-bit to 8-bit (k-quants) | CPU/Apple Silicon/Metal | CPU / Mac / Consumer GPUs | Local edge devices, developer laptops, consumer PCs |
| FP8 (Hopper Native) | 8-bit Float (E4M3 / E5M2) | Native hardware execution on H100/H200 | NVIDIA Hopper (H100) | Zero perplexity loss with 2x speedup and 50% VRAM reduction |
3. VRAM Sizing Rule of Thumb
$$\text{Model VRAM (GB)} \approx \text{Param Count (Billions)} \times \frac{\text{Bits per Weight}}{8} \times 1.25\text{ (Overhead)}$$
- 70B Model in FP16: $70 \times 2 \times 1.25 \approx \mathbf{175\text{ GB}}$ (Requires 2x 80GB H100s)
- 70B Model in AWQ INT4: $70 \times 0.5 \times 1.25 \approx \mathbf{43.7\text{ GB}}$ (Fits on a single 80GB H100 with massive room for KV cache!)