AI Engineering & LLMs (50 Qs)•Part 5: Model Customization, Fine-Tuning & Evaluation
Parameter-Efficient Fine-Tuning (PEFT) & LoRA: Low-Rank Adaptation Mechanics
1. The Cost of Full Parameter Fine-Tuning
Full fine-tuning updates all weights in a neural network:
- For a 70B parameter model, storing gradients, activations, and AdamW optimizer states requires over 1,200 GB of GPU VRAM (demanding multi-node GPU clusters).
- Updating all weights frequently leads to Catastrophic Forgetting (destroying general reasoning abilities).
2. Low-Rank Adaptation (LoRA) Mathematical Formulation
Introduced by Hu et al. (Microsoft), LoRA hypothesizes that weight updates during adaptation have a low "intrinsic dimension". Instead of modifying the original frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA freezes $W_0$ and decomposes the update $\Delta W$ into two low-rank matrices $A$ and $B$:
$$h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} (B \cdot A) x$$
css
Input Vector x (d dimensions)
│
┌───────────────┴───────────────┐
▼ ▼
Frozen Weight Matrix W0 LoRA Matrix A (d x r)
(d x k) │
│ Rank r (e.g., r = 8)
│ ▼
│ LoRA Matrix B (r x k)
│ │
└───────────────┬─────────────────────┘
▼
Output Vector h
Where:
- $A \sim \mathcal{N}(0, \sigma^2)$ and $B = 0$ at initialization (so $\Delta W = 0$ on step 0).
- $r \ll \min(d, k)$ (typically $r = 8, 16, \text{ or } 32$).
- $\alpha$ is a scaling constant (often $\alpha = 2r$).
3. VRAM & Storage Benefits
- Trainable Parameters: Reduced by 99.9% (e.g., training only 40 million parameters instead of 70 billion).
- Storage: The base model stays unchanged. You only store the small LoRA adapter weights (20 MB to 200 MB vs 140 GB).
- Zero Inference Latency: At inference time, you can merge weights: $W_{\text{final}} = W_0 + \frac{\alpha}{r}(B \cdot A)$ with zero added latency!