Q43: RLHF vs. DPO (Direct Preference Optimization): Alignment Mechanics & Loss Formulations - AI Engineering & LLMs (50 Qs)

1. The Classical RLHF Pipeline (PPO-Based)

Reinforcement Learning from Human Feedback (RLHF) was foundational to InstructGPT and early LLMs. It requires a cumbersome 3-stage process:

less

  Stage 1: Supervised Fine-Tuning (SFT) on clean instruction dataset.
                 │
                 ▼
  Stage 2: Train a separate Reward Model (RM) on human preference pairs (y_win > y_loss).
                 │
                 ▼
  Stage 3: Optimize Actor Policy using PPO (Proximal Policy Optimization).
           Requires hosting 4 separate models in VRAM concurrently:
           [ Actor (Active) ] [ Critic ] [ Reward Model ] [ Reference (Frozen) ]

Downside: Notoriously unstable training, hyperparameter sensitivity, and extreme GPU memory requirements.

2. Direct Preference Optimization (DPO)

Pioneered by Rafailov et al. (Stanford), DPO derives an analytical closed-form solution proving that the optimal policy can be trained directly on preference pairs without ever training a separate reward model or using reinforcement learning!

$$\mathcal{L}{\text{DPO}}(\pi\theta; \pi_{\text{ref}}) = -\mathbb{E}{(x, y_w, y_l)} \left[ \log \sigma \left( \beta \log \frac{\pi\theta(y_w | x)}{\pi_{\text{ref}}(y_w | x)} - \beta \log \frac{\pi_\theta(y_l | x)}{\pi_{\text{ref}}(y_l | x)} \right) \right]$$

3. Comparison Matrix

Vector	RLHF with PPO	Direct Preference Optimization (DPO)
Required Models in VRAM	4 models (Actor, Critic, Reward, Reference)	2 models (Active Policy $\pi_\theta$ and Reference $\pi_{\text{ref}}$)
Training Stability	Low (PPO policy collapses, mode drop)	High (Standard cross-entropy classification loss)
Complexity	Extremely High (requires RL expertise)	Low (Simple PyTorch training loop)
Alignment Quality	State-of-the-art	Matches or exceeds PPO on human evaluation benchmarks