AI Engineering & LLMs (50 Qs)•Part 5: Model Customization, Fine-Tuning & Evaluation
Model Evaluation Benchmarks: MMLU, GSM8K, HumanEval, SWE-bench & Their Failure Modes
1. The Standard Foundation Model Benchmark Suite
| Benchmark | Primary Capability Measured | Evaluation Methodology | Real-World Predictive Power |
|---|---|---|---|
| MMLU (Massive Multitask) | World knowledge across 57 subjects (law, math, medicine, history) | Multiple choice (A, B, C, D) | Saturated (> 88%). Susceptible to multiple-choice guessing heuristics. |
| GSM8K | Multi-step grade school arithmetic & reasoning | Exact numeric final match | High for elementary math, but easily overfitted via synthetic fine-tuning. |
| HumanEval | Basic Python function generation (164 coding problems) | pass@k on unit test executions | Moderate. Tests single, isolated functions; does not test real repositories. |
| SWE-bench | Real-world software engineering on GitHub issues | Resolves actual GitHub pull requests and passes integration test suites | Highest. Considered the gold standard for coding and agentic capability. |
| Chatbot Arena (LMSYS) | Crowdsourced human preference evaluation | Blind A/B side-by-side human voting (Elo rating) | High. Captures human vibe and conversational preference. |
2. Failure Modes & Limitations of Static Benchmarks
- Benchmark Saturation: Multiple benchmarks have hit 90%+ scores, requiring harder replacements (MMLU-Pro, SWE-bench Verified).
- Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Models are frequently overfitted specifically to benchmark phrasing.
- Multi-Choice Limitations: Selecting "B" from 4 choices does not evaluate an agent's ability to handle ambiguous real-world API workflows.