AI Engineering & LLMs (50 Qs)•Part 5: Model Customization, Fine-Tuning & Evaluation
LLM-as-a-Judge Design: Mitigating Position, Verbosity & Self-Enhancement Biases
1. Automated Evaluation via Judge LLMs
Evaluating thousands of complex model outputs with human reviewers costs $10,000s and takes weeks. LLM-as-a-Judge (Zheng et al.) uses an advanced model (e.g. GPT-4o, Claude 3.5 Sonnet) to judge candidate model outputs:
css
User Query + Reference Solution
│
├── Candidate Model Output A
├── Candidate Model Output B
│
▼
[ Judge LLM Prompt & Rubric ]
"Evaluate outputs on correctness, depth, and safety.
Assign numeric score (1-5) and declare Winner (A, B, or Tie)."
2. The Three Systematic Biases in LLM Judges & Mitigations
| Bias Type | Description | Production Mitigation |
|---|---|---|
| Position Bias | The judge model disproportionately prefers whichever answer is presented as "Candidate A" (First) or "Candidate B" (Second). | Swap Positions & Average: Run evaluation twice: (A, B) and (B, A). Declare a win only if the model remains consistent. |
| Verbosity Bias | The judge model routinely prefers longer, fluffier responses over concise, correct answers. | Explicitly instruct the rubric: "Penalize unnecessary verbosity; prioritize conciseness." Normalize length. |
| Self-Enhancement Bias | Models give systematically higher ratings to outputs generated by their own model family. | Use cross-provider judges (e.g., use Claude to evaluate GPT-4, and GPT-4 to evaluate Claude). |
3. Production Swapped-Pairwise Judge Implementation
python
async def pairwise_judge_with_swap(judge_client, query: str, ans_a: str, ans_b: str) -> str:
# Round 1: A first, B second
eval_1 = await run_judge(judge_client, query, candidate_1=ans_a, candidate_2=ans_b)
# Round 2: B first, A second (Position Swap)
eval_2 = await run_judge(judge_client, query, candidate_1=ans_b, candidate_2=ans_a)
# Reconcile judgments
win_1 = "Candidate 1" in eval_1
win_2 = "Candidate 2" in eval_2
if win_1 and win_2:
return "Model A Wins"
elif not win_1 and not win_2:
return "Model B Wins"
return "Tie / Inconclusive"