Q9: Automated RAG Evaluation: The RAGAS Triad & Synthetic Benchmark Generation - AI Engineering & LLMs (50 Qs)

1. The RAG Triad of Evaluation

Evaluating RAG pipelines cannot rely solely on traditional NLP metrics like BLEU or ROUGE, which measure surface n-gram overlap. The industry-standard RAGAS (Retrieval Augmented Generation Assessment) framework decomposes evaluation into three core orthogonal dimensions:

graphql

                     ┌───────────────────────┐
                     │      User Query       │
                     └──────────┬────────────┘
                                │
               Context Relevance │ Answer Relevance
                                ▼
  ┌───────────────────┐                  ┌───────────────────┐
  │ Retrieved Context │◄─────────────────┤ Generated Answer  │
  └───────────────────┘   Faithfulness   └───────────────────┘

2. The Core RAG Metrics Explained

Metric	Measured Components	Failure Mode Detected
Faithfulness (Grounding)	$\frac{\text{Claims in Answer verifiable by Context}}{\text{Total Claims in Answer}}$	Hallucination: The LLM made up facts not present in retrieved context.
Answer Relevance	Semantic cosine similarity between generated answer and original query.	Drift / Evasion: The answer is truthful to context but failed to answer the user's question.
Context Precision	Signal-to-noise ratio of retrieved chunks (rank of relevant chunks).	Retrieval Noise: Irrelevant documents were retrieved ahead of relevant ones.
Context Recall	Extent to which retrieved context contains all facts needed to answer query.	Retrieval Miss: Critical ground truth was never retrieved from the vector store.

3. Automated Faithfulness Verification Script

python

import json

async def evaluate_faithfulness(llm_evaluator, context: str, answer: str) -> float:
    prompt = f"""Given the context and answer below, extract all factual statements from the answer. 
Then, verify if each statement is directly supported by the context.

Context:
{context}

Answer:
{answer}

Output valid JSON only:
{{
  "statements": [
    {{"statement": "...", "supported": true}},
    {{"statement": "...", "supported": false}}
  ]
}}"""

    res = await llm_evaluator.generate(prompt)
    data = json.loads(res)
    statements = data.get("statements", [])
    if not statements:
        return 0.0

    supported_count = sum(1 for s in statements if s.get("supported") is True)
    return supported_count / len(statements)