Module 8 — Evaluation: faithfulness, relevance, coverage

Modules 4 to 7 built a pipeline that produces answers. This module answers the harder question — are those answers right? Without evaluation, every design decision so far has been guesswork; with a small annotated set and three metrics, the same decisions become measurable.

Why a single accuracy number is not enough

A RAG pipeline has three failure modes that no single score can tell apart:

• The retriever misses the right passage → recall problem
• The retriever finds it, the model ignores or fabricates → faithfulness problem
• The retriever finds part of it, the model summarises the visible part and leaves out the rest → coverage problem

Report a single "accuracy" number and every problem looks like the same problem. Three separate metrics let you spend engineering time where the loss actually is.

Building the annotated set

An evaluation set for the red thread is 100 questions — enough for stable numbers, small enough that one afternoon of work produces it. For each question, record four fields:

Field	Purpose
question	What a user might ask
gold_answer	The correct answer, in one or two sentences
gold_sources	The document(s) that contain the answer
type	Simple lookup, multi-hop, contradiction, abstention

The type column is what makes evaluation actionable. On the red thread, roughly 60 % of questions are simple lookups, 25 % require combining two documents, 10 % test contradictions, and 5 % should trigger abstention (they are about something the corpus does not cover). Tracking per-type metrics catches regressions the average would hide.

import json
from pathlib import Path

def load_gold(path: str) -> list[dict]:
    return [json.loads(l) for l in Path(path).read_text(encoding="utf-8").splitlines()]

gold = load_gold("eval/gold-procedures.jsonl")

Retrieval recall: the cheapest and most useful metric

Recall at $k$ measures the fraction of gold questions for which at least one gold source appears in the top-$k$ retrieved passages. It is the fastest metric to compute — no language model call — and it upper-bounds every downstream metric. If retrieval recall at $k = 5$ is 0.60, no reranker or prompt can push end-to-end accuracy above 0.60.

$$\text{recall}@k = \frac{1}{|\text{gold}|} \sum_{q \in \text{gold}} \mathbb{1}\!\left[\, \text{gold sources of } q \cap \text{top-}k(q) \neq \varnothing \,\right]$$

def evaluate_retrieval(gold: list[dict], retriever, k: int = 5) -> dict:
    hits = 0
    for row in gold:
        passages = retriever(row["question"], user_context=row["user_context"])[:k]
        found = {p["source_name"] for p in passages}
        if found & set(row["gold_sources"]):
            hits += 1
    return {"recall@k": hits / len(gold), "k": k}

Run it after every change to extraction, chunking, embedding or retrieval. It takes seconds and is the most reliable regression detector in the whole pipeline.

Answer faithfulness: is every claim grounded?

Faithfulness asks a different question: given the passages that were shown to the model, are every claim in the answer supported by those passages? An answer can be relevant (about the right topic) and still unfaithful (invents a number). Both matter, and they measure different things.

The practical method is LLM-as-judge: another model call, on a cleanly designed prompt, that reads the passages and the answer and outputs a score.

JUDGE_FAITHFULNESS = """You are evaluating a factual answer.

Passages:
{passages}

Answer:
{answer}

For each factual claim in the answer, decide if it is supported by the
passages. Reply as JSON:
{{
  "claims": [{{ "text": "...", "supported": true }}, ...],
  "score": <fraction of supported claims, from 0 to 1>
}}"""

def score_faithfulness(passages: list[dict], answer: str) -> float:
    prompt = JUDGE_FAITHFULNESS.format(
        passages="\n\n".join(f"[S{i+1}] {p['text']}" for i, p in enumerate(passages)),
        answer=answer,
    )
    out = json.loads(llm.generate(prompt, temperature=0.0))
    return float(out["score"])

Three warnings worth their weight in bugs saved:

• Same-model bias: a judge from the same family as the generator systematically over-rates its output. Use a different family (or at least a different provider) as judge.
• Temperature 0: the judge must be reproducible; a run-to-run variance of 10 % ruins the signal.
• Human calibration: annotate 30 items by hand once, and check that the automatic judge agrees at least 85 % of the time. If not, the judge prompt is the bug, not the pipeline.

Coverage: did the answer say what it needed to?

Coverage checks that the gold answer's key facts appear in the produced answer. It is the mirror of faithfulness: faithfulness catches over-claiming, coverage catches under-claiming.

JUDGE_COVERAGE = """You are evaluating a factual answer against a reference.

Reference:
{gold}

Answer:
{answer}

List each fact from the reference. For each, decide if the answer contains it
(literally or in equivalent form). Reply as JSON with the same structure as
faithfulness: claims list plus a score from 0 to 1."""

An answer that abstains has coverage 0 by construction, which is what you want: it is honest, not correct. Report abstention separately as a fourth metric.

Recall versus faithfulness: the diagnostic table

Once all three metrics are running, a two-by-two table diagnoses most problems in an afternoon:

Recall	Faithfulness	Diagnosis	Where to look
High	High	Pipeline is healthy	Look for coverage gaps or edge cases
High	Low	Model ignores the context	Prompt, temperature, abstention rule (module 7)
Low	High	Retrieval misses the gold source	Chunking, embeddings, hybrid search (modules 3–5)
Low	Low	Both broken	Fix retrieval first — faithfulness cannot rise above what is retrieved

This is the single most useful table in the course. Print it, pin it, come back to it every time a metric drops.

RAGAS in one glance

RAGAS is a Python library that packages the metrics above (and a few more: context precision, context recall, answer semantic similarity) with prebuilt judge prompts. It is a fine second step: use it once your own scripts are working, to cross-check numbers and to add metrics you did not think of.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall

dataset = build_ragas_dataset(gold, run_pipeline)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_recall])
print(result.to_pandas())

Do not start with RAGAS. Its abstractions hide precisely the details this module made explicit, and debugging a bad number is much harder when you cannot see the judge prompt. Start with the three scripts above, understand what they measure, then let RAGAS take over the boilerplate.

Chunk-level attribution: the qualitative debugger

Numbers tell you what is wrong, not why. For every failing question, log the top-10 retrieved chunks, the reranked shortlist, the prompt and the answer. Read 20 of them by hand every week. Patterns emerge fast: a specific document that is systematically mis-chunked, an acronym that only BM25 finds, a class of questions that trigger the same wrong abstention.

def log_run(question: str, gold: dict, run: dict) -> dict:
    return {
        "question": question,
        "gold_sources": gold["gold_sources"],
        "retrieved_top_10": [c["chunk_id"] for c in run["retrieved"][:10]],
        "reranked_top_5": [c["chunk_id"] for c in run["reranked"][:5]],
        "answer": run["answer"],
        "metrics": {
            "recall@5": recall_at_k(gold, run["retrieved"], 5),
            "faithfulness": score_faithfulness(run["reranked"][:5], run["answer"]),
        },
    }

A good average hides a bad worst case

Report percentiles, not just means. A pipeline with 92 % average faithfulness can still be unusable in the 10 % of cases where it invents deadlines. Track P10 (the worst 10 %) and, for anything user-facing, gate deployments on it, not on the mean.

In summary

• Split evaluation into three metrics — recall, faithfulness, coverage — plus abstention, because they diagnose different failures.
• Build a 100-question gold set with typed questions (lookup, multi-hop, contradiction, abstention) and per-type metrics catch what averages hide.
• Use LLM-as-judge with temperature 0, from a different model family than the generator, and calibrate it on 30 hand-annotated items before trusting it.
• The recall × faithfulness table diagnoses most problems in an afternoon; start with your own scripts, then add RAGAS as a cross-check.

Next module: bringing the bill under control — caching embeddings and answers, breaking down the cost per question, reindexing incrementally, and knowing what to log.