Skip to main content

Module 9 — Evaluation: benchmarks and human judgment

Modules 3 to 8 assembled the customer-support assistant. Before it goes to production, one question dominates: is it good enough? And the follow-up question, at least as important: is the version we are about to deploy better than the one currently running?

Public benchmarks give a first answer. It is far less trustworthy than most model comparisons suggest, and this module explains why, and how to build the evaluation you actually need.

MMLU and what it does not measure​

MMLU (Massive Multitask Language Understanding, 2021) is the most-cited benchmark in every model card. It contains ~15 000 multiple-choice questions across 57 subjects, from elementary maths to professional law.

MMLU has three properties worth remembering:

  1. It is English-only and heavily academic — biased towards US-style curricula.
  2. It measures factual recall and pattern matching on short questions, not reasoning, not tone, not agent behaviour.
  3. It has been partially memorised by every recent model, because MMLU questions are on the public web and end up in pretraining crawls.

That third point is the elephant in the room. When a new model announces "76.5 on MMLU", you cannot tell how much of that score is genuine capability and how much is memorisation of the test set. This is not a hypothetical concern: several papers have documented near-verbatim MMLU questions appearing in the training corpora of major open models.

Contamination, quantified​

Test-set contamination happens when questions from a benchmark leak into training data. Two ways to detect it:

  • N-gram overlap: search the pretraining corpus for the exact wording of test questions. A hit means direct leakage.
  • Reverse-completion probe: give the model the first half of a benchmark question and see if it completes with the exact second half. Verbatim completion of a 20-token question is not a coincidence.

The 2024 paper by Golchin and Surdeanu showed that flagship models complete 20 to 40 % of MMLU questions verbatim. Corrections exist — MMLU-Redux, hand-cleaned versions with rewritten wording — and they routinely drop reported scores by several points.

Consequence: use public benchmarks to rule out candidates ("this 3B model scores 30 on MMLU, do not consider") but never to pick between top contenders ("this 7B beats that 7B by 1 point on MMLU"), which is noise plus contamination.

Arenas and Elo​

The Chatbot Arena (LMSYS) sidesteps contamination by asking humans to compare two anonymous model outputs on an open-ended prompt and pick the better one. Millions of pairwise comparisons feed into an Elo ranking, the same system used in chess.

Two properties make Elo useful:

  • Preferences are cheaper than ground truth. Nobody has to write a reference answer; they just pick.
  • Open-ended prompts are hard to memorise. Contamination is much lower than on structured benchmarks.

Two properties make Elo imperfect:

  • Style bias. Longer, more confident answers win preferences more often, whether or not they are correct. Some models game this deliberately.
  • Distribution mismatch. Arena prompts are what curious enthusiasts type, not what your users type. A model that ranks #1 on Arena may rank #5 on your traffic.

LLM-as-a-judge​

Human evaluation does not scale to every code change. The standard cheap approximation is the LLM-as-a-judge pattern: use a strong model (often GPT-4-class) to score outputs of a weaker one.

judge_prompt = """You will be given a customer question and two candidate answers.
Judge which answer is better on the criteria: factual correctness, helpfulness,
tone appropriate for customer support. Reply with exactly A, B or TIE.

Question: {question}
Answer A: {answer_a}
Answer B: {answer_b}

Verdict:"""

def judge(question, a, b, model):
    prompt = judge_prompt.format(question=question, answer_a=a, answer_b=b)
    return model.generate(prompt, temperature=0).strip()

Known limitations of this pattern:

  • Position bias. LLM judges favour whichever answer appears first. Always run each pair in both orders and count only agreed verdicts.
  • Verbosity bias. Longer answers win more often, again regardless of correctness.
  • Self-preference. GPT-4 judged its own outputs as best. Do not judge a model with itself.

Used carefully, LLM-as-a-judge correlates ~80 % with human preferences on general tasks, and it is what makes evaluation cheap enough to run on every deploy.

Build your own business evaluation set​

The single highest-leverage evaluation for the customer-support project of the running example is your own set of 200 to 500 realistic questions, each with a reference answer or a set of allowed answers. This is what tells you whether the model works for your users, and it is immune to public contamination by construction.

The recipe:

  1. Sample 300 real support tickets from the last quarter, redacted.
  2. Have a senior agent write the reference answer for each.
  3. Split 80/20 into a validation set and a held-out test set.
  4. Run every candidate model on the validation set; use LLM-as-a-judge and spot-check with humans.
  5. Only rank models on the held-out test set once — the moment you use it to pick, it stops being held-out.
import json
from pathlib import Path

def evaluate(model, dataset_path):
    ds = [json.loads(l) for l in Path(dataset_path).read_text().splitlines()]
    scored = []
    for row in ds:
        answer = model.generate(row["question"], temperature=0.3)
        judgement = judge(row["question"], row["reference"], answer, judge_model)
        scored.append({"id": row["id"], "answer": answer, "judgement": judgement})
    return scored

Categorise your questions: refund, delivery, warranty, account, product info. Track scores per category. Aggregate scores hide catastrophic regressions on rare but important categories.

Do not trust a single-number leaderboard

"Model X beats Y by 2 %" on any single benchmark is not a decision-grade signal. Read at least three benchmarks, one Arena view, and your own business evaluation before switching production. The cost of a bad switch — degraded quality, opaque regressions, angry users — is much higher than the cost of one extra week of evaluation.

Freeze the judge across comparisons

When comparing two versions of your model, keep the judge model, the judge prompt, the temperature and the seed identical. Any variation in the judge is a variation in what you are measuring. Judge upgrades belong to their own separate PR.

In summary​

  • MMLU and public benchmarks are contaminated to a measurable degree; use them to reject weak candidates, never to pick between top contenders on a small gap.
  • Chatbot Arena Elo avoids most contamination but suffers from style bias and a distribution mismatch with your users' traffic.
  • LLM-as-a-judge scales evaluation to every deploy, at the cost of position and verbosity biases that must be controlled for.
  • The evaluation that matters is your own 200 to 500 realistic questions, categorised by business intent and split into a validation and a held-out test set.

Next module: costs, latency and architecture. With behaviour, safety and evaluation in place, the last decision is how to serve it and at what price.