Module 9 — Systematic prompt evaluation

Every previous module ended with the same refrain: "measure on your actual test set". This one builds that test set and the code around it. Without a proper evaluation loop, prompt engineering degenerates into the "vibes-based iteration" that ate the first year of most LLM projects: someone tries something, thinks it looks better, ships it, and finds out three weeks later that it was worse.

What a proper test set looks like

For the running case, a useful test set has three properties:

• Fifty emails, hand-annotated with the four expected fields. Fifty is a working minimum: below it, differences between prompts are drowned in noise; above two hundred, annotation cost outgrows marginal insight.
• Realistic distribution: proportions of urgency levels, product types, message lengths and languages match your production traffic. A test set full of easy polite emails guarantees a false sense of quality.
• Hard cases included on purpose: five sarcastic emails, three with missing product names, three with attempted injection, two ALL-CAPS threats. These are the emails a production system fails on, so they belong in the test set on day one.

The annotation is the expensive step and the one no library can save you from. Two people annotating the same fifty emails independently and reconciling their disagreements produces a gold set that is worth ten times a solo annotation.

The data structure

Store the test set as a plain file — JSON Lines is convenient:

# eval_set.jsonl
{"id": "e001", "email": "Hi, my QuietPro X3 stopped...", "gold": {"reason": "charging failure", "product": "QuietPro X3", "urgency": "high", "requested_action": "replacement or refund"}}
{"id": "e002", "email": "Whenever you get a chance...", "gold": {...}}

The id matters more than it looks. It lets you point to a single row, share it with a colleague, and track which specific email a prompt version got wrong across time.

Per-field metrics

A single "did the model get the answer right" score hides everything useful. Split it per field, and by kind of error:

from collections import defaultdict

def evaluate(prompt_fn, eval_set):
    scores = defaultdict(lambda: {"correct": 0, "total": 0})
    errors = []
    for row in eval_set:
        pred = prompt_fn(row["email"])
        for field, gold_value in row["gold"].items():
            scores[field]["total"] += 1
            if match(pred.get(field), gold_value, field):
                scores[field]["correct"] += 1
            else:
                errors.append({"id": row["id"], "field": field,
                               "gold": gold_value, "pred": pred.get(field)})
    return scores, errors

The match function embeds domain knowledge: exact match for urgency (a categorical), lower-cased substring match for product, semantic tolerance for reason and requested_action where wording varies. Bundle those matchers in one place, treat them like unit tests, and change them only deliberately — a changed matcher changes every score you compare afterwards.

Per-field scoring lets you see, for instance, that a change lifts urgency from $0.72$ to $0.84$ while dropping product from $0.96$ to $0.91$. That trade-off is the real conversation, and a single scalar hides it entirely.

A/B comparing two prompt versions

Compare two prompts by running both on the same test set with the same matchers:

scores_v1, errors_v1 = evaluate(prompt_v1, eval_set)
scores_v2, errors_v2 = evaluate(prompt_v2, eval_set)

for field in scores_v1:
    a1 = scores_v1[field]["correct"] / scores_v1[field]["total"]
    a2 = scores_v2[field]["correct"] / scores_v2[field]["total"]
    print(f"{field}: v1={a1:.2f}  v2={a2:.2f}  Δ={a2-a1:+.2f}")

Two useful refinements:

• Statistical significance: fifty examples is too small to trust a two-point difference. Compute a binomial confidence interval, or use a paired bootstrap. If the interval crosses zero, do not claim an improvement.
• Error diffs: the set of examples v1 got wrong and v2 got right is often the most informative output. It shows you what the change actually did — sometimes not what you intended.

The rule of thumb: a two-point lift on fifty examples is noise, a five-point lift is a signal, a change that regresses hard cases is a regression even if the aggregate improves.

Cost of evaluation

Running fifty examples on every prompt change is not free. At $2000$ input tokens plus $200$ output tokens per call on a model at $0.15$/$0.60$ per million:

200 \times 6 \times 10^{-7}) \approx 0.021 \text{ dollars.}$$ Two cents per full evaluation. Cheap enough to run on every commit, which is where continuous evaluation belongs. Do it locally during development, in continuous integration on merges, and against a larger sample nightly. Latency is more of a concern: fifty calls in sequence take a while. Parallelise them with a bounded thread pool — most providers tolerate five to ten concurrent calls per key. :::warning[Evaluating with an LLM is not free of bias] "LLM as judge" — asking a model to score another model's output — is practical for open-ended fields, but the judge shares failure modes with the model under test. Use it in addition to string-level matchers, never in place of them, and periodically compare judge scores to human scores on a small sample. ::: ## Detecting regressions after a model change Providers deprecate models, roll out new versions, and quietly change defaults. The test set that guards against your prompt changes also guards against theirs. A minimal regression harness: ```python for model in ["gpt-4o-mini", "gpt-4o-mini-2026-07-15"]: prompt_v = build_prompt(model=model) s, _ = evaluate(prompt_v, eval_set) print(model, {k: v["correct"]/v["total"] for k, v in s.items()}) ``` Run this on every provider release announcement. A per-field drop of more than three points on any field is worth investigating before switching, or before letting the provider auto-upgrade you. ## What not to evaluate on Two mistakes are worth naming, because they are common and expensive: - **Do not evaluate on inputs also used as few-shot examples.** The model has literally seen the answer in the prompt; the score is meaningless. Keep the example pool separate from the eval set. - **Do not evaluate only on the inputs the model got right last time.** Confirmation bias is real. Sample includes hard cases and failures each time; the size does not have to grow. :::tip[Log every production run] The best test sets are grown from production, not written up front. Log every input and every output; sample the log periodically; annotate the samples that surprised you; add them to the eval set. Six months in, your eval set is a photograph of your product's real failure modes, not the guess you made on day one. ::: ## In summary - A useful test set is **fifty hand-annotated examples**, matching production distribution and **including hard cases on purpose**; two-person annotation with reconciliation is worth the time. - **Score per field, not overall**; different fields exercise different parts of the prompt and hide their own trade-offs when aggregated. - **A/B compare** on the same test set with the same matchers, treat a two-point lift on fifty examples as noise, and read error diffs for what actually changed. - The same harness **detects provider-side regressions** on model changes and **grows from production logs** into a photograph of your real failure modes, not your day-one guesses. Next module: turning the winning prompts into a reviewed, versioned, reusable library.