Skip to main content

Module 10 — Evaluation before and after fine-tuning

The training loss ended low, the samples at every epoch looked reasonable, and the Ollama model responds fast. That is not evaluation — that is a set of leading indicators. This module builds a real evaluation: a comparison between the base model and the fine-tuned one, measured on data neither has seen, along the two axes that matter (target task, general capability).

The held-out test set​

Module 2 asked you to freeze a few hundred examples from the very start, before any hyperparameter tuning. This is where they come out of the freezer. That test set is the one number you get to report, and it is the one number you must not touch during development — every look at it that shapes a decision reduces it to a second validation set, and eventually to a second training set.

For our meeting-minutes task, 300 held-out (transcript, minutes) pairs is a good size. Small enough to run through the model in ten minutes, large enough that a two-point difference is statistically meaningful.

Two axes to measure​

Fine-tuning has two goals in tension. The first is task improvement: the model does the meeting-minutes job better than it did before. The second is capability preservation: the model has not lost the ability to reason, follow multi-turn conversations, refuse unsafe requests, or handle English grammar. A fine-tune that scores well on the first axis and badly on the second is a regression, not a success.

You measure the two axes separately.

Axis one: task metrics​

For a structured-output task like meeting minutes, task metrics split into format metrics and content metrics.

Format metrics are cheap and mechanical. Every generated minutes should parse as valid JSON, contain the required fields, use the right types (list, string, ISO date). Write a validator that returns a boolean per example, then report the pass rate on the held-out set.

import json

def format_ok(text, schema):
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return False
    for field, expected_type in schema.items():
        if field not in obj:
            return False
        if not isinstance(obj[field], expected_type):
            return False
    return True

schema = {"attendees": list, "decisions": list, "action_items": list}
format_rate = sum(format_ok(g, schema) for g in generated) / len(generated)

A well-trained meeting-minutes fine-tune scores 98 % or better on format. The base model with the same prompt typically scores 60 to 80 % — the format is where fine-tuning shines, and the improvement will be dramatic.

Content metrics are harder. A generated summary is not "correct" or "incorrect" in a single number; it is faithful or not, complete or not, concise or not. Three approximate measures work in practice.

  • Field-level exact match on structured fields (attendees, action-item owners). A fine-tuned model that misspells attendee names has learned the wrong thing.
  • ROUGE or BERTScore on free-text fields (decisions, meeting summary). Not perfect, but comparable across model versions.
  • LLM-as-a-judge: prompt a stronger model (GPT-4-class) to score each generated output against a reference on faithfulness and completeness, on a 1-to-5 scale. Cheap, fast, and calibrated well enough for internal comparisons — but never for external claims.

Axis two: human judgment, blind, on the hard cases​

LLM-as-a-judge is convenient. It is also biased in ways you cannot always predict. For every fine-tune you plan to ship, do a blind human evaluation on at least fifty examples.

The mechanics matter. Present the human evaluator with pairs of outputs — one from the base, one from the fine-tune — in random order, without labels. The evaluator picks the better output, or declares a tie. Aggregate across fifty pairs and compute the win rate.

A win rate of 60 % or above against the base is a real improvement. 50 to 60 % is a modest gain that may or may not justify the training cost. Below 50 %, the fine-tune is not helping and you have work to do.

Two rules to protect the blindness. The evaluator must not know which side is which. The examples must be from the held-out test set, not from the training data (a common mistake). If either rule breaks, throw the numbers away — a biased evaluation is worse than none.

Axis two continued: capability regression​

Module 3 warned about catastrophic forgetting: even parameter-efficient methods can degrade general capabilities on tasks unrelated to the fine-tune. The measurement is straightforward and worth doing on every fine-tune, no matter how well it scores on the target task.

Pick a small general-capability benchmark, run it on both the base and the fine-tuned model, and compare. MMLU (56 subjects of general knowledge, four-choice questions), HellaSwag (common-sense completion), ARC (elementary science reasoning) each take about an hour on a 7B model with lm-evaluation-harness.

lm_eval --model hf --model_args "pretrained=./merged-mistral-minutes,dtype=bfloat16" \
        --tasks mmlu,hellaswag,arc_easy \
        --batch_size 16 --output_path ./eval-fine-tuned

lm_eval --model hf --model_args "pretrained=mistralai/Mistral-7B-v0.3,dtype=bfloat16" \
        --tasks mmlu,hellaswag,arc_easy \
        --batch_size 16 --output_path ./eval-base

Expected result for a well-trained LoRA fine-tune: a drop of 0 to 2 points on each benchmark. Anything larger — 5 points or more — means the fine-tune has degraded the base's general skills, and you should investigate. Common causes: too high a learning rate, too many epochs, a dataset with a very narrow style. The fix is usually a shorter run at a lower learning rate, sometimes with a fraction of general instruction data mixed in.

Putting it in a report​

Everything above rolls up into a single evaluation report that a decision-maker can read in three minutes.

Fine-tune: mistral-7b-minutes-v3
Held-out set: 300 meeting-minutes pairs

Task metrics (target axis)
  Format pass rate:      98.3 %  (base: 71.7 %,  +26.6 pt)
  Field-level EM:        94.1 %  (base: 62.8 %,  +31.3 pt)
  ROUGE-L on summary:    0.47    (base: 0.31,    +0.16)
  LLM-judge (avg 1-5):   4.2     (base: 3.1,     +1.1)

Human evaluation (blind, 50 pairs)
  Win rate over base:    72 %   (28 wins / 11 ties / 11 losses)

Capability regression (general axis)
  MMLU:                  62.4 %  (base: 62.9 %,  -0.5 pt)
  HellaSwag:             81.1 %  (base: 81.6 %,  -0.5 pt)
  ARC-Easy:              79.8 %  (base: 80.4 %,  -0.6 pt)

That layout tells a full story. Big gains on the target task, meaningful human preference, negligible general regression. This fine-tune ships. Change any of the three sections in a bad direction, and the recommendation changes with it.

Save the evaluation results with the checkpoint

The moment you produce a numbers table like the one above, save it as eval.json next to the model checkpoint, and never separate the two. Six months later, the question "what was the MMLU number of the model we shipped in production?" has an answer that does not require rerunning anything.

Summary​

  • Use a held-out test set frozen at dataset-preparation time; every look at it during development turns it into a second validation set.
  • Measure the target task on two levels: format compliance (mechanical, high-signal) and content quality (LLM-judge or ROUGE plus a blind human evaluation on 50 pairs).
  • Always measure capability regression with MMLU, HellaSwag or ARC on both the base and the fine-tune; a drop above 5 points is a red flag.
  • Roll everything into one three-section report — target metrics, human evaluation, general regression — and store it next to the checkpoint.

Next: the recap and the 40-question exam that ends the course.