Module 8 — Monitoring training and stopping at the right time

A fine-tune is not a fire-and-forget operation. You launch it, and then for the next three or four hours you watch it, in the same way an anesthetist watches a patient. Not because you plan to act on every twitch, but because a handful of specific signals — if they happen — mean the run is going wrong and no amount of extra epochs will save it. This module names those signals and shows how to react to them before you burn a whole GPU-hour.

The two curves and what they mean together

Every training run produces two loss curves: training loss measured on the batches used for gradient updates, and validation loss measured on the held-out validation split every epoch (or every few hundred steps). The information is in how they move relative to each other.

Four canonical shapes cover almost every situation.

Both descending together. The healthy shape. The model is learning, and it is generalizing. Do nothing.

Training loss descending, validation loss flat. The model is memorizing the training set without extracting patterns that transfer. Common on tiny datasets or on datasets with a heavy proportion of duplicates. If it appears from step 0, your data is the problem, not your training.

Training loss descending, validation loss rising. Classic overfitting. The model is fitting noise. Stop training, take the checkpoint from just before the validation loss started rising, and either add data or reduce the number of trainable parameters (lower LoRA rank).

Both flat from the start. Learning rate too low, or dataset too small for the signal to emerge. Double the learning rate and try again.

Overfitting on a small dataset

On a two-thousand-example dataset — the size of our red-thread task — overfitting is the failure mode you should assume, not the one to be surprised by. Two or three epochs and the training loss is already noticeably below the validation loss. Do not fight it by adding regularization; instead, catch it with early stopping.

The mechanism is straightforward: at every evaluation, compare the current validation loss to the best one seen so far. If it has not improved for N consecutive evaluations (a common patience value is 3), stop training and roll back to the best checkpoint.

from transformers import EarlyStoppingCallback

trainer = Trainer(
    ...,
    args=TrainingArguments(
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=2,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        greater_is_better=False,
        ...,
    ),
    callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
)

The three arguments that matter here are load_best_model_at_end (which restores the LoRA weights from the best checkpoint at the end), metric_for_best_model="eval_loss" (which tells the trainer which checkpoint is best), and the patience. Set save_total_limit to keep only a couple of recent checkpoints so you do not fill the disk.

The loss number is not enough — generate at every epoch

Validation loss is a scalar. A scalar cannot tell you whether the model is producing valid JSON, whether the schema fields are present, whether the meeting-minutes format is respected. To catch format regressions and behavior collapses early, generate a handful of samples at the end of every epoch and inspect them by hand.

def evaluate_samples(model, tokenizer, prompts, epoch):
    model.eval()
    for i, prompt in enumerate(prompts[:5]):
        out = model.generate(
            **tokenizer(prompt, return_tensors="pt").to(model.device),
            max_new_tokens=400, do_sample=False, temperature=0.0,
        )
        text = tokenizer.decode(out[0], skip_special_tokens=True)
        print(f"--- epoch {epoch}, sample {i} ---\n{text}\n")

Five well-chosen prompts — a short meeting, a long meeting, a meeting with only one attendee, a meeting in a slightly different domain, a corrupted transcript — give you a qualitative reading of the fine-tune's state that no loss number provides.

Checkpoints, and the discipline of keeping the right ones

A LoRA checkpoint on our task is around 20 MB. Storage is not the constraint. Discipline is — knowing which checkpoint you actually deployed, which one produced which evaluation numbers, and which one you would roll back to.

The convention that works for teams that fine-tune regularly:

• Save one checkpoint per epoch during the run.
• At the end of the run, keep only two: the best-validation-loss checkpoint (the one you will deploy) and the last-epoch checkpoint (in case the run continues later).
• Delete the intermediate ones the moment the run finishes — before someone gets confused about which one is in production.

The save_total_limit=2 argument in the Trainer handles most of this automatically, but the moment you introduce experiments across multiple runs, spending fifteen minutes to set up a directory-per-run and a metrics.json at each run's root pays back tenfold.

Gradient norms: the leading indicator of a bad run

Loss is a lagging indicator; gradient norm is a leading one. A run about to blow up has its gradient norm shoot up in the seconds before the loss spikes. Watching gradient norms lets you kill a bad run in the first minute rather than after the first epoch.

Most trainers log a grad_norm metric per step. A healthy run has a gradient norm that is roughly constant across steps (with max_grad_norm=1.0 clipping keeping it below 1). A run about to fail shows the norm creeping up over ten to twenty steps before the loss curve reflects the problem.

The mid-run kill decision

The single most valuable habit for fine-tuning is being willing to kill a run at step 50 if the loss shape is wrong. A hundred steps into a three-hundred-step run, you already have most of the information about whether the configuration is workable. If the training loss is still at its initial value, the learning rate is too low. If it has spiked and not recovered, the learning rate is too high. If the validation loss is already above the training loss by a factor of two, the dataset is not diverse enough.

Kill the run, change one hyperparameter (module 7's priority list), restart. You will burn less compute doing three runs of a hundred steps each than one full run that ends nowhere.

Log to a place you will actually read

The Trainer's report_to="none" in module 7 is a sane default for a first run — but the moment you do more than one run, wire it to Weights & Biases, MLflow or even a local TensorBoard. Comparing three runs side by side is what turns hyperparameter tuning from guesswork into engineering, and grepping through terminal logs is not that.

Summary

• The two loss curves (training, validation) diagnose most failure modes: rising validation is overfitting, flat training is learning rate too low, both flat is a data problem.
• On a small dataset like our two thousand pairs, expect overfitting and set up early stopping with load_best_model_at_end before you launch the run.
• Generate a handful of qualitative samples at each epoch — the loss scalar cannot detect format regressions or schema drift.
• Keep only two checkpoints per run (best and last), and use gradient norm as a leading indicator of a run that is about to blow up.

Next module: turning the best LoRA checkpoint into a merged, exportable model ready for llama.cpp and Ollama.