Lesson 4 — Evaluating honestly
This is the lesson that separates people who produce results from people who produce numbers. Building a model is comparatively easy. Knowing whether it works is the skill, and the discipline required is procedural rather than clever.
The three-way split
You cannot evaluate a model on data it learned from, for the same reason you cannot assess a student using the exact questions they revised. The defence is to divide the data before doing anything else.
| Portion | Typical share | What it is for | How often you look |
|---|---|---|---|
| Training | 70% | the model learns from this | constantly |
| Validation | 15% | choosing settings, deciding when to stop | many times |
| Test | 15% | one final honest estimate | once |
The reason for three rather than two is subtle and important. Every time you use a set to make a decision, you leak a little information about it into your model. Choose between twenty configurations on the validation set and you have implicitly fitted to it, so its score is now optimistic. The test set exists to be untouched by that process.
Once you have looked at the test set to choose between two models, it is no longer a test set. Its score no longer estimates real-world performance. Teams do this without noticing, then find production accuracy several points below the figure in their presentation, and blame the deployment.
Cross-validation
With limited data, a single 15% test set may hold only a hundred rows, and a hundred rows produce a score that swings depending on which hundred you got.
k-fold cross-validation solves this. Split the data into k parts, typically five. Train on four, evaluate on the fifth. Repeat so each part serves as the evaluation set once. You end up with five scores.
Two things you gain, and the second is the one people undervalue:
A more reliable average, since every row was evaluated exactly once.
A measure of variability. Five scores of 84, 85, 84, 86, 85 mean something quite different from 78, 91, 82, 89, 80 with the same average. The second model is unstable, its performance depends heavily on which data it saw, and reporting only the mean hides that entirely. Always report the spread.
For imbalanced problems, stratified k-fold keeps the class proportions in each fold, without which a fold may contain almost none of the rare class and produce nonsense.
Time series need a different split
If your data is ordered in time, random splitting is invalid and produces beautifully wrong results. Training on rows from next week to predict this week means the model has seen the future, which is not available in production.
The correct approach splits chronologically: train on the past, evaluate on the following period, and roll the window forward. It gives lower scores than random splitting, and those lower scores are the real ones.
Always build a stupid baseline
Before any modelling, establish what a trivial approach achieves. This single habit prevents more embarrassment than any other.
- Classification: always predict the majority class
- Regression: always predict the average
- Time series: predict that tomorrow equals today
- Recommendation: recommend the most popular item
Then compare. A churn model scoring 94% accuracy sounds excellent until you notice that 94% of customers do not churn, so "nobody churns" scores identically while detecting nothing. That comparison takes two minutes and has saved countless projects from presenting a useless model as a success.
The baseline also tells you whether the problem is worth pursuing. If a sophisticated model beats the trivial one by half a point, the signal in your data may simply be weak, and no amount of tuning will change that.
Choosing the metric
The metric encodes what you consider a good outcome, and choosing it carelessly means optimising for the wrong thing with great precision.
The confusion matrix is where to start. For a binary problem it is four numbers:
| Model says yes | Model says no | |
|---|---|---|
| Truly yes | true positive | false negative — a miss |
| Truly no | false positive — a false alarm | true negative |
Every metric is a way of weighing those four cells, and which weighting is right depends entirely on the cost of each mistake:
- Disease screening: a miss can be fatal, a false alarm means an extra test. Maximise recall.
- Spam filtering: a miss is a nuisance, a false alarm loses an important email. Maximise precision.
- Fraud detection: both cost money, and there are few real cases. Use precision-recall AUC, not accuracy.
- Price prediction: use mean absolute error when all errors are equally bad, root mean squared error when large errors are disproportionately bad.
Ask the person who will use the model: "which would you rather have, a false alarm or a miss?" Their answer determines your metric and your threshold. Choosing the metric yourself, in isolation, is how you deliver a technically excellent model that nobody wants.
Reading the results
A few diagnostics worth internalising, from the Introduction to AI course:
| What you see | What it means | What to do |
|---|---|---|
| Poor on training, poor on test | underfitting | richer model, better features |
| Excellent on training, poor on test | overfitting | more data, simplify, regularise |
| Good on both | genuine learning | check the baseline, then ship |
| Suspiciously perfect | almost certainly leakage | audit every feature |
| Wide spread across folds | unstable | more data, simpler model, report the range |
| Great offline, poor in production | drift, or a train-serve gap | compare live inputs with training data |
That fourth row deserves emphasis. A first model that scores 99% on a problem experts find hard has almost certainly found a leak. The correct response is to go looking for it, not to present the number.
In three sentences
Split the data before anything else, use the validation set for decisions and look at the test set exactly once, because every decision made on a set makes its score optimistic. Cross-validation gives both a more reliable average and a measure of stability, and time-ordered data must be split chronologically or the model sees the future. Always compare against a trivial baseline and choose the metric by asking whether a false alarm or a miss costs more, because a model that cannot beat "predict the majority class" has demonstrated nothing.