Module 7 — Evaluation metrics and how to read them honestly
"Our model reaches 97% accuracy." That sentence, as is, says nothing. This module gives you the reflexes to read performance numbers — yours and your vendors' — and to spot, in a few questions, measures that are flattering but hollow.
The accuracy trap
Accuracy is the most intuitive metric: the proportion of correct predictions. It is also the most misleading as soon as classes are imbalanced — which is the case in nearly every interesting problem.
The demonstration takes two lines. In a stream of transactions where 1% are fraudulent, the "model" that always answers "legitimate" reaches 99% accuracy. It detects strictly nothing. Yet on that metric it beats many real models.
Fraud, breakdowns, rare diseases, churn, manufacturing defects: the events worth predicting are almost always the minority. First question in front of any performance number: what is the class distribution?
The confusion matrix: the truth in four cells
For a two-class problem, everything fits in four numbers:
| Predicted: positive | Predicted: negative | |
|---|---|---|
| Actual: positive | True positives (TP) | False negatives (FN) — the missed cases |
| Actual: negative | False positives (FP) — the false alarms | True negatives (TN) |
The two errors — FP and FN — almost never carry the same business cost. Missing a cancer (FN) and calling a healthy patient back for extra tests (FP) do not compare. Blocking an honest customer's transaction and letting a fraud through do not either. Every serious evaluation starts by pricing each of the two errors.
Precision and recall: the two questions that matter
From the matrix follow the two central metrics, each answering a different question.
Precision: among the cases the model flags, what share is actually positive? TP / (TP + FP). A precision of 80% means one alert in five is false. It is the metric of trust in the alerts — critical when every alert triggers a costly action.
Recall: among the actually positive cases, what share does the model catch? TP / (TP + FN). A recall of 75% means one case in four slips through. It is the metric of coverage — critical when missing a case is expensive.
The two pull against each other. A model generally produces a score between 0 and 1, and it is the chosen threshold that turns the score into a decision. Lowering the threshold catches more cases (recall ↑) but multiplies false alarms (precision ↓); raising it does the opposite. The threshold is not a technical detail: it is the numeric translation of your error costs, and it should be decided with the business.
The F1-score — the harmonic mean of precision and recall — summarizes both in one number when a single ranking is needed; it assumes both errors weigh the same, which is rarely true. AUC measures the quality of the ranking produced by the scores, independently of any threshold: useful for comparing models, insufficient for driving a decision.
Facing "97% performance", ask: which metric exactly? On what class distribution? Measured on which data — a watertight test set, or the training data? At what threshold? And what does the trivial baseline achieve (always predicting the majority class, carrying over yesterday's value)? It regularly happens that the trivial baseline does almost as well — in which case the model adds nothing.
Regression metrics
When the output is a number, you measure the size of the gaps.
MAE (mean absolute error): the average gap in real units — "we are off by €12,400 on average". Directly meaningful to the business.
RMSE (root mean squared error): same unit, but large gaps weigh more. Prefer it when a big error is disproportionately costly; the gap between MAE and RMSE also signals the presence of extreme errors.
R²: the share of variation explained by the model. Beware of out-of-context comparisons: an R² of 0.3 can be remarkable on a very noisy phenomenon (human behavior) and mediocre on a regular physical one.
The baseline reflex applies here too: compare against "predict the mean" or "carry over the previous value". In time-series forecasting, yesterday's value is a surprisingly hard baseline to beat.
The metric is not the objective
Last point, the most important one. The metric is a measuring instrument, not the goal. A churn model does not exist for its F1-score but to retain customers; the number that ultimately matters is the impact — customers retained, frauds avoided, hours saved — measured if possible by a controlled test in production.
Blind optimization of a metric produces documented absurdities: the résumé-ranking model that maximizes "relevance" by replicating past hiring biases, the recommender that maximizes clicks by pushing extreme content. What gets measured gets optimized — including when it is the wrong thing. Module 10 returns to this.
Key takeaways
- Accuracy is unreadable on imbalanced classes — that is, nearly all real problems. Always ask for the distribution.
- The confusion matrix separates the two errors, whose business costs almost always differ; precision = trust in alerts, recall = coverage of real cases.
- The threshold arbitrates between the two: it is a priced business decision, not a technical setting.
- Regression: MAE to talk to the business, RMSE when large errors are costly, and always a trivial baseline as the point of comparison.
- The metric serves the objective; it does not replace it.
Next module: the complete life cycle of an AI project — from framing to production and monitoring.