Lesson 4 — Monitoring
A web service that breaks returns errors. A model that breaks returns confident predictions that are quietly worse. Nothing in the logs, nothing in the alerts, no complaints for months.
Monitoring is what converts that silent failure into a signal.
Four layers, in order of how quickly they tell you something
Layer 1 — Operational health
The ordinary service concerns: is it up, how fast, error rate, resource use. Necessary and insufficient, since a perfectly healthy service can be serving predictions that stopped being useful in March.
Layer 2 — Data quality
Checks on the inputs arriving, and this is where most real incidents are actually caught:
- Missing value rates per field, compared with training
- Values outside expected ranges
- New categories the model has never seen
- Schema changes: a renamed field, a changed type, a unit switched from euros to cents
- Volume anomalies: far more or far fewer requests than usual
Most production surprises are upstream data problems rather than model problems. A team changing a form, a supplier altering an export format, a pipeline silently dropping records. Data quality monitoring catches these within hours, and it is the cheapest monitoring to implement.
Layer 3 — Drift
The distribution of inputs or outputs shifting away from training. Two kinds, and the distinction determines the remedy.
Data drift — the inputs change. New customer segment, different geography, a replaced camera, changed marketing driving a different visitor mix. The relationship between inputs and outcome still holds; the model is simply operating outside where it was fitted.
Concept drift — the relationship itself changes. Yesterday's indicators of fraud are today's normal behaviour because fraudsters adapted. What predicted churn before a price change no longer does. Here the model is wrong about the world, not merely unfamiliar with it.
| Data drift | Concept drift | |
|---|---|---|
| What changed | The inputs | The input-to-outcome relationship |
| Detectable from | Inputs alone | Only with outcomes |
| Typical remedy | Retrain on recent data | Rethink features, possibly the whole problem |
| Example | New country's customers | Fraud tactics adapting to your controls |
Data drift is detectable immediately, because you have the inputs. Concept drift is only visible once you know what actually happened, which brings us to the hard part.
Layer 4 — Prediction quality
The thing you actually care about, and usually the slowest to learn.
Prediction distribution is available immediately. If a model that used to flag two percent of transactions now flags eight, something changed even before you know which. This is a genuinely useful early signal and costs almost nothing to track.
Actual accuracy requires ground truth, and ground truth arrives late or never.
The delayed ground truth problem
This shapes monitoring design more than anything else.
| Prediction | When you learn the truth |
|---|---|
| Fraud on a transaction | Days to months, when disputed |
| Customer will churn | After the churn window, months |
| Loan default | Years |
| Which article to recommend | Immediately, from the click |
| Defect in a part | When it fails in the field, possibly years |
| Medical diagnosis | Sometimes never, definitively |
With a months-long delay, accuracy monitoring tells you about a model that stopped working a quarter ago. So you rely on proxies that are available now:
- Input drift, which precedes accuracy loss
- Prediction distribution shifts
- Human override rates, where a person reviews the output — an excellent signal, and free
- Downstream business metrics: conversion, complaint volume, manual rework
- Agreement with a simple baseline model, where divergence is worth investigating
And you invest in getting labels faster: sampling a small share of predictions for immediate human review is often the single most valuable monitoring investment available, because it converts a months-long feedback loop into a daily one.
Feedback loops, which are a real trap
When a model's predictions influence the data it later trains on, it can reinforce its own mistakes.
A fraud model flags certain transactions for review. Only flagged transactions get investigated, so only they generate labels. The model trains on those labels and becomes more confident about the pattern it already believed, while fraud of a kind it never flagged stays permanently invisible.
The same shape appears in recommendation (users can only click what was shown), in hiring (only interviewed candidates produce outcomes), and in predictive policing (patrols generate the arrests that justify the patrols).
The remedy is deliberate exploration: review a random sample outside the model's recommendations, accept some cost for the information, and track outcomes for that sample separately. Skipping this produces a model that appears to improve while its blind spot grows.
When to retrain
Four defensible triggers:
Monitoring says so. Drift crosses a threshold, or measured accuracy drops. The most principled trigger and it requires the monitoring to exist.
New data materially helps. Enough new labelled examples have accumulated that the training set is meaningfully better, particularly if they cover cases the model handled poorly.
The world changed. A new product, a new market, a process change, a regulatory change. Do not wait for monitoring to confirm what you already know.
Scheduled, as a safety net. Monthly or quarterly retraining catches slow degradation nobody noticed. Reasonable as a backstop and a poor substitute for monitoring, since a schedule cannot tell you whether the retrain was needed or whether it helped.
Always validate before promoting. A retrained model is not automatically better: new data can be worse, a bug can enter the pipeline, or a distribution change can hurt the retrained version. Evaluate it against the current production model on a held-out set, then shadow it.
An automated retraining pipeline, and when to build one
Fully automatic retraining and deployment sounds appealing and belongs behind two conditions: reliable, timely ground truth, and automated validation you genuinely trust. Without both, you have automated the deployment of an unvalidated model.
A safer default that most teams should use: automate the training and the evaluation, produce a comparison report, and require a human to approve promotion. You get most of the speed and keep the judgement where it belongs.
Log every prediction with its inputs, the model version and a timestamp. Then add two checks: missing-value rates per field, and the prediction distribution. That combination catches a surprising majority of real incidents, and it takes a day. Everything else is refinement.
In three sentences
Model monitoring has four layers — service health, input data quality, drift, and prediction quality — and the second catches most real incidents because production surprises are usually upstream data changes rather than model problems. Data drift means the inputs moved and is detectable immediately, while concept drift means the input-to-outcome relationship changed and is only visible once outcomes arrive, which is the central difficulty: ground truth is often delayed by months or years, so you monitor proxies such as input drift, prediction distribution and human override rates, and sampling predictions for immediate review is frequently the highest-value investment available. Retrain when monitoring, new data or a known change in the world justifies it rather than by calendar alone, always validate a retrained model against the one in production before promoting it, and watch for feedback loops where a model's own predictions shape the labels it later learns from.