Lesson 5 — Why models fail in production
A model that passed every offline test can still be useless once deployed. There are five recurring causes, they account for the large majority of failures, and every one of them is invisible in a notebook.
1. Leakage inflated the offline score
The model never worked. The evaluation was wrong.
We have met this twice already, from the Python and features angles. The reason it comes back here is that leakage is the first hypothesis whenever a model performs far worse in production than in testing, and checking it is quick.
The audit is short:
- Was anything scaled, imputed or encoded before the train-test split?
- Does any feature depend on information recorded after the moment of prediction?
- Was time-ordered data split randomly?
- Are there duplicate rows appearing in both train and test?
- Was the test set used to choose between models?
Any yes explains the gap, and no amount of retraining fixes it until the evaluation is rebuilt.
A large gap between offline and online performance is much more often leakage than deployment. Investigating the pipeline before investigating the infrastructure saves days.
2. The training data did not resemble reality
The model works exactly as designed, on a population that is not the one it now serves.
This takes several forms, and all of them come from how the data was collected rather than from any modelling error:
Selection bias. A credit model trained only on approved applications never observed how rejected applicants would have behaved. Its view of risk is systematically incomplete, and it cannot know that.
Temporal bias. A demand model trained on 2019 data forecast 2020 catastrophically. Nothing in its experience resembled what happened, and a model interpolates within its experience rather than extrapolating beyond it.
Geographic and demographic bias. A vision model trained on one region's imagery degrades elsewhere. A speech model trained on one set of accents fails on others. The failures land unevenly, on whoever was under-represented.
Survivorship bias. Analysing only customers who stayed tells you nothing about why people left.
More data does not fix any of these. It makes the model more confidently narrow, which is worse, because confidence is what people act on.
3. The training-serving gap
The model is fine and the pipeline around it is not.
During development, features were computed in a notebook with pandas over a historical table. In production they are computed by a different service, in a different language, from a live stream. The two implementations differ subtly, so the model receives inputs unlike anything it was trained on.
Where this bites in practice:
- A unit difference: seconds versus milliseconds, or a currency conversion applied in one path and not the other
- Missing values filled differently: median in training, zero in production
- A category encoding whose order differs, so the model reads a different meaning from the same value
- Time zones, which quietly shift every date-derived feature
- An aggregation window defined as "last 30 days" in training and "current month" in production
The structural fix is to compute features once, in shared code used by both training and serving — which is exactly what a feature store exists to provide, and part of why MLOps is a discipline rather than a deployment step.
4. Drift: the world moved
The model is frozen and reality is not. This is the failure that arrives slowly and is therefore noticed last.
Two kinds, worth distinguishing because they need different responses:
Data drift — the inputs change shape. Your customers get younger, a new product launches, a marketing campaign brings a different audience. The relationships the model learned may still hold; it is simply seeing inputs it has little experience of.
Concept drift — the relationship itself changes. What predicted fraud last year does not this year, because fraudsters adapted. What predicted churn changed when a competitor cut prices. Here the model's learned mapping is genuinely wrong, and retraining on recent data is the only remedy.
The dangerous property of drift is silence. A drifted model does not raise an exception. It returns confident predictions that are steadily less correct, and unless something is watching the input distribution and the outcomes, the first signal is a business metric moving for reasons nobody can explain.
5. Feedback loops: the model changed the world
The subtlest failure, and the one with the worst consequences, because the model creates the data that trains its successor.
Predictive policing is the canonical example. Send more patrols where the model predicts crime, and more crime is recorded there simply because more officers are present. That record trains the next model, which sends yet more patrols. The loop is self-confirming and looks like accuracy.
Recommendation does the same, more benignly. Show a user what the model expects them to like, and they mostly interact with what was shown. The next model learns from those interactions and narrows further. Preferences that were never surfaced never get discovered.
Credit completes the pattern: reject an applicant and you never learn whether they would have repaid. The model's own decisions determine what evidence exists, so its blind spots are permanent by construction.
A model deployed into a system it influences does not observe reality. It observes the consequences of its own past decisions. Detecting this requires deliberately holding out a fraction of traffic from the model's influence, which costs something and is the only way to keep a source of unbiased evidence.
The pre-deployment checklist
| Check | Why |
|---|---|
| Beats a trivial baseline by a meaningful margin | otherwise there is nothing to deploy |
| No feature depends on post-event information | the top cause of inflated scores |
| Time-ordered data split chronologically | random splits let the model see the future |
| Features computed by shared code in both paths | closes the training-serving gap |
| Performance measured on recent data, not just a random sample | reveals drift already under way |
| Performance checked per subgroup, not only overall | an average hides who it fails |
| Input distributions and outcomes monitored after launch | drift is silent |
| A retraining trigger defined in advance | otherwise it happens when someone complains |
| Some traffic held out from the model's influence | preserves unbiased evidence |
In three sentences
Most production failures are not modelling failures: leakage inflated the offline score, the training data did not resemble the population being served, or features are computed differently in production than in training. Once deployed, a frozen model decays silently as the world drifts, returning confident predictions that quietly stop matching reality unless something monitors inputs and outcomes. The worst case is a feedback loop, where the model shapes the data that trains its successor, so it observes the consequences of its own decisions rather than the world.
Next — Lesson 6: recap and FAQ →