Lesson 1 — Why models never ship
Estimates of how many machine learning projects reach production vary and are consistently unflattering — most surveys land somewhere between a third and a half, with the pessimistic ones considerably lower. The number matters less than the pattern behind it.
Almost none of these projects failed because the model was not accurate enough. They failed for reasons that had nothing to do with modelling, and every one of them is identifiable in advance.
Failure 1 — No decision was going to change
The most common and least discussed. A model predicts churn with good accuracy. Nobody had agreed what happens when it flags a customer. There is no retention offer, no team to call them, no budget. The prediction arrives and nothing happens.
The check: name the decision, the person or system that makes it, and what they will do differently. If you cannot, you have a reporting request rather than a machine learning project, and building it will produce a dashboard nobody opens.
Failure 2 — The data is not there at prediction time
A model performs beautifully in testing, using a feature that only exists after the event you are predicting.
The textbook version is a hospital readmission model using discharge summaries — written after readmission is known. Real ones are subtler: a field populated overnight by a batch job when the model needs to run at nine in the morning, a customer attribute completed only for existing customers when you are scoring prospects, a value updated retroactively so historical records look nothing like live ones.
This is temporal leakage, and it inflates evaluation scores in a way that survives every standard check, because the leaked information is genuinely in your training data.
The check: for every feature, ask what value it would have held at the exact moment the prediction must be made, and whether that value is retrievable then.
Failure 3 — Training-serving skew
The model trained on features computed one way, and production computes them slightly differently.
The differences are always small and always consequential. Missing values filled with a mean in training and with zero in production. A date parsed in a different timezone. A text field lowercased in one path and not the other. A currency conversion using a different rate source. A pandas transformation reimplemented in Java for the serving layer, with one rounding difference.
The model does not error. It receives inputs subtly unlike anything it saw and returns confidently wrong predictions.
The check: compute features with the same code in both paths, or verify equivalence with tests that compare outputs on identical inputs. This is what feature stores exist to enforce, and it is why they exist at all.
Failure 4 — Nobody owns it after the pilot
The data scientist who built it moved to another project. There is a notebook, no documentation, and a model file whose training data nobody can locate. Six months later performance has degraded and no one can retrain it because the preprocessing steps live in someone's memory.
The check: before the pilot, name who operates this in a year and what they will need. If the answer is "we'll figure that out", the model has an expiry date.
Failure 5 — The integration was the actual project
The modelling took three weeks. Connecting to the legacy order system, getting security approval, agreeing an interface with a team that has its own roadmap, handling the case where the service is down, and training the people who use the output takes nine months.
This is not a failure of machine learning. It is a failure of scoping, and it is extremely common because the modelling is the visible and interesting part.
The check: estimate the integration work explicitly, with the teams who own the systems involved, before promising a date.
Failure 6 — The evaluation was optimistic
Covered in the Machine Learning and Computer Vision courses, and it belongs here too because it produces production failures rather than merely bad papers: random splits on correlated records, tuning on the test set, a metric that ignores class imbalance, or a test set drawn from the same narrow source as training.
The check: split by time or by entity, keep one untouched set, and report the metric that reflects the actual cost of each error type.
Failure 7 — It worked, then quietly stopped
Nothing in the code changed. A supplier changed their packaging, a form added a field, a camera was replaced, customer behaviour shifted after a price change. The model still runs, still returns confident predictions, and is now wrong more often than it used to be.
Without monitoring, you learn this from a complaint months later.
The check: decide before launch what you will monitor and what threshold triggers investigation. Lesson 4.
What MLOps actually is
Given that list, MLOps is the set of practices that address it: reproducible training, versioned data and models, tested and repeatable deployment, monitoring, retraining, and clear ownership.
The reason it is a distinct discipline rather than a subset of DevOps comes down to one asymmetry:
| Ordinary software | Machine learning system | |
|---|---|---|
| Behaviour changes when | Someone edits code | Someone edits code, or the world changes |
| Correctness | Tests pass or fail | Statistical, and degrades gradually |
| Reproducing a result needs | The code version | Code, data, parameters, environment, random seeds |
| Failure looks like | An error, a stack trace | Confident predictions that are quietly worse |
| Rollback | Redeploy the previous version | Redeploy, and also decide what to do about data already scored |
The row that matters most is the fourth. Software that breaks tells you. A degrading model does not, which is why monitoring is not optional and why "it worked when we launched" is not evidence of anything.
Before any modelling: what decision changes, who makes it, and what will they do differently? A large share of the failures above are prevented by a satisfactory answer, and the projects that cannot answer it are the ones that produce a well-validated model nobody uses.
In three sentences
Machine learning projects overwhelmingly fail for reasons unrelated to model accuracy: no decision was going to change, the data is unavailable at prediction time, features are computed differently in training and serving, nobody owns the system after the pilot, the integration work was never scoped, the evaluation was optimistic, or it worked and then quietly degraded. Each of these is detectable before writing code, and the single most useful question is what decision changes, who makes it, and what they will do differently. MLOps exists as its own discipline because a model's behaviour changes when the world changes rather than when someone edits code, and because its failures look like confident predictions that are quietly worse rather than errors that announce themselves.
Next — Lesson 2: reproducibility →