Lesson 6 — Recap and frequent questions
The five lessons in five sentences
- Projects fail because no decision was going to change, because data is unavailable at prediction time, because features are computed differently in training and serving, because nobody owns the system afterwards, or because it degraded silently — almost never because the model was insufficiently accurate.
- Reproducing a model needs five things versioned — code, data, parameters, environment and seeds — with experiment tracking to keep the reasoning and a registry to give you rollback and an audit trail.
- Batch prediction suits more cases than teams assume, infrastructure should start at the simplest level that meets the requirement, and releases should be progressive because a model's behaviour is statistical rather than testable.
- Monitor input data quality first because most incidents are upstream data changes, expect ground truth to arrive late, and rely on drift and prediction-distribution proxies in the meantime.
- Assign a model owner and a system owner from the start, recognise that labelling and human review cost more than compute, and tier governance to the consequence of being wrong.
The full lifecycle
A pre-launch checklist
| Check | Why |
|---|---|
| The decision this changes is named, with an owner | Prevents the most common failure outright |
| Every feature is retrievable at prediction time | Prevents temporal leakage |
| Feature code is shared between training and serving | Prevents training-serving skew |
| Training data is stored with an identifier | Makes the result reproducible |
| Library versions pinned, seeds set | Same |
| Evaluation split by time or entity, one untouched set | Prevents optimistic scores |
| Metrics reported per relevant subgroup | Surfaces disparities an average hides |
| Input validation rejects malformed values | Prevents confident predictions on nonsense |
| Predictions logged with inputs and model version | Makes investigation possible later |
| A fallback exists if the model is unavailable | Prevents an outage becoming a business outage |
| Rollback tested, not just documented | Because you will need it under pressure |
| Monitoring and thresholds agreed before launch | Converts silent failure into a signal |
| An owner named for a year from now | Prevents the handover cliff |
Twelve frequent questions
1. Do I need Kubernetes?
Probably not. A scheduled script writing predictions to a table, or a single containerised service, covers a large share of production use cases. Adopt orchestration when scale, model count or reliability requirements genuinely demand it, not at the start.
2. Which MLOps tools should I use?
Start with what you already have: Git, a container runtime, a scheduler, object storage. Add an experiment tracker early because it costs a few lines. Add a feature store or a full platform when you have multiple models sharing features or multiple teams needing self-service.
3. How often should I retrain?
When monitoring, materially better data, or a known change in the world justifies it. A monthly or quarterly schedule is a reasonable backstop and a poor primary strategy, because a calendar cannot tell you whether the retrain was needed or whether it helped.
4. How do I know the model is still working if labels take months?
Monitor proxies available now: input drift, prediction distribution, human override rates, and downstream business metrics. Then invest in getting labels faster — sampling a small share of predictions for immediate expert review turns a months-long feedback loop into a daily one and is usually the best monitoring money you can spend.
5. What is the difference between data drift and concept drift?
Data drift means the inputs moved while the input-to-outcome relationship still holds, so retraining on recent data usually fixes it. Concept drift means the relationship itself changed, and retraining may not be enough — the features or the problem framing may need rethinking.
6. Should retraining and deployment be fully automatic?
Only with reliable, timely ground truth and automated validation you genuinely trust. Otherwise you have automated the deployment of an unvalidated model. Automate training and evaluation, produce a comparison report, and require human approval to promote.
7. Do I need a feature store?
If you have one model and features computed in one place, no — a shared feature-computation library gets you most of the protection. Feature stores earn their complexity when several models share features, when training-serving skew has already bitten you, or when you need point-in-time correctness you would otherwise implement yourself and get wrong.
8. How do I explain a model's decision to a customer or regulator?
Use methods that attribute a specific prediction to specific input contributions, and record them alongside the prediction. For high-stakes decisions, prefer a model whose reasoning is inspectable from the start rather than trying to explain a complex one afterwards — the accuracy difference is often smaller than expected and the auditability difference is large.
9. What do I do about a feedback loop?
Deliberately sample outside the model's recommendations: review a random share of cases the model did not flag, accept the cost, and track their outcomes separately. Without that, the model reinforces what it already believes and its blind spot grows invisibly.
10. Where does the money actually go?
Usually labelling, data infrastructure, human review of outputs and investigation time — all of which typically exceed training compute. Real-time serving is the compute line that surprises people, because you provision for peak and pay while idle.
11. Our model works but nobody uses it. What went wrong?
Almost certainly failure one from lesson 1: no decision was going to change, or the people who would act on the output were not involved in defining it. This is not fixable with a better model, and it is fixable by working backwards from the decision with the people who make it.
12. Where should I start if I am at stage zero?
Get one model deployed, reproducible and logged. Pin versions, set seeds, store the training data with an identifier, deploy the simplest way that works, and log every prediction with its inputs and the model version. That single loop teaches you more than any amount of platform evaluation. The premium catalogue covers MLflow, Docker, CI/CD for models, feature stores and monitoring hands-on.
Vocabulary you can now use precisely
| Term | Meaning in one line |
|---|---|
| MLOps | The practices that get models into production and keep them working |
| Training-serving skew | Features computed differently in training and in serving |
| Temporal leakage | Using information that would not exist at prediction time |
| Experiment tracking | Recording parameters, metrics and artefacts for every training run |
| Model registry | A versioned store of models with stages and an audit trail |
| Feature store | One definition per feature, served to both training and serving |
| Point-in-time correctness | Retrieving historical features as they were at that moment |
| Batch prediction | Scoring many records on a schedule and storing the results |
| Real-time serving | A service returning a prediction per request in milliseconds |
| Shadow deployment | Running a new model on live traffic without using its output |
| Canary release | Routing a small share of traffic to a new version |
| Data drift | The input distribution moving away from training |
| Concept drift | The input-to-outcome relationship itself changing |
| Ground truth | The actual outcome, needed to measure accuracy |
| Feedback loop | Predictions influencing the data the model later learns from |
| Model card | Documentation of purpose, data, performance and limitations |
And now?
You can look at a machine learning project and see where it will fail, which is a more useful skill than being able to improve its accuracy by two points.
Validate it: take the 5-question quiz →
Continue free: Cloud AI covers the managed platforms that implement much of this.
Go deeper: the premium catalogue covers MLflow, Docker, CI/CD for models, feature stores and production monitoring hands-on, with a verifiable certificate after a 40-question examination.