Lesson 2 — Reproducibility
Here is the test. A regulator, a customer or a colleague asks: this prediction was made in March, show me exactly how the model that made it was produced.
If you cannot, you cannot debug it, cannot audit it, cannot safely improve it, and cannot demonstrate it was built responsibly.
The five things that must be versioned
Code alone is not enough, which is the whole reason this is a distinct problem.
1. Code. Training scripts, preprocessing, evaluation. Standard version control, and the easy part.
2. Data. The exact dataset used. This is where most teams stop, because it is genuinely awkward: datasets are large, they live in databases that get updated, and "the customer table" means something different every day.
3. Parameters. Hyperparameters, feature lists, preprocessing choices, the split definition. Configuration files in version control rather than values typed into a notebook.
4. Environment. Library versions, and it matters more than expected — a minor version change in a modelling library can alter results, and a preprocessing library can change a default. Pin versions, and containerise for anything that must survive.
5. Randomness. Weight initialisation, data shuffling, dropout, train-test splits. Set seeds explicitly. Note that full determinism on GPU hardware needs specific settings and costs some speed, which is a reasonable trade for anything auditable.
Miss any one and reproducibility is a coincidence.
Versioning data, practically
Three approaches, and the right one depends on how your data arrives.
Snapshot and store. Copy the dataset used for each training run to durable storage with an immutable identifier. Crude, effective, and cheap for anything up to a few hundred gigabytes.
Content-addressed tooling. Tools in the mould of DVC store a hash of the data in Git alongside the code and keep the bytes in object storage. You get "check out this commit and get exactly that data" without putting large files in Git.
Query-based versioning. For data in a warehouse, record the query plus a timestamp or snapshot identifier, relying on the warehouse's own time travel features. Elegant when available, and it depends on retention policies you may not control.
What to avoid: filenames as version control. training_data_final_v3_fixed.csv is not a versioning strategy, and everyone reading this has seen one.
Experiment tracking
Training runs happen dozens of times a week during development. Without a record you lose the thread within days: which configuration produced the good result, whether that improvement was real or noise, what you already tried.
An experiment tracker records, for every run: parameters, metrics, the code version, a data reference, the environment and the resulting artefacts. Tools such as MLflow and Weights & Biases do this with a few lines added to a training script.
The value is less in the dashboard than in what it prevents: repeating experiments you already ran, and being unable to explain six months later why the production model is configured as it is.
Model registry
Trained models need to be findable, described and staged. A registry holds each model version with:
- The artefact itself
- What produced it: code, data, parameters
- Its evaluation metrics
- A stage: development, staging, production, archived
- Who approved the promotion, and when
Two things this enables that scripts on a laptop do not. Rollback — the previous version is one command away, with its metrics recorded so you know what you are going back to. And an audit trail — for any prediction, you can identify the model version that made it and everything about how it was built.
Feature stores, and what they are actually for
A feature store is frequently oversold. Its central purpose is narrow and important: eliminating training-serving skew by computing each feature once, with one definition, and serving it to both paths.
It typically provides:
- One definition per feature, so "customer lifetime value" means the same thing everywhere
- Two access paths: batch, for training, and low-latency, for serving
- Point-in-time correctness, meaning historical features are retrieved as they were at that moment rather than as they are now — which is what prevents the temporal leakage from lesson 1
- Reuse, so a feature built for one model is available to the next
The point-in-time guarantee is the part that justifies the complexity. Implementing it correctly yourself is harder than it sounds and getting it wrong produces inflated evaluation scores that survive every review.
When you do not need one: a single model, features computed in one place, a small team. The overhead exceeds the benefit, and a shared feature-computation library gets you most of the protection.
Documenting a model
Not bureaucracy — this is what makes a model usable by anyone who did not build it, and increasingly what regulation expects.
A model card covers:
| Section | What it answers |
|---|---|
| Purpose | What decision this supports, and what it must not be used for |
| Training data | Source, period, size, known gaps and biases |
| Performance | Metrics overall and broken down by relevant subgroup |
| Limitations | Where it performs poorly, and known failure modes |
| Inputs and outputs | Exact schema, units, ranges |
| Ownership | Who maintains it, who to contact |
| Review | Under what conditions it should be re-evaluated |
The intended-use section repays the effort most. A model built to prioritise a review queue will eventually be used by someone to make an automatic decision, and the record of what it was validated for is your only defence.
If you do nothing else: pin your library versions, set your seeds, store a copy of the training data with an identifier alongside the model file, and record parameters and metrics in a text file committed with the code. That takes an afternoon and covers most of what teams later wish they had.
In three sentences
Reproducing a model requires versioning five things — code, data, parameters, environment and random seeds — and missing any one makes reproducibility a coincidence rather than a property. Experiment tracking prevents you repeating work and losing the reasoning behind the production configuration, a model registry gives you rollback and an audit trail from any prediction back to how the model was built, and feature stores exist for one narrow but important purpose: computing each feature once so training and serving cannot diverge, with point-in-time correctness preventing the temporal leakage that inflates evaluation scores invisibly. Documenting intended use matters more than it appears, because a model built to prioritise a queue will eventually be used to make an automatic decision, and the record of what it was validated for is the only defence available then.