Module 8 — Data leakage: the most frequent cases
The supervised learning course introduced leakage from the validation standpoint: how to measure without fooling yourself. This module treats it from the feature construction standpoint: how to build, unintentionally, a feature that already knows the answer. This is the major professional risk of feature engineering, because leakage causes no error — it produces an excellent score and an unusable model.
Definition, and why metrics cannot detect it
There is leakage as soon as a feature contains information unavailable at the actual moment of prediction. The consequence is perverse: the stronger the leakage, the better the score. Cross-validation, a sealed test set, all the safeguards of course 04 remain blind, because the leakage is present in the data itself, hence in the test set too.
No metric will save you. Only reasoning about the nature of the features will.
Family 1 — The post-outcome feature
The most frequent. A feature whose value only exists after the predicted event, or which is a consequence of it.
| Prediction goal | Leaking feature | Why |
|---|---|---|
| will a patient be hospitalized | length of stay | only exists after hospitalization |
| will a customer churn | account closing date | that is churn itself |
| is a transaction fraudulent | amount refunded | consequence of the fraud finding |
| will a prospect sign | contract number | created by the signature |
The test to apply to every feature is always the same, and it should be asked out loud: "at the precise moment I must produce this prediction, is this information already available in my systems?" If the answer is no, discard the feature, whatever its predictive power — that power is exactly what should worry you.
Family 2 — Preprocessing computed on the whole dataset
The most discreet, and it runs through every preceding module. Computing a statistic over all the data before splitting injects test information into training:
- mean, standard deviation, minimum or maximum of a scaling step (module 3);
- median or mode of an imputation (module 2);
- per-category means of a target encoding (module 4);
- vocabulary and IDF weights of a text vectorization (module 6).
The effect is weaker than that of a post-outcome feature, but quite real: the reported score is systematically optimistic. The remedy is structural and lives in module 10.
Family 3 — Temporal leakage
Specific to time-ordered data, and already met in module 7: a rolling window including the current row, a rolling without shift(1), or an aggregation computed over the entire period when the model will predict as data arrives.
To which the split itself adds: a random split on temporal data means learning the future to validate on the past. TimeSeriesSplit is what you need, as seen in course 04.
Family 4 — The identifier that encodes information
A subtle and frequent case. An identifier assumed to be neutral sometimes carries unintended information: case numbers assigned sequentially, where cases of a certain type were entered in the same range; a patient identifier whose prefix indicates the admitting department. The model then learns the identifier assignment logic, not the phenomenon.
The reflex: do not feed raw identifiers to the model, and be wary of any technical feature that is inexplicably predictive.
Investigating: the warning signals
Three symptoms should trigger an immediate check:
- performance that is too good: 98% where the state of the art caps at 80%. Joy is the wrong reflex; investigation is the right one;
- a feature that massively dominates feature importance. Examine it: is it a consequence of the target?
- an abrupt jump in performance after adding a feature. What does that feature know that it should not know at this instant?
A simple and effective method: train a model with the suspicious feature alone. If it reaches a near-perfect score by itself, there is no longer any doubt.
The decisive check, however, remains human: have the business describe how and when each piece of data is produced. Most leaks become visible by listening to the description of the collection process, not by staring at a correlation matrix.
Leakage is not paid for with an error but with a decision. The project is approved, presented, put into production on the strength of an exceptional score — and the model collapses on contact with reality. The damage is not technical, it is one of credibility: what is hit is trust in the team and in the approach, often lastingly. Hence the value of checking before announcing, never after.
Summary
- Leakage is information unavailable at prediction time; it improves the score, which makes every metric blind.
- Four families: post-outcome feature, preprocessing on the whole dataset, temporal leakage, informative identifier.
- The test to apply to every feature: is this information already available at the instant of prediction?
- Warning signals: a score too good, a dominating feature, an abrupt performance jump; and the best check remains the business description of the collection process.
Next module: feature selection, to shrink a set that has grown too wide without losing signal.