Module 8 — Cross-validation, time splits and data leakage

Seven modules of models; time for the discipline that makes their scores trustworthy. A model is worth nothing without an honest measurement — and dishonest measurements are frighteningly easy to produce. This module locks in the evaluation method: cross-validation, splits that respect time, and the hunt for the number-one enemy, data leakage.

Why one validation split is not enough

Module 1 set aside a sealed test set. To tune models along the way, you could carve a single validation set out of the training data — but with medium-sized data, that single carve-out is luck-sensitive: an "easy" validation set flatters the model, a "hard" one buries it. The estimate has high variance, and comparing two models on it becomes shaky.

k-fold cross-validation: measuring several times

k-fold cross-validation stabilizes the measurement by rotating roles. Split the training data into $k$ blocks (folds), typically 5. Then, $k$ times over: train on $k-1$ folds, validate on the remaining one. Every point is used for validation exactly once.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="roc_auc")
scores.mean(), scores.std()   # performance AND its stability

The gain is twofold: a more reliable mean of performance, and the standard deviation across folds, which quantifies the estimate's stability. Two models with close means but very different standard deviations aren't equivalent — the more stable one wins. In classification, prefer StratifiedKFold (the default in scikit-learn), which keeps class proportions in each fold — the module 1 reflex, generalized.

Time data: never validate on the past

Standard k-fold randomly shuffles rows. On temporal data (sales, prices, logs), that's a serious error: the model would train on the future to be validated on the past — a performance impossible to reproduce in production. The rule: always train on the past, validate on the future, as reality will do. TimeSeriesSplit implements this expanding scheme: train on months 1–6, validate 7–8; train 1–8, validate 9–10; and so on.

Data leakage: the enemy that inflates every score

Leakage is any information from the validation or test set (or the future) seeping into training. The score becomes stellar; the model, in production, collapses. Classic forms:

• Preprocessing computed on all the data: standardizing before splitting injects the test's mean and standard deviation into training — subtle and very common;
• Post-target features (module 1): information unavailable at prediction time;
• Duplicates straddling train and test;
• Temporal leakage: shuffling time data, or features computed over a window that includes the future.

The pipeline, structural weapon against leakage

Rather than policing yourself by hand, make leakage structurally impossible: chain preprocessing and model in a Pipeline. Cross-validation then re-fits preprocessing within each fold, on that fold's training data only.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X_train, y_train, cv=5)   # scaler fitted per fold: no leakage

A too-good score is an alarm, not a joy

95% where the state of the art gets 80%? Look for the leak before celebrating. Sudden performance jumps after adding a feature deserve the same reflex: what does this feature know that it shouldn't yet know at prediction time?

Summary

• One single validation split is luck-sensitive; k-fold cross-validation (stratified in classification) gives a mean and a standard deviation — reliability plus stability.
• Temporal data demands time-respecting splits (TimeSeriesSplit): train on the past, validate on the future.
• Data leakage — preprocessing on all the data, post-target features, straddling duplicates — silently inflates scores.
• The pipeline re-fits preprocessing inside each fold: leakage becomes structurally impossible.

Next module: the metrics themselves — accuracy, precision, recall, F1, ROC AUC — and choosing the one that matches the business problem.