Module 9 — Monitoring: data and concept drift

The churn model is now trained, packaged, tested, and deployed. From this moment on, its quality is a function of the world, not of your code. Monitoring is the discipline of noticing that the world has changed before the business does. This module distinguishes the two kinds of drift, gives the tests that detect them, and, more importantly, warns about the trap of drift alerts that fire on nothing.

Data drift versus concept drift

Data drift is a change in the distribution of the inputs. In June, 40 % of subscribers had a two-year contract; in September, after a marketing push, 55 % do. The features shift; the relationship between features and target may not.

Concept drift is a change in the relationship between inputs and target. In June, a customer with a two-year contract had a 5 % churn rate; a competitor launches in September and now the same profile churns at 12 %. The inputs may look identical; the answer has changed.

Both matter, and they call for different responses. Data drift is often benign (the model was trained on a broad enough range) or a data pipeline bug (a column started being imputed to zero). Concept drift is almost always a business event and demands retraining. The confusion between the two is the most common alerting mistake in production ML.

Detecting data drift per feature

For each feature, compare a reference distribution (the training set, or a stable recent window) to a current distribution (the last day, the last hour). Two classical tests:

For numerical features, the Kolmogorov–Smirnov two-sample test returns a statistic $D$ that is the maximum vertical gap between the two empirical CDFs, and a $p$-value. A $p$-value below 0.01 says the two distributions almost certainly differ; whether the difference matters for the model is a separate question.

For categorical features and, in practice, most tabular features, the Population Stability Index (PSI) is more useful because it grows with the magnitude of the shift rather than with sample size:

$PSI = \sum_i (p_i - q_i) \cdot \log(p_i / q_i)$

Rules of thumb: PSI < 0.1 no material shift, 0.1 <= PSI < 0.25 monitor, PSI >= 0.25 investigate. The values are not laws of physics but they hold up across contexts.

Detecting concept drift

Concept drift is harder because it requires the target to be observable. For churn, the target arrives with a delay: you learn today whether a subscriber who was scored in July has since canceled. That delay is intrinsic and shapes the monitoring cadence.

Once the target is available, the model's realized metrics — ROC AUC on the last four weeks, calibration on the last month, precision at the operating threshold — are the ground truth. A drop of 0.02 on ROC AUC month over month is a concept-drift signal that no feature-level test can produce.

Without the target (yet), the two useful proxies are prediction distribution drift (the histogram of the model's outputs shifts noticeably) and residual drift on a held-out sample where the target trickles in fast (e.g., customers who cancel within 30 days).

Evidently in practice

Evidently is the open-source library that packages these tests. It builds an HTML report from two data frames, a reference and a current, and can also emit a JSON summary consumed by an alerting stack.

from evidently import Report
from evidently.presets import DataDriftPreset, RegressionPreset

report = Report(metrics=[DataDriftPreset(), RegressionPreset()])
snapshot = report.run(reference_data=ref_df, current_data=cur_df)
snapshot.save_html("reports/drift_2026-09-05.html")

Two ways to use it. Ad-hoc, when someone spots something off, you rerun the report and read the HTML. Scheduled, the report runs every night, the JSON summary is compared to thresholds, and any exceedance opens a ticket.

The misleading alert: why noise is worse than silence

The most common failure of a monitoring system is not that it misses drifts. It is that it fires so often on nothing that the team stops reading it. Three symptoms of this pathology, and their causes:

• The alert fires every Monday morning. Business volume changes on weekends; a distribution comparing "the last 24 hours" to a week-long reference always shows structural differences at cadence boundaries. Fix: align windows to the business cycle, not to the wall clock.
• A feature crossed PSI 0.25 for one hour and the alert never resolved. A single hour is not a distribution; you compared 300 events to 30 000. Fix: minimum sample size before comparing, and re-evaluate at a longer horizon before paging.
• Every feature drift triggers a "model may be broken" ticket. Data drift is not concept drift. Fix: alert only when data drift plus a proxy signal on predictions or residuals both breach thresholds.

The rule: an alert that cannot cause an action is noise. Before wiring one, ask "if this fires at 3 a.m., what will the on-call do?". If the answer is "look at it and go back to sleep", it is a dashboard, not an alert.

What else to monitor: the application layer

Data and concept drift are the ML-specific part. The rest is regular observability: latency (p50, p95, p99 of /predict), error rate (5xx from the API, timeouts against the feature store), throughput (requests per second), resource use (CPU, memory, GPU if applicable). A model whose HTTP handler crashes 5 % of the time has a service problem that no drift test will surface.

The two layers overlap in one useful way: a spike in errors often precedes drift by hours, because a broken feature-store lookup returns defaults that then flow through the model and shift its output distribution. Monitoring both catches the cause and the effect.

Alert on impact, not on statistics

A PSI of 0.28 on a feature the model treats as almost non-informative is meaningless. Weight your drift indicators by the model's feature importance — a shift on the top-three features is worth ten shifts on the tail. This is the single change that most reduces false-positive alerts.

Summary

• Data drift is a change in inputs; concept drift is a change in the input–output relationship. They call for different responses.
• Use KS for numeric features, PSI for categorical and most tabular; thresholds around 0.1 (watch) and 0.25 (investigate).
• Concept drift requires the target — realized metrics on recent labeled data — plus useful proxies (prediction distribution, fast-labeled residuals).
• The dominant failure mode is noisy alerts; align windows to the business cycle, require sample sizes, and combine data drift with a prediction/residual signal before paging.

Next module: what to do about it — automated retraining, safe promotion, and rollback in one command.