Skip to main content

Module 9 — Monitoring feature quality

Course 20 taught us to monitor the model. That is the last line of defense. Long before a prediction drifts, the features that feed it are already misbehaving — missing more often, distributed differently, stale by minutes. Monitoring the store catches problems where they start, not where they hurt.

Three metrics, not one​

"Feature quality" is not a single number. A store worth monitoring exposes at least three metrics per feature, and each of them alerts on a different failure mode.

Missingness. The share of served values that are null. A stable feature with 2 % null that suddenly jumps to 15 % is not a data-science problem; it is a broken upstream join or a source that stopped ingesting for a subset of entities. This is the fastest signal, because missingness moves in seconds and no statistical test is needed.

Distribution. Basic summary statistics per feature: for numerical features, min / max / mean / percentiles; for categorical features, the top-kk categories and their frequencies. Compared against a reference — typically the training set — a drift shows up as a shift in mean, a widening tail, or a category that used to be rare taking 30 % of the mass.

Freshness. The metric from module 6, monitored per feature view. Freshness is the earliest indicator of a broken pipeline: it moves the instant materialization stops, hours before missingness or distribution notice, because Redis still returns the last known value.

Missing values, distribution, and freshness together cover the three failure modes: upstream ingest, upstream computation, and upstream schedule.

Where to compute these metrics​

Two locations, both useful, both easily confused.

On the offline side, per materialization run. After each materialize, aggregate over the newly written batch: missingness, distribution, count. This is cheap because you are already scanning the data. It catches problems where the store was fed bad numbers.

On the online side, at request time. Every get_online_features call is an opportunity to log the returned values and the freshness. Aggregated by feature and by minute, these logs give you the actually served distribution — which can differ from the offline one if the online store is missing entities the batch had.

The pattern in practice is a lightweight middleware around the online read:

from feast import FeatureStore
import time, logging

fs = FeatureStore(repo_path=".")
log = logging.getLogger("feature_serving")

def serve(features, entity_rows):
    t0 = time.perf_counter()
    resp = fs.get_online_features(features=features, entity_rows=entity_rows).to_dict()
    latency_ms = (time.perf_counter() - t0) * 1000
    for name, values in resp.items():
        n_null = sum(v is None for v in values)
        log.info("feature=%s n=%d n_null=%d latency_ms=%.1f",
                 name, len(values), n_null, latency_ms)
    return resp

Ship those logs to your metrics backend (Prometheus, Datadog, CloudWatch) and you have per-feature dashboards without changing the model.

Alerts that page a human vs alerts that inform​

Not every alert deserves a page at 3 a.m., and mixing the two is how teams stop reading their alerts entirely.

Page. Freshness p99 above SLO for more than 10 minutes. Missingness above a fixed threshold (say 5 %) on a feature marked as critical. Materialization job failure. These stop the model from getting good input; someone must act.

Inform. A moderate distribution shift on a low-critical feature. Missingness moving from 1 % to 2 %. These belong to a weekly review, not to a pager, because the response is a decision, not an action.

The critical-versus-not distinction lives in the registry, tagged per feature. A feature the fraud model depends on is critical; a feature only used by an experimental notebook is not. Without that tag, either every drift wakes someone (and no one pays attention), or none do (and drift is discovered from a stakeholder complaint).

Course 20 introduced drift as the reason models degrade silently. Feature monitoring is where you catch drift before it degrades a model, if you know how to read the signals.

Drift shows up in features first. A shift in the distribution of n_tx_last_1h — say the mean rising from 4 to 6 because a new merchant category is generating more small transactions — precedes any drift in the model's calibration by however long the model tolerates the shift. If the store's dashboard flags the feature shift on Monday, you can retrain and redeploy before the model's precision-recall curve moves visibly on Friday.

Two anti-patterns are worth naming.

Alerting on the model instead of the features. Course 20 taught prediction drift too, and it is a valid signal. But by the time the model's output distribution has shifted, the harm is already done. Feature monitoring is a leading indicator; model monitoring is a lagging one. Both are useful; the leading one is where you find time to react.

Recomputing the reference every day. If the reference distribution used for drift detection is "yesterday", then any slow drift is invisible — every day looks like yesterday. The reference is the training set of the currently deployed model, or a fixed window at deployment time. Rolling it defeats the whole exercise.

What Feast, Evidently and the vendors give you​

Feast itself does not do monitoring — it is a serving layer. The idiomatic setup pairs Feast with a monitoring tool that reads the same source and the same online snapshots.

Evidently is the common open-source pick: it computes a JSON report per feature (missingness, distribution comparison, drift tests) and ships it to a database Grafana can read. Ten lines of Python turn each materialization into a report.

Tecton and Hopsworks ship dashboards built in. That is convenient and locks you in to their reference — a trade-off worth naming explicitly rather than accepting by default.

Whylabs and Arize are commercial monitoring layers that plug on top of any store; they cost money but they cover the "prediction, features, ground truth" triangle in one product, which teams without an MLOps engineer find easier.

One reference, two windows

Keep one reference distribution per feature — the training set of the currently deployed model — and compare against it in two windows: last 15 minutes for freshness and hard alerts, last 7 days for statistical drift. The 15-minute window catches breakages; the 7-day window catches drift. Comparing 15 minutes against 7 days catches nothing because both are recent.

Summary​

  • Missingness, distribution and freshness are the three orthogonal metrics; each alerts on a different upstream failure mode (ingest, computation, schedule).
  • Measure both on the offline side per materialization run and on the online side per request; the two can disagree, and each disagreement is a bug worth chasing.
  • Page on freshness SLO breaches and critical-feature missingness; inform on distribution shifts and non-critical drift; the critical tag lives in the registry.
  • Feature drift is a leading indicator of model drift; monitoring the store gives you the days-to-weeks lead time you need to retrain before predictions degrade.

Next module: the full project — building a feature store for a scoring use case, measuring the skew before and after, and honestly asking when the store paid for itself.