Module 1 — Training-serving skew and its consequences
The previous MLOps course insisted that a deployed model degrades silently when the world drifts from the training distribution. This course begins with a subtler cause of the same failure — one where the world has not moved at all but the model still becomes wrong: training and serving compute the same feature two different ways.
The same feature, two implementations
Take a single feature from a card-fraud scoring model: n_tx_last_1h, the number of transactions the same card has authorized in the last hour.
At training time, a data scientist opens a Jupyter notebook and writes something like this on a Parquet file of historical transactions:
import pandas as pd
df = pd.read_parquet("transactions.parquet")
df = df.sort_values(["card_id", "ts"])
df["n_tx_last_1h"] = (
df.groupby("card_id")
.rolling("1h", on="ts")["amount"]
.count()
.reset_index(level=0, drop=True)
)
Rolling window of one hour, grouped by card, counted on the amount column so NaN amounts still count as a transaction. Clean, readable, correct on the training set.
Weeks later a backend engineer implements the same feature inside the scoring service. The team has no repository for features; the engineer reads the notebook, understands "count the transactions of the same card in the last hour", and writes a Redis lookup:
import time, redis
r = redis.Redis()
def n_tx_last_1h(card_id: str) -> int:
now = time.time()
key = f"tx:{card_id}"
r.zremrangebyscore(key, 0, now - 3600)
return r.zcard(key)
The two look identical from a distance. They are not.
Where the two implementations already diverge
At least five gaps are already baked into the two snippets above, and they will show up as wrong predictions in production.
Inclusivity of the window. The pandas rolling window is closed on the right by default: a transaction whose ts equals now is counted. The Redis version uses now - 3600 as strict lower bound, so it excludes transactions from exactly one hour ago that the training pipeline included. On a batch of 10 000 predictions per day, that shifts the counter by one for perhaps 300 borderline events.
Timezone. The Parquet file was written with timestamps in UTC. The scoring service uses time.time(), also UTC, but the notebook lived on a laptop set to Europe/Paris and the pandas to_datetime inferred local time. The training feature is systematically shifted by two hours in summer.
Definition of a transaction. In training, every row of the Parquet counts. In production, Redis only stores authorized transactions — declined ones never reach the sorted set. The training feature averages 4.1 transactions per card per hour; the production feature averages 3.7. Not a bug: a definitional gap.
Handling of the current transaction. The training pipeline includes the transaction being scored in the count, because it is a row of the batch. The production service is called before the current transaction is written to Redis, so it excludes it. The counter is systematically off by one on active cards.
Precision. Redis stores integer timestamps. Pandas kept microseconds. On rare cards with two transactions in the same second, the two pipelines see different counts.
None of these gaps raise an error. All of them silently produce different values for the same feature name, on the same card, at the same moment.
A worked example: fraud recall drops from 92 % to 74 %
We ran this exact scenario on an anonymized transaction stream. The offline model, trained on features computed by the notebook, reached a recall of 92 % on the fraud class at a fixed 1 % false-positive rate. The same model, evaluated on features recomputed by the production service on the same transactions, dropped to 74 % recall.
The model had not moved a comma. The world had not moved either — we replayed the same transactions. The features had moved. Because the model had learned to lean on n_tx_last_1h around thresholds that no longer meant the same thing at serving time, its decision boundary was, in effect, being crossed at the wrong places.
Read as a business number, the gap is expensive: at a 1 % false-positive budget, we were missing roughly one fraud in four we would have caught if the training-time feature were what production also computed.
What a feature store promises
A feature store is not a database. It is the operational answer to that gap: a single implementation of each feature, called by training and serving through the same code path. Concretely, the same Python function computes n_tx_last_1h for the historical training set and for the live decision at the payment terminal, and the store keeps a fresh online copy ready to answer in a few milliseconds.
That single-source property is what the rest of the course is about. Module 2 opens the store to name its parts. Modules 3 to 6 explain the mechanisms that make it work — offline versus online serving, feature definitions, point-in-time joins, materialization. Module 7 shows the same setup with Feast on Parquet and Redis. Modules 8 to 10 return to organizational and operational questions.
The 18-point recall drop above did not come from any single mistake. It came from five 1 % gaps stacked in the same direction. That is the shape of training-serving skew: no offender is dramatic enough to trigger a review, and the sum quietly ruins a model that passed every offline test.
Summary
- The same feature is often implemented twice — once in a notebook for training, once in the serving service — and the two implementations diverge on window inclusivity, timezone, definition of the event, current-transaction handling and precision.
- The gaps are silent: they raise no error, they just produce different values for the same feature name on the same input.
- The visible symptom is a model whose metrics degrade at deployment while the world has not moved and the code has not changed.
- A feature store solves the class of problem by enforcing a single implementation of each feature, shared between training and serving.
Next module: the anatomy of a feature store, and where its pieces plug into the MLOps architecture you already know.