Module 3 — Offline and online store
Module 2 named two stores; this one explains why we cannot live with just one. The offline store and the online store answer two different questions, at two different scales, with two different SLAs, and choosing one technology to do both is how teams end up either paying too much for training or missing the latency budget at serving.
Two questions, two stores
The offline store answers: "what was n_tx_last_1h for card X at 2026-05-14 13:47:03 UTC?" — for millions of (card, timestamp) pairs at once, when a training pipeline rebuilds a labelled dataset. The right technology here is a columnar warehouse or lake: Parquet on S3, Snowflake, BigQuery, Delta or Iceberg. Reads are large, scans are wide, latency per row is irrelevant; throughput and cost per scanned terabyte are what matter.
The online store answers: "what is n_tx_last_1h for card X right now?" — one entity at a time, called by a payment terminal that has a budget of maybe 20 ms to accept or refuse the transaction. The right technology is a key-value store optimized for point lookups: Redis, DynamoDB, Cassandra, or Postgres with a proper index for lower QPS. Reads are tiny, one row per call, but there are thousands per second and each one is on the critical path of a business decision.
Using Parquet for online lookups is technically possible and operationally lethal: a single-row scan of a Parquet directory takes hundreds of milliseconds, blowing every latency budget an interactive service has. Using Redis to hold the full history is equally lethal: keeping ten years of hourly counters for every card in RAM is a bill no company will approve.
Two SLAs
Offline pays for throughput and completeness: it must return the exact history for the exact events the training set contains, and it must do it in minutes for millions of rows. If it returns the wrong values at , the model is trained on a lie. If it takes an hour to build one dataset, the ML team simply builds fewer datasets and the project slows down.
Online pays for latency and availability: it must return the latest known value in under a few milliseconds, at four or five nines of uptime, because a failed lookup halts a payment. If a lookup takes 200 ms, transactions are refused for timeout even when their features would have said "accept". If it is down for one minute, the business loses that minute.
The two SLAs are almost orthogonal. A design that treats them as one — "we'll put everything in the warehouse and hit it live" or "we'll put everything in Redis and query it batchwise for training" — always ends up bad at one of them, usually both.
Typical technologies
The pragmatic choices in 2026 look like this, and they are not surprising.
| Layer | Common choices | Suits |
|---|---|---|
| Offline | Parquet on S3/GCS/ADLS, Snowflake, BigQuery, Redshift, Delta, Iceberg | Wide scans, cheap cold storage, seasonal reprocessing |
| Online | Redis, DynamoDB, Cassandra, ScyllaDB, Postgres (small QPS), Bigtable | Point lookups under 5 ms, high QPS, per-entity reads |
Feast, Tecton, Hopsworks and Databricks Feature Store all abstract this pair. Behind their APIs it is the same physics: a scan-friendly store paired with a lookup-friendly one, connected by a materialization job.
For our fraud thread we will use Parquet on the local disk as the offline store and Redis in a container as the online store. That is what an evaluation setup looks like on a laptop; production swaps them for S3 and a managed Redis without changing a line of code that consumes features.
How the two stay consistent
Consistency between offline and online is the store's second promise, after the single implementation. It is enforced by three rules that Feast, Tecton and the others all implement the same way.
Same computation function. A feature has one definition, applied on the same source, and the offline history and the online latest value are two projections of that one computation. Materialization does not "compute a new feature"; it takes what the definition produces and pushes it to Redis.
Same source of truth. Both stores read the raw data from the same source described in the registry. If the source is a Parquet directory, both offline and online derive from that directory. If the source is a Kafka topic augmented by a historical bootstrap, both stores agree on which events they see.
A timestamp per feature value. Every value in both stores carries the timestamp for which it is valid. Offline is queried by point-in-time (module 5); online is queried "give me the latest", which internally means "the value with the largest timestamp that has been materialized". That timestamp is what lets the two stores be compared and the store's own freshness be measured (module 6).
from datetime import datetime, timedelta
from feast import FeatureStore
fs = FeatureStore(repo_path=".")
# Offline: reconstruct history for the training set
training_df = fs.get_historical_features(
entity_df=labelled_events,
features=["card_activity_1h:n_tx_last_1h"],
).to_df()
# Online: fetch the latest value at serving time
online = fs.get_online_features(
features=["card_activity_1h:n_tx_last_1h"],
entity_rows=[{"card_id": "c-42"}],
).to_dict()
The two calls read from different physical stores but return values produced by the same definition on the same source; that is what makes them comparable.
Why the two are still not enough
Even with the pair well designed, one silent gap remains: the serving pattern by which the model queries the store. If the training pipeline reads seven features per event but the serving service only requests six because a developer forgot one in the request payload, the model receives a systematically wrong input at prediction time. The store cannot catch this on its own; module 8 and module 9 will tackle it with schemas at the feature-view level and monitoring at the request level.
Pick one entity — one card_id — and compare, for the same wall-clock moment, get_historical_features at that timestamp against get_online_features for that entity. They should return the same value to the byte, up to rounding. If they do not, either your materialization is stale, or your online store's row is from a different definition version. This is a five-minute test and it catches most silent inconsistencies before they reach a model.
Summary
- The offline store answers historical queries for training (throughput and completeness); the online store answers latest-value queries for serving (latency and availability).
- Typical pairs: Parquet / Snowflake offline, Redis / DynamoDB online; using one technology for both bankrupts one SLA.
- Consistency between the two rests on three rules: one definition, one source, one timestamp per value.
- The store cannot fix a serving pattern that requests the wrong list of features; a schema at the feature-view level and request-time monitoring do that.
Next module: defining and versioning features so that evolving them does not break the models that already depend on them.