Module 6 — Materialization and freshness
We have a definition, an offline history, and an online store waiting to be read. The missing piece is what puts the latest value into that online store, at what frequency, and how we know it is current enough. This is materialization, and freshness is the number that measures whether the store is keeping its promise.
What materialization does, in one sentence
Materialization is the job that reads the offline history, computes the latest value of each feature for each entity, and writes it to the online store. In Feast:
feast materialize-incremental $(date -u +%Y-%m-%dT%H:%M:%S)
That command finds every feature view, looks at the last materialization timestamp, computes values from that timestamp up to now, and pushes them to Redis or DynamoDB. The next call picks up where the previous one stopped. Nothing is recomputed twice, nothing is skipped.
Materialization is where the SLA of "the online store answers with the latest value" becomes real. A store that is never materialized still answers — with stale numbers.
Batch, on a cadence chosen by the feature
Every feature has a natural cadence, dictated by how fast the underlying event stream produces new information and how tight the model's decision loop is.
A country of residence changes for a card at most once every few years. Materializing daily is already over-engineered; weekly is fine, and a nightly job that runs anyway can carry it for free.
A "number of transactions in the last hour" changes every minute. Materializing hourly makes the value up to an hour late — 60 minutes of stale counter for a feature whose whole point is to detect a spike. Fifteen minutes is a reasonable batch cadence; anything shorter usually calls for streaming.
A user's session count for a recommendation service may change per click. Batch is the wrong tool; streaming is.
Two numbers frame the choice: information delay (how long between the event and its arrival in the source) and decision delay (how long between the source ingesting the event and the decision that needs it). Batch materialization at cadence makes the feature at most later than its source; streaming makes it seconds later.
Streaming features, and why they are not always what you want
For truly fast-changing features, a streaming job — Flink, Spark Structured Streaming, or a lightweight custom consumer — subscribes to a Kafka topic, updates a rolling aggregate in memory, and writes the value directly to the online store as it changes.
Feast supports streaming through a push source: the streaming job pushes rows to the store via fs.push() or the HTTP push endpoint, and the store applies the same schema and TTL as any other source. The offline history is still fed by an occasional batch dump from the topic, so training remains reproducible.
Streaming is worth the operational cost when the feature genuinely needs it. Two situations do not need it. First, features that only enter models retrained daily — no matter how fresh you keep them, they still meet a model that only sees them at 3 a.m. Second, features whose freshness is bottlenecked upstream — if the transaction feed itself is 10 minutes late, a streaming aggregator on top of it still produces 10-minute-late features, at ten times the cost of a batch job.
Measuring freshness
"Fresh enough" is a promise that must be measured, not asserted. Freshness is a duration: for each feature and each entity, the time between now and the timestamp of the latest value in the online store.
from datetime import datetime, timezone
import redis
r = redis.Redis()
def freshness_seconds(entity_key: str) -> float:
ts_bytes = r.hget(f"feast:card_activity_1h:{entity_key}", "event_ts")
ts = datetime.fromisoformat(ts_bytes.decode())
return (datetime.now(timezone.utc) - ts).total_seconds()
In practice the store computes freshness for you and exports it as a metric. What you decide is the SLO per feature: "the 99th percentile of card_activity_1h freshness is under 20 minutes". That SLO becomes a monitor whose failure page a human. Course 20's dashboards are where these numbers live.
Two aggregate views of freshness matter, and confusing them hides problems.
Per-feature freshness exposes a broken pipeline: if card_activity_1h is 4 hours stale for every card, materialization has failed globally.
Per-entity freshness exposes a dropped partition: if card_activity_1h is fresh for 95 % of cards but 3 hours stale for a subset, a partition of the source is not being ingested, and the average freshness reported at the feature level looks fine while the affected cards silently get wrong scores.
Cost, and why the wrong cadence is the biggest bill
The dominant cost of a feature store is not storage; it is the materialization job. A window aggregate over 90 days of transactions, recomputed every 15 minutes, is a warehouse scan every 15 minutes.
Two techniques trim this bill without giving up freshness.
Incremental computation. Instead of recomputing "number of transactions in the last hour" from scratch, keep a rolling window in state and only add new events / evict old ones. Feast's incremental materialization handles the housekeeping between runs; the aggregate itself is your responsibility to write incrementally in the pipeline.
TTL-aware TTL. Do not materialize features whose consumers no longer need them at the same cadence. If the fraud model retrains weekly, a 15-minute cadence for a rarely-changing feature it consumes is money you burn. Tag features with their consumer's decision cadence and align to the fastest active consumer, not to a nominal ideal.
The single dashboard that catches most materialization problems shows three panels: (1) freshness p99 per feature view, (2) materialization job duration per run, (3) row count materialized per run. A spike in duration with a drop in row count means the job is scanning wide and finding little — usually a source with the timestamp column mislabelled. A flat freshness with growing job duration means the aggregate is quadratic in the window and about to run out of budget.
Summary
- Materialization is the job that pushes the latest value from the offline history to the online store; without it the online store answers with stale values.
- Batch at a cadence chosen per feature is the default; streaming (via a push source) is worth it only when the feature genuinely needs it and the source itself is fast.
- Freshness is the duration between now and the latest online value; per-feature freshness spots a broken pipeline, per-entity freshness spots a dropped partition.
- Cost is dominated by the materialization job, not by storage; incremental aggregates and consumer-aware cadence keep the bill honest.
Next module: Feast in practice — a working end-to-end setup on Parquet and Redis for the fraud scoring use case.