Module 8 — Batch, online, and streaming serving

Module 6 packaged the churn model as an HTTP service. That decision is not automatic. Depending on when predictions are needed and how fresh the inputs must be, three serving modes exist — and picking the wrong one has quiet, expensive consequences. This module names the modes, gives the latency test that decides between them, and warns about the trap that unites them all.

Three modes, defined by the latency they promise

Batch scoring runs on a schedule (nightly, hourly) and writes predictions to a table. The retention team opens their dashboard in the morning and sees the top 5 000 at-risk subscribers. Latency between a customer's action and a scored prediction: hours to a day.

Online serving exposes the model as a synchronous API. A customer opens their account page and the personalized offer is chosen from the fresh churn score. Latency: tens of milliseconds, tail latency in the low hundreds.

Streaming inference consumes events from a message bus (Kafka, Kinesis, Pub/Sub) and produces predictions as events land. A customer's payment fails; within seconds, the retention system knows this raised their churn score, and a workflow kicks off. Latency: seconds.

The three modes are not variants of one thing. They imply different infrastructure, different testing, different monitoring, and — the trap this module is really about — often different feature computations.

The latency test

The temptation is to build the online service first and use it for everything, because that seems more sophisticated. It usually is not the right call. The correct question is: when does the prediction actually need to be available?

If the answer is "the retention team calls the top-N tomorrow morning", batch is right. Batch is cheaper (a single scheduled job, no service to keep alive, no scaling issues), simpler (a Python script writing to a table), and easier to monitor (a job succeeded or failed). Building an API to feed a daily dashboard is a solved problem being turned into an interesting one.

If the answer is "the answer must appear on a page the customer just loaded", online is right. Batch cannot serve a UI: the prediction must be computed for this subscriber, right now, based on their current session.

If the answer is "the world will change in seconds and the prediction must follow", streaming is right. A single failed payment, a support call that opens a ticket, a rate change — anything that reshuffles the churn score in real time.

Batch scoring for the churn project

The retention workflow scores subscribers every night at 3 a.m.:

# jobs/score_batch.py
import pandas as pd
import mlflow.pyfunc

model = mlflow.pyfunc.load_model("models:/churn-classifier@production")
subs = pd.read_sql("SELECT * FROM subscribers_features_daily", conn)
subs["churn_score"] = model.predict(subs[FEATURES])
subs[["subscriber_id", "churn_score", "as_of_date"]].to_sql(
    "churn_scores", conn, if_exists="append", index=False
)

Two properties matter. The job is idempotent: rerunning it for the same date overwrites the same rows. It is auditable: as_of_date and the model version (log it explicitly) let anyone trace a score back to its origin.

The online API — same model, different constraints

The FastAPI service from module 6 is the online mode. Its constraints are strict: p99 latency under 100 ms, no cold-start pauses, no external calls in the request path except cached lookups.

The feature lookup problem appears here. The batch job runs SELECT * FROM subscribers_features_daily — a table computed by a nightly Spark job. The online API cannot afford that table's latency; it needs the features for one subscriber in milliseconds. Two approaches: precompute the features per subscriber into a low-latency store (a feature store, cover in Course 33) and look them up by ID; or compute the features on the fly from raw data, which usually kills the latency budget.

Streaming for the churn project

A payment failure lands as a Kafka event. A consumer joins the event with the subscriber's cached features, calls the model, and pushes a score update:

# services/score_stream.py
for event in consumer:  # from Kafka
    sub_id = event["subscriber_id"]
    features = feature_store.get(sub_id)  # low-latency lookup
    features["payment_failed_last_day"] = True
    score = model.predict([features])[0]
    producer.send("churn_scores", {"subscriber_id": sub_id, "score": score})

Streaming inherits the online mode's latency budget and adds two constraints: the consumer must be fault-tolerant (a redeploy cannot lose messages), and the model must be stateless or its state must be externalized to the feature store.

The trap that unites the three modes: feature consistency

Batch trains on features computed by a Spark SQL query. The online API computes them in Python. The streaming consumer computes them yet another way. The three implementations drift, subtly. A rolling seven-day sum computed in Spark treats the current day as inclusive; the Python one excludes it; the streaming one uses "the last 168 hours". The model was trained on the Spark version. In production, the API and the streaming consumer feed it slightly different numbers, and its predictions degrade — for reasons that will look, to the on-call engineer, like model drift.

The cure is one source of truth for feature definitions. A shared feature module, invoked identically at training time (against the historical table) and at inference time (against fresh data). Course 33 shows a feature store implementing exactly this contract. Until then, a single Python module used from all three serving jobs, with a unit test per feature, is a reasonable start.

Reach for batch before online

For internal dashboards, retention campaigns, credit review overnight, monthly report generation — batch is almost always the right answer. Deploying a REST API to feed a daily table is a common overbuild: it doubles the surface area to monitor and gives the illusion of freshness that the downstream process cannot use.

Summary

• Three serving modes: batch (hours), online (tens of ms), streaming (seconds). Latency of the consumer decides.
• Batch is cheaper and simpler; do not build an API where a scheduled script would suffice.
• Online adds the feature lookup problem in milliseconds; streaming adds fault tolerance and state.
• The unifying trap is feature inconsistency between training and each serving mode — one source of truth for feature definitions, tested identically wherever it runs.

Next module: monitoring — noticing that the world has changed before the business does.