Module 8 — Batch prediction

The endpoint of module 7 answers real-time payment requests. Some fraud-detection work does not need real time: overnight scoring of the previous day's transactions to feed a review queue, monthly re-scoring of a cohort to measure model drift. That is what batch prediction is for, and it is often 10 to 30 times cheaper than the same volume through an endpoint.

When batch is the right tool, and when it is not

The rule of thumb is not "batch is cheaper" but "batch is appropriate when latency is not part of the requirement". Concretely:

Batch fits	Online fits
Nightly scoring of yesterday's transactions	Live decisioning on the checkout page
Monthly cohort refresh for the CRM	Chatbot response
Backfilling a new model over a year of history	A/B tests where a request must return in 100 ms
Feature computation for a downstream job	Any user-facing surface

The fraud red thread has both. The live decision at payment time goes through the endpoint at p99 100 ms. The overnight scoring — every transaction of the previous day re-scored to catch cases that slipped past the live model or to feed an analyst dashboard — goes through a batch job.

The BigQuery-to-BigQuery job

The model registered in module 6 already carries the serving container declaration. That means a BatchPredictionJob needs to know only where to read and write:

from google.cloud import aiplatform

aiplatform.init(project="fraud-detection-dev", location="europe-west1")

model = aiplatform.Model("projects/…/models/1234567890@7")  # champion version

batch = model.batch_predict(
    job_display_name="fraud-batch-2026-09-06",
    bigquery_source="bq://fraud-detection-dev.raw.transactions_2026_09_06",
    bigquery_destination_prefix="bq://fraud-detection-dev.predictions",
    machine_type="n1-standard-4",
    starting_replica_count=4,
    max_replica_count=20,
    generate_explanation=False,
    labels={"env": "prd", "team": "fraud", "job": "nightly-scoring"},
)

batch.wait()

Two Vertex conventions apply here.

bigquery_destination_prefix — Vertex creates a dataset, not a table. The output tables predictions_<timestamp> and errors_<timestamp> are written under that dataset. The next section shows how to join them back.

Autoscaling is per-batch. starting_replica_count=4, max_replica_count=20 tells Vertex to start with 4 workers and scale up if the queue backs up. On the 12 M rows of a daily partition, the job finishes in about 15 minutes with 8 workers active — a fraction of the cost of pushing the same 12 M rows through the online endpoint.

The cost gap between batch and online

Roughly:

• Online: the endpoint runs min_replica_count replicas 24/7. At n1-standard-2 ($0.10/hour per replica) and min=2, that is about $150 per month even at zero traffic. Live QPS adds proportional cost.
• Batch: replicas run only during the job. A 15-minute job on 8 n1-standard-4 replicas costs about $0.40 — for 12 M predictions. That is roughly $0.03 per million rows.

For the nightly scoring of 12 M rows, an equivalent volume through the endpoint would cost $8–$12 in serving time alone (per-request compute plus the always-on baseline of the extra replicas needed to absorb the spike). The 20–30× gap is why nobody sends nightly refresh traffic to an online endpoint.

The trap is running batch when latency actually matters. A weekly fraud audit that must catch new patterns "as soon as possible" is not a latency requirement. A checkout decision is.

Joining the predictions back to the source

The output table has three columns: instance (the row Vertex read), prediction (the model's output) and prediction_meta (deployment id, model version, timestamp). The prediction column is a nested struct — for XGBoost binary classification, it contains the probability of the positive class.

CREATE OR REPLACE TABLE `fraud-detection-dev.scored.transactions_2026_09_06` AS
SELECT
  t.transaction_id,
  t.amount,
  t.merchant_id,
  t.created_at,
  p.prediction[OFFSET(0)] AS fraud_probability,
  IF(p.prediction[OFFSET(0)] >= 0.85, TRUE, FALSE) AS flag_for_review
FROM `fraud-detection-dev.raw.transactions_2026_09_06`  AS t
JOIN `fraud-detection-dev.predictions.predictions_<ts>` AS p
  ON t.transaction_id = JSON_VALUE(p.instance, "$.transaction_id")

Two rules that avoid a class of silent bugs.

The join key must be present in the instance you predict on. If transaction_id is not in the feature set the model was trained on, it must still appear in the source table so it can appear in the output. Vertex passes the entire input row through as instance, which is exactly what makes the join possible.

Choose the threshold on the validation set, not by inspection. The 0.85 in the query is not a magic number. Module 6 attached PR-AUC to the model; the operating threshold is picked to hit the target precision at that recall level, then held constant across runs. Any drift in the population is a signal to review the threshold, not to adjust it on the fly.

Scheduling: Cloud Scheduler and Vertex AI Pipelines

A batch job that runs "when someone remembers" is not a batch job. Two mechanisms schedule it.

Cloud Scheduler + Cloud Function. A cron entry (0 3 * * * for 3 a.m. every day) triggers a Cloud Function that calls model.batch_predict(...). Simple, cheap, appropriate for standalone jobs.

Vertex AI Pipelines scheduled runs (module 9). A KFP pipeline that reads the day's partition, runs batch prediction, joins the output back to the source table, and writes a monitoring metric — all as one lineage-tracked execution. This is what a production pipeline looks like, and it is what module 9 builds.

For now, a Cloud Scheduler cron entry is enough:

gcloud scheduler jobs create http fraud-nightly-scoring \
    --schedule="0 3 * * *" \
    --time-zone="Europe/Paris" \
    --uri="https://europe-west1-fraud-detection-dev.cloudfunctions.net/trigger-batch" \
    --http-method=POST \
    --oidc-service-account-email=vertex-training@fraud-detection-dev.iam.gserviceaccount.com

Comparing champion and challenger at scale

The 90/10 split of module 7 gives a small sample per day. A batch prediction with both versions on the same population yields a full-population comparison — the ground truth for whether v7 is really better than v5.

scores_v5 = aiplatform.Model("projects/…/models/1234567890@5").batch_predict(
    job_display_name="fraud-batch-v5-2026-09-06",
    bigquery_source="bq://fraud-detection-dev.raw.transactions_2026_09_06",
    bigquery_destination_prefix="bq://fraud-detection-dev.predictions_v5",
    machine_type="n1-standard-4",
    starting_replica_count=4,
)

scores_v7 = aiplatform.Model("projects/…/models/1234567890@7").batch_predict(...)

A simple SQL join then computes disagreement rate, per-category deltas, and (once labels arrive a few days later) precision and recall for each version. This is the honest challenger evaluation the small live split of module 7 only hints at.

Batch is the right place for explanations

Setting generate_explanation=True on a batch job adds attribution scores (roughly like SHAP values) to every prediction, at a modest extra cost per row. Doing this on an online endpoint doubles or triples latency; doing it in batch is affordable and gives the review team a "why" column next to the score.

In summary

• Batch prediction is 20–30× cheaper than the same volume through an online endpoint; the choice is not about cost but about whether latency is part of the requirement.
• A batch_predict from BigQuery to BigQuery lets Vertex create the output dataset; the join key must be present in the input row so the score can be linked back to the transaction.
• Choose the decision threshold on the validation set (module 6's evaluation) and hold it constant; drift in the population is a signal, not permission to adjust.
• Schedule the job with Cloud Scheduler for standalone workflows, and with Vertex AI Pipelines (module 9) once the batch is one step of a larger DAG with lineage and quality gates.

Next module: wrapping the whole flow — data pull, training, evaluation, batch scoring — into a Vertex AI Pipeline with KFP components, artifacts and lineage.