Module 8 — Online and batch endpoints

The registered model from Module 7 is not yet reachable. Azure ML deploys models behind endpoints — a stable URL — and each endpoint hosts one or more deployments, each pointing at a specific model version on specific compute. The distinction between endpoint and deployment is what makes zero-downtime rollouts possible.

Online versus batch: two different services

Online (managed) endpoints serve individual requests with low latency. They run on always-on VMs behind a REST URL, expose per-request scoring, and bill for the deployed VMs whether traffic arrives or not. Use case: an in-store tablet that asks for a single store's forecast interactively.

Batch endpoints score files. They queue requests, spin up an Azure ML compute cluster, run the scoring in parallel, write results back to a datastore, and shut the cluster down. Bill for compute only while it runs. Use case: the weekly re-forecast of every store's every SKU — 200 stores × 5 000 SKUs × 8 weeks — as a single overnight job.

The demand-forecast project uses both: online for interactive queries during the day, batch for the weekly refresh scheduled in Module 9.

Deploying the online endpoint

An MLflow model registered as in Module 7 needs no scoring script:

# endpoint.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: forecast-online
auth_mode: key
tags:
  project: demand-forecast

# deployment-baseline.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: baseline
endpoint_name: forecast-online
model: azureml:forecast-lgbm:1
instance_type: Standard_DS3_v2
instance_count: 1

az ml online-endpoint create --file endpoint.yml
az ml online-deployment create --file deployment-baseline.yml --all-traffic

The --all-traffic flag sends 100 % of requests to this deployment. It creates the endpoint if it did not exist and provisions the VM in about six minutes.

Test it with the CLI:

az ml online-endpoint invoke --name forecast-online \
  --request-file sample-request.json

Blue-green with traffic splitting

Now register forecast-automl:1 from Module 6 and deploy it as a second deployment on the same endpoint — the classic blue-green setup:

# deployment-challenger.yml
name: challenger
endpoint_name: forecast-online
model: azureml:forecast-automl:1
instance_type: Standard_DS3_v2
instance_count: 1

az ml online-deployment create --file deployment-challenger.yml

Note: no --all-traffic. The new deployment is provisioned but receives zero traffic. Now shift 10 % of live requests to it:

az ml online-endpoint update --name forecast-online \
  --traffic "baseline=90 challenger=10"

The endpoint URL and the request format do not change; clients see nothing. On the server side, one request in ten reaches the AutoML model, and Application Insights (Module 1) records latency and predictions for both. After a week of shadow production data, if the challenger holds up, promote it:

az ml online-endpoint update --name forecast-online \
  --traffic "baseline=0 challenger=100"

Rollback is symmetric — one CLI command, no rebuild. That is the payoff of the two-object model.

Custom scoring for non-MLflow cases

If the registered model were custom_model, deployment would need a score.py:

# score.py
import os, json, joblib
import pandas as pd

_model = None

def init():
    global _model
    path = os.path.join(os.environ["AZUREML_MODEL_DIR"], "model.pkl")
    _model = joblib.load(path)

def run(raw_data):
    payload = json.loads(raw_data)
    df = pd.DataFrame(payload["data"])
    preds = _model.predict(df)
    return preds.tolist()

Referenced from the deployment YAML with code_configuration, alongside an environment that installs whatever score.py imports. The MLflow route saves this file and its associated bug surface — miss an import in score.py and every request 500s until you notice.

Batch endpoints for the weekly refresh

For the 200 × 5 000 × 8 rows of the weekly re-forecast, an online endpoint would be absurd: you would call it once, then never for six days. The batch endpoint uses the compute cluster from Module 2:

# batch-endpoint.yml
name: forecast-batch
auth_mode: aad_token

# batch-deployment.yml
name: baseline
endpoint_name: forecast-batch
model: azureml:forecast-lgbm:1
compute: azureml:cpu-cluster-forecast
resources:
  instance_count: 2
max_concurrency_per_instance: 4
mini_batch_size: 200
output_action: append_row
output_file_name: predictions.csv

az ml batch-endpoint create --file batch-endpoint.yml
az ml batch-deployment create --file batch-deployment.yml --set-default

Invoke it with an input folder of Parquet files — one per store — and it fans them out across the cluster:

az ml batch-endpoint invoke --name forecast-batch \
  --input azureml:forecast-input-week-2026-09-06:1

The cluster spins up, runs, writes predictions.csv to the default datastore, and scales back to zero. Cost is proportional to actual runtime, roughly one hour per week for this workload.

When to pick which

Trait	Online	Batch
Latency budget	Milliseconds to seconds per request	Minutes to hours per job
Cost model	Pay for always-on VMs	Pay only during the run
Failure isolation	Retry the one request	Retry the mini-batch
Right fit for the forecast	Interactive per-store queries	Weekly full-catalogue refresh

Nothing forbids serving one model behind both. The demand-forecast project does exactly that: same model version, one URL for tablets, one endpoint for the weekly pipeline.

The always-on cost trap

A managed online endpoint with two Standard_DS3_v2 instances left running for a demo costs about €150/month per deployment, even with zero traffic. Delete unused deployments (az ml online-deployment delete), or scale their instance_count to zero when idle. Batch endpoints do not have this problem.

Summary

An endpoint is a stable URL; deployments are model-version-plus-compute pairs behind it. Traffic splits between deployments.
MLflow-registered models deploy with no scoring script; custom models need a score.py you own end to end.
Blue-green rollouts are a single az ml online-endpoint update --traffic away, with symmetric rollback.
Pick online for interactive low-latency queries, batch for large scheduled scoring; one model can back both.

Next module: pipelines — composing the training and scoring jobs into a reusable graph that reruns every Sunday.

Online versus batch: two different services​

Deploying the online endpoint​

Blue-green with traffic splitting​

Custom scoring for non-MLflow cases​

Batch endpoints for the weekly refresh​

When to pick which​

Summary​