Skip to main content

Module 10 — Monitoring, quotas and costs

The pipeline from Module 9 retrains the forecast every Sunday. That does not guarantee it stays useful: the world shifts between retrains, requests move faster than the schedule can react, and the whole workspace burns money whether the model helps or not. This module closes the loop on the operational side — model drift, resource limits, and the bill.

Model monitoring: watching for drift​

Azure ML ships a native model monitoring service that compares production inference data against the training baseline on a schedule. Set it up once per endpoint, receive an alert when the world moves.

# monitor-forecast.yml
$schema: https://azuremlschemas.azureedge.net/latest/monitorSchedule.schema.json
name: forecast-drift-monitor
trigger:
  type: recurrence
  frequency: week
  interval: 1
create_monitor:
  compute:
    instance_type: standard_ds3_v2
    runtime_version: "3.4"
  monitoring_target:
    ml_task: regression
    endpoint_deployment_id: azureml:forecast-online:baseline
  monitoring_signals:
    data_drift:
      type: data_drift
      production_data:
        input_data:
          type: uri_folder
          path: azureml:endpoint-logs-forecast@latest
      reference_data:
        input_data:
          type: mltable
          path: azureml:sales-mltable:1
      features:
        top_n_feature_importance: 10
      metric_thresholds:
        numerical:
          jensen_shannon_distance: 0.1
        categorical:
          pearsons_chi_squared_test: 0.05

Two signal types matter for the forecast. Data drift compares the distribution of each feature — store, SKU, week — between training and production; when a store closes for renovation, its share of requests drops and the monitor flags it. Prediction drift compares the distribution of the model's outputs against the training target; a promotion that spikes demand shifts the output distribution before the effect on error is measurable.

The monitor writes results back to the workspace: dashboards in Studio, alerts through Azure Monitor. The threshold matters more than the metric: a Jensen-Shannon of 0.1 catches meaningful moves without paging on every seasonal ripple.

Azure Monitor and Application Insights​

The four attached resources from Module 1 finally pay off. Application Insights, wired into every managed online endpoint, collects per-request telemetry: latency percentiles (p50, p95, p99), success rate, exception traces. A p99 that grows from 150 ms to 400 ms after a deployment is the earliest warning of a slower model — Module 6's AutoML ensemble catches this in production before the drift monitor catches it in metrics.

Azure Monitor aggregates the App Insights signals plus workspace-level metrics (job successes, job failures, node hours, quota usage). Alert rules there translate a threshold into an email, a Teams message, or a webhook into an incident tool.

The minimum alert set worth the ceremony:

  • Endpoint availability drops below 99 %
  • Endpoint p95 latency exceeds the SLA (say 250 ms)
  • Weekly pipeline fails on Sunday night
  • Quota usage above 80 % on the region

Cores quotas: the invisible ceiling revisited​

Module 2 introduced the per-region cores quota. Two months into the project, that ceiling starts to bite: the sweep from Module 5 competes with the batch endpoint from Module 8 for Standard_DS cores, and both queue.

Check the current picture:

az ml compute list-usage --location westeurope -o table

The output shows current versus limit per VM family and tier. Two habits scale.

Split VM families across workloads. Training on Standard_DS, GPU inference on Standard_NC — they draw on different quota buckets, so one does not starve the other. Batch scoring can often run on the cheaper Standard_DS2_v2 family too, spreading the load.

Request quota increases before you need them. The portal request opens a ticket that resolves in a few hours to a few days. The night before a demo is not the moment to discover you are one node short.

Cost analysis: tag your way to visibility​

The tags planted on the workspace and its resources in Module 1 come home now. Azure Cost Management can group any cost view by tag:

  • project=demand-forecast attributes every resource — workspace, storage, ACR, compute nodes, endpoints — to a project. When the executive asks "how much does the forecast cost us per month?", one query answers.
  • costcenter=CC-4711 feeds internal chargeback. Data teams that cross-charge cannot rely on Azure's default subscription view; they need the tag.
  • owner=data-team points at a human when a resource is orphaned.

Tags apply top-down: tag the resource group and the workspace, and every resource created inside inherits the tags — for future resources only. Existing resources need a bulk retag pass; a monthly Azure Policy that flags untagged resources catches the drift automatically.

A useful monthly budget in Cost Management sends an alert when spend crosses a percentage of a target. For the forecast project, a €1 000/month budget with alerts at 50 %, 80 % and 100 % is enough to catch a runaway sweep the same day it happens.

Auto-shutdown for compute instances​

Compute instances (Module 2) bill continuously until stopped. The forecast team's data scientists tend to leave them running "in case". Every workspace can enforce auto-shutdown:

schedules:
  compute_start_stop:
    - action: stop
      trigger:
        type: recurrence
        frequency: day
        interval: 1
        schedule:
          hours: [20]
          minutes: [0]
        time_zone: Europe/Paris

An idle instance stopped at 20:00 local time saves 12 hours a day of billing per person. Restart in the morning takes about a minute. This one policy typically halves the instance line on the monthly bill.

The full monitoring dashboard, once a week​

The routine that keeps a production Azure ML setup healthy is short and boring, which is the point:

  1. Open Cost Management, filter by project=demand-forecast, compare this week to last week.
  2. Check quota usage per region.
  3. Open the drift monitor, look for red signals.
  4. Open App Insights, spot-check p95 latency and error rate on the endpoint.
  5. Read the last Sunday pipeline run in Studio.

Fifteen minutes, once a week. The moment any of the five items screams, follow the thread.

The auto-shutdown blind spot

Auto-shutdown covers compute instances, not compute clusters, not online endpoints, not batch endpoints. A cluster with min-instances 0 shuts itself; an endpoint with two instances runs forever. The line items live in different places on the bill; you have to check each. Do not assume one policy protects all.

Summary​

  • Model monitoring compares production data to the training baseline; set the thresholds to catch real shifts, not seasonality noise.
  • Application Insights + Azure Monitor cover the endpoint side — latency, availability, error rate — with alerts you actually act on.
  • Cores quotas are per region and per VM family; split workloads across families, request increases early.
  • Tags + Cost Management + budgets turn a monthly Azure bill into an answerable question; auto-shutdown halves the compute-instance line but does not cover endpoints.

Next module: the recap and the 40-question exam.