Module 10 — Monitoring, alerts and cost control

The churn model is trained, tuned, deployed and orchestrated. Two loose ends remain: making sure it keeps working as the world changes, and making sure it does not silently bankrupt the team. Both are matters of setting up a few alarms once and forgetting them; the "once" is the part that gets skipped.

Data capture, the prerequisite

Model Monitor cannot compare production traffic to training data if it does not see the traffic. Enable data capture on the endpoint before anything else.

from sagemaker.model_monitor import DataCaptureConfig

capture = DataCaptureConfig(
    enable_capture=True,
    sampling_percentage=100,
    destination_s3_uri=f"s3://{bucket}/churn/endpoints/churn-scoring/capture/",
)
predictor = model.deploy(
    endpoint_name="churn-scoring",
    initial_instance_count=1, instance_type="ml.m5.large",
    data_capture_config=capture,
)

At 100 % sampling every request and response is written to S3. On a low-traffic endpoint this is fine; at ten thousand requests per second, drop to 5 %. The capture is what all four Monitor jobs consume.

Data quality: the sanity check

A data quality baseline is computed once, on the training set. It records the schema, the type of every feature, and the empirical distribution.

from sagemaker.model_monitor import DefaultModelMonitor
from sagemaker.model_monitor.dataset_format import DatasetFormat

monitor = DefaultModelMonitor(role=role, instance_count=1, instance_type="ml.m5.xlarge")

monitor.suggest_baseline(
    baseline_dataset=f"s3://{bucket}/churn/processed/train/",
    dataset_format=DatasetFormat.parquet(),
    output_s3_uri=f"s3://{bucket}/churn/monitor/baseline/data-quality/",
)

monitor.create_monitoring_schedule(
    monitor_schedule_name="churn-data-quality",
    endpoint_input=predictor.endpoint_name,
    statistics=monitor.baseline_statistics(),
    constraints=monitor.suggested_constraints(),
    schedule_cron_expression="cron(0 * ? * * *)",   # hourly
    output_s3_uri=f"s3://{bucket}/churn/monitor/reports/data-quality/",
)

Every hour, a small job compares the last hour of captures to the baseline. A missing column, a change of type or a value outside the training range writes a violation to the output S3 prefix and emits a CloudWatch metric. This catches the boring failures: the schema change nobody warned you about, a nullable column suddenly full of nulls, a categorical column with a new modality the encoder does not know.

Model quality: does the model still work

The three other Monitor jobs — Model Quality, Bias Drift, Feature Attribution Drift — need the ground truth. For the churn model, that means the actual churn flag, observed one or two months after the prediction. Once labels arrive and are merged with the captured predictions on S3, Model Quality computes AUC, F1 or whatever metric was baselined and raises an alarm if it drops.

The delay is inherent: you cannot know at prediction time whether a customer will churn in ninety days. The way to close the loop is to define the label pipeline at the same time as the training pipeline, not later — a scheduled Glue or Athena job that emits the truth once observed, into a fixed S3 prefix Model Monitor reads.

Alerts: CloudWatch and SNS

Model Monitor emits metrics into a AWS/SageMaker/ModelBuildingPipeline and aws/sagemaker/Endpoints namespaces. A CloudWatch alarm turns them into pages.

import boto3
cw = boto3.client("cloudwatch")

cw.put_metric_alarm(
    AlarmName="churn-data-quality-violations",
    Namespace="aws/sagemaker/Endpoints/data-metrics",
    MetricName="feature_baseline_drift_ Age",
    Dimensions=[{"Name": "Endpoint", "Value": "churn-scoring"},
                {"Name": "MonitoringSchedule", "Value": "churn-data-quality"}],
    Statistic="Maximum", Period=3600, EvaluationPeriods=1, Threshold=0.2,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:eu-west-1:123456789012:ml-alerts"],
)

The SNS topic on the last line is what carries the message to a PagerDuty, a Slack channel or an email list. An alarm without a subscription is documentation, not monitoring.

The invoice, at last

Two orthogonal tools cover cost.

AWS Budgets sets a monthly limit and sends an email at 80 % and 100 % of it. Set it per project with a cost allocation tag applied to every resource — SageMaker respects sagemaker:* tags on training jobs, endpoints, notebook instances. The tag is the linchpin: without it, budgeting rolls up to the account and blames everything on "SageMaker".

Cost Explorer, on the same tag, produces the monthly breakdown you use in reviews.

The five causes of a surprise bill

Every SageMaker invoice surprise falls into one of these buckets. Learning them saves the phone call to the CFO.

1. A notebook or Studio kernel left running. ml.g4dn.xlarge at $0.74 per hour is $18 per day; the auto-shutdown of module 2 fixes it.
2. A real-time endpoint you forgot to delete after a demo. ml.m5.large at $0.115 per hour is $83 per month, silently. aws sagemaker list-endpoints and a weekly cron that flags anything running is a fifty-line script.
3. A misconfigured autoscaling policy. MaxCapacity=100 on a target of 100 invocations per instance means SageMaker happily scales to a hundred instances if traffic spikes; set a MaxCapacity that reflects the real budget, not what looks safe.
4. Tuning jobs with too many trials on an expensive instance. max_jobs=200 on ml.p3.2xlarge at $3.83 per hour, at 30 minutes per trial, is $383 for one experiment. Start on a cheap instance with max_jobs=30, then scale up on the winning region of the space.
5. Data capture at 100 % sampling on a high-traffic endpoint. A million requests per hour with a 5 KB payload each is 120 GB per day into S3, plus the Monitor jobs that read it. Drop to 5 % on production endpoints.

Weekly review, thirty minutes

Every Monday, ten minutes on Cost Explorer for the previous week, five on the endpoint list, five on the tuning-job history, five on the Model Monitor reports for open violations, five on the notebook instances. Once a rhythm is in place, none of the surprises above lasts more than a week — which is the point.

Summary

• Data capture on the endpoint is the prerequisite; without it, Model Monitor has nothing to inspect.
• Data quality catches schema breaks hourly; Model Quality needs the ground-truth pipeline planned upfront.
• CloudWatch alarms + SNS turn metrics into pages; an alarm without a subscription is documentation.
• The invoice has five recurring causes: forgotten kernel, forgotten endpoint, wide autoscaling ceiling, expensive tuning, over-sampled capture. All are cheap to prevent.

Next: the recap and the 40-question exam.