Skip to main content

Module 7 — Real-time and serverless endpoints

The winning tuning job of module 6 produced a model.tar.gz on S3. This module turns that artifact into an HTTPS endpoint the rest of the company can call. SageMaker offers three shapes of endpoint — real-time, serverless, and asynchronous — and the right choice depends more on traffic patterns than on the model itself.

Three objects to deploy anything​

Deployment splits into three objects, in this order.

The Model is the pointer to the artifact and the image that serves it. The Endpoint configuration describes how many instances, which variants, and how they scale. The Endpoint is the running resource with a URL.

from sagemaker.sklearn.model import SKLearnModel

model = SKLearnModel(
    model_data=tuner.best_estimator().model_data,   # s3://.../model.tar.gz
    role=role,
    entry_point="inference.py",
    source_dir="src/",
    framework_version="1.2-1",
)

predictor = model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.large",
    endpoint_name="churn-scoring",
)

model.deploy does the three steps at once. In production, you split them so the same endpoint configuration can be reused for blue-green updates.

The inference script​

The training script train.py is not what serves predictions; a separate inference.py provides four functions SageMaker calls, in order, on every request. All four have defaults; you override only the ones the built-in behavior gets wrong.

import joblib, os, json
import pandas as pd

def model_fn(model_dir):
    """Load the model once when the container starts."""
    return joblib.load(os.path.join(model_dir, "model.joblib"))

def input_fn(request_body, content_type):
    if content_type == "application/json":
        return pd.DataFrame(json.loads(request_body))
    if content_type == "text/csv":
        return pd.read_csv(io.StringIO(request_body), header=None)
    raise ValueError(f"unsupported content type: {content_type}")

def predict_fn(input_data, model):
    return model.predict_proba(input_data)[:, 1]

def output_fn(prediction, accept):
    return json.dumps(prediction.tolist()), accept

model_fn runs once at container startup; expensive loading belongs here, not in predict_fn. The other three run per request.

Autoscaling​

An endpoint on one ml.m5.large handles roughly 300 requests per second on this scikit-learn model. Above that, latency climbs and requests queue. Autoscaling adjusts the instance count based on a target metric.

import boto3

client = boto3.client("application-autoscaling")
resource_id = "endpoint/churn-scoring/variant/AllTraffic"

client.register_scalable_target(
    ServiceNamespace="sagemaker", ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    MinCapacity=1, MaxCapacity=10,
)
client.put_scaling_policy(
    PolicyName="churn-invocations", ServiceNamespace="sagemaker",
    ResourceId=resource_id,
    ScalableDimension="sagemaker:variant:DesiredInstanceCount",
    PolicyType="TargetTrackingScaling",
    TargetTrackingScalingPolicyConfiguration={
        "TargetValue": 200.0,
        "PredefinedMetricSpecification": {
            "PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
        },
        "ScaleInCooldown": 300, "ScaleOutCooldown": 60,
    },
)

Two facts to keep in mind. Scale-out is quick, scale-in is slow: the cooldowns above give scale-out a minute and scale-in five, which matches how bad each mistake feels — a slow scale-out drops requests, a fast scale-in drops instances that were about to be needed again. And MinCapacity never goes to zero on a real-time endpoint: at least one instance is billed 24/7. That is the boundary where serverless becomes attractive.

Serverless inference​

Serverless endpoints run only when a request arrives. There is no instance cost between requests; you pay a small per-request fee and per-second compute during the invocation.

from sagemaker.serverless import ServerlessInferenceConfig

serverless_config = ServerlessInferenceConfig(memory_size_in_mb=2048, max_concurrency=20)
serverless_predictor = model.deploy(
    endpoint_name="churn-scoring-serverless",
    serverless_inference_config=serverless_config,
)

The catch is the cold start. When no container is warm, the first request pays for spinning one up — one to five seconds on a small scikit-learn model, ten to twenty on a PyTorch model with heavy weights to reload from S3. Subsequent requests hit the warm container and return in the model's normal latency. When traffic dies for a few minutes, the container is torn down and the next request pays for another cold start.

The break-even is easy to compute. A real-time ml.m5.large costs about $83 per month up. On serverless at 2 GB, each invocation costs about $0.00002 of memory-time on this model plus $0.20 per million requests. Below roughly 4 million invocations per month, or on any workload with clumpy traffic and tolerance for the cold start, serverless wins. Above that, or when p99 latency matters more than the invoice, real-time wins.

Production variants and blue-green​

An endpoint can host several production variants side by side, each with its own model, instance type, and weight. Sending 10 % of traffic to a candidate and 90 % to the current model is one API call, and rolling back is one more.

predictor.update_endpoint(
    endpoint_config_name=new_config_name,     # includes both variants
    wait=True,
)

client = boto3.client("sagemaker")
client.update_endpoint_weights_and_capacities(
    EndpointName="churn-scoring",
    DesiredWeightsAndCapacities=[
        {"VariantName": "Baseline",  "DesiredWeight": 90.0},
        {"VariantName": "Candidate", "DesiredWeight": 10.0},
    ],
)

Course 20's canary story ported: no gateway to configure, no Kubernetes controller to write. What did not move is the discipline — the candidate must be gated on a metric read from real traffic, not on a smile.

The cold start you did not measure

Cold-start numbers vary by an order of magnitude depending on the model size and image. Measure yours by hitting the endpoint after five minutes of idleness, three times, and report the p99, not the p50. On a churn model with 40 features and a Random Forest, expect 1–2 s. On an XGBoost model of 500 MB, expect 5–10 s. On a PyTorch model over 1 GB, expect 15–30 s — enough to break most user-facing use cases.

Summary​

  • Deploy in three objects: Model, endpoint configuration, endpoint; deploy chains them, split them for blue-green.
  • inference.py overrides model_fn, input_fn, predict_fn, output_fn; model_fn runs once at startup.
  • Autoscaling with a target invocations-per-instance metric; asymmetric cooldowns, and one instance is always billed.
  • Serverless pays only for invocations, at the cost of a cold start; break-even around 4 M requests per month for a small model.

Next module: batch transform, for the churn scoring job that runs once a month.