Module 7 — Endpoints, traffic split and scaling

Module 6 put two versions of the fraud model in the registry: version 5 currently serving as champion, version 7 as challenger. This module puts both behind a single Endpoint, splits the payment traffic 90/10 between them, and sets the replica autoscaling and logging that make the deployment observable in production.

What an endpoint actually is

A Vertex Endpoint is a stable network address (region-scoped, one URL) that fronts one or more DeployedModel entries. Each DeployedModel binds:

• a specific model version from the registry (module 6),
• a machine type for the serving VM,
• a min and max replica count for autoscaling,
• a traffic share as an integer percentage.

The consumer (the fraud-detection microservice calling the endpoint) sees one URL; Vertex handles the routing. That indirection is what enables safe rollouts: adding a new version at 10 % traffic is a config change, not a client change.

Creating the endpoint and deploying the champion

from google.cloud import aiplatform

aiplatform.init(project="fraud-detection-dev", location="europe-west1")

endpoint = aiplatform.Endpoint.create(
    display_name="fraud-xgb-endpoint",
    labels={"env": "prd", "team": "fraud"},
)

model_v5 = aiplatform.Model(
    "projects/…/models/1234567890@5",  # explicit version, not "default"
)

model_v5.deploy(
    endpoint=endpoint,
    deployed_model_display_name="fraud-xgb-v5",
    traffic_percentage=100,
    machine_type="n1-standard-2",
    min_replica_count=2,
    max_replica_count=6,
    service_account="fraud-serving@fraud-detection-dev.iam.gserviceaccount.com",
    enable_access_logging=True,
    explanation_metadata=None,
)

Two decisions in that block deserve naming.

min_replica_count=2, not 1. One replica means one machine; that machine will be updated, restarted or preempted at some point, and the endpoint will drop requests during that minute. Two replicas is the minimum for zero-downtime updates.

machine_type="n1-standard-2". For an XGBoost model with tabular features, a 2-vCPU machine handles a few hundred requests per second per replica. GPUs on serving are only worth it for deep-learning models with actual matrix compute; a tabular model on a GPU serves the same QPS at 10× the cost.

Adding the challenger with a 90/10 split

Deploying v7 at 10 % and pushing v5 to 90 % is one call:

model_v7 = aiplatform.Model("projects/…/models/1234567890@7")

model_v7.deploy(
    endpoint=endpoint,
    deployed_model_display_name="fraud-xgb-v7",
    traffic_split={"fraud-xgb-v5": 90, "fraud-xgb-v7": 10},
    machine_type="n1-standard-2",
    min_replica_count=2,
    max_replica_count=6,
)

The traffic_split dictionary must cover every deployed model on the endpoint and sum to exactly 100. A 90/10 split with a third leftover model at 0 % is fine — the model stays deployed and warm. Missing a model or summing to 99 returns a 400 Bad Request.

Every prediction request is routed to one deployed model, picked at random with the given weights. This is a per-request split, not a per-user split. If two calls to the endpoint must go to the same version for consistency (a fraud verdict and its later explanation, for example), the client must handle that stickiness itself by calling the version-specific deployed_model_id explicitly.

The service call, and its shape

The client calls the endpoint with the SDK or a plain HTTPS request:

prediction = endpoint.predict(
    instances=[
        {
            "amount": 149.90,
            "merchant_category": "electronics",
            "device_type": "web",
            "hour_of_day": 22,
            "days_since_signup": 3,
            "amount_vs_avg": 4.7,
        }
    ]
)

print(prediction.predictions)          # list of scores
print(prediction.deployed_model_id)    # which version served this call

Note deployed_model_id on the response: it tells the caller which version answered, which is essential when comparing challenger metrics against champion metrics in downstream analysis (module 8's batch job reads this field).

Autoscaling: what the two knobs really do

Vertex autoscaling adds and removes replicas based on CPU utilisation, with a target around 60%. Two knobs frame the behaviour:

• min_replica_count — replicas that are always running, paid for continuously. This determines the baseline monthly cost and the response to a sudden traffic spike (only the min replicas can absorb the first 60 seconds).
• max_replica_count — the ceiling. When traffic exceeds max × per-replica capacity, requests queue and latency rises.

For the fraud endpoint, the traffic pattern is: baseline of 20 QPS during the day, evening peak of 200 QPS around Friday-Sunday 8 p.m. UTC. min=2, max=8 covers both without paying for a 24/7 idle fleet.

Autoscaling has a cool-down of a few minutes: adding replicas takes 30–60 seconds; removing them waits about five minutes to avoid flapping. A pure spike shorter than that will not scale up in time — the answer is min_replica_count, not max.

Latency budgets and how to hit them

Latency on an endpoint is dominated by three things:

• Model inference time — for XGBoost tabular it is around 1–2 ms; for a large transformer it is 100–500 ms.
• Serving container overhead — usually 2–5 ms for HTTP handling and payload parsing.
• Cross-region network — a client in europe-west3 calling an endpoint in us-central1 pays 100 ms just for the round trip. Deploy the endpoint in the same region as the caller.

For the fraud microservice, the SLO is p99 under 100 ms. That budget is met on n1-standard-2 at the current QPS, and the observability of the next section confirms it.

Request logging: the raw material for module 8

Setting enable_access_logging=True writes one Cloud Logging entry per request with the payload, the response, latency and the deployed_model_id. That log is BigQuery-friendly with a sink:

gcloud logging sinks create fraud-endpoint-logs \
    bigquery.googleapis.com/projects/fraud-detection-dev/datasets/monitoring \
    --log-filter='resource.type="aiplatform.googleapis.com/Endpoint"
                  AND resource.labels.endpoint_id="…"'

Once the sink is set, the endpoint's per-request data is queryable in BigQuery within minutes of the call. Module 8 reads this table to compare v5 vs v7 scores on the same population — that comparison is the actual A/B test the 90/10 split enables.

Log payloads and PII

The full request payload lands in Cloud Logging by default. For fraud detection the payload contains merchant IDs and amounts, which are not PII on their own; a payment amount tied to a customer email would be. Configure predictRequestResponseLoggingConfig with a sampling rate and a schema exclusion for any field that carries direct identifiers.

Rolling forward and rolling back

Promoting v7 from 10 % to 100 % is one call once the metrics justify it:

endpoint.update(traffic_split={"fraud-xgb-v5": 0, "fraud-xgb-v7": 100})

Setting v5 to 0 % leaves the deployed model warm, so a rollback is another one-line call ({"fraud-xgb-v5": 100, "fraud-xgb-v7": 0}) that takes effect in seconds, not minutes. Only after a rollback window (typically a week) do you actually undeploy v5:

endpoint.undeploy(deployed_model_id="fraud-xgb-v5")

The undeploy call frees the replicas and stops billing for them.

In summary

• A Vertex Endpoint fronts one or more DeployedModel entries with a traffic split; consumers see one URL, and rollouts become a config change instead of a client change.
• Always deploy with min_replica_count=2 for zero-downtime updates, size machines for CPU-bound tabular models modestly, and keep the endpoint in the same region as the caller.
• The traffic_split must cover every deployed model and sum to 100; the response carries deployed_model_id, which downstream comparisons rely on.
• Enable access logging, sink it to BigQuery, and reserve the full-payload logging for cases that are not PII — that is the raw material for the batch comparison of module 8.

Next module: the offline sibling of the endpoint — batch prediction on the full BigQuery table, and when it is the right choice over an online call.