Module 10 — Load testing and sizing

The container from module 9 runs, the probes from module 8 answer, and the token from module 7 gates access. What nobody can tell yet — not the on-call engineer, not the finance team, not the dashboard product manager — is how much traffic one replica of this service actually holds and how much a target load costs per month. This module answers both, with a repeatable Locust scenario, a small vocabulary for reading the results, and a decision procedure for the number of workers per container and the number of container replicas behind the load balancer.

The load target, written down first

A load test with no target is a benchmark with no verdict. Before writing a single line of Locust, write three numbers down. They come from the product, not from the engineering team.

• Peak QPS: the busiest expected requests per second, across all callers. For the churn service the Streamlit dashboard from course 38 shows a mid-morning peak around 40 QPS, and the nightly batch job from course 20 fires ten batches per second for twelve minutes. Peak is 40 QPS single-row plus 10 QPS batch-of-200.
• Latency budget: the value above which the dashboard becomes unpleasant. The product owner set p95 under 250 ms for the single-row route and p95 under 2 s for the batch route.
• Error budget: the worst tolerable failure rate. Anything above 1 % of 5xx during peak is treated as an incident.

Every decision that follows is measured against these three numbers. Nothing else counts.

The Locust scenario

Locust is a Python load testing tool that describes user behaviour as classes and tasks. One file is enough for the churn service:

# tests/load/locustfile.py
import json
import random
import uuid

from locust import HttpUser, task, between


def _one_subscriber() -> dict:
    return {
        "subscriber_id": f"sub-{uuid.uuid4().hex[:12]}",
        "tenure_months": random.randint(1, 72),
        "monthly_charges": round(random.uniform(20, 120), 2),
        "total_charges": round(random.uniform(20, 8000), 2),
        "contract_type": random.choice(["month_to_month", "one_year", "two_year"]),
        "payment_method": random.choice(["credit_card", "bank_transfer", "check"]),
        "is_senior": random.random() < 0.15,
    }


class ChurnUser(HttpUser):
    # Between-task pause. Combined with the number of users, sets the QPS.
    wait_time = between(0.5, 1.5)

    def on_start(self) -> None:
        # The key is passed on the command line to keep it out of the repo.
        self.key = self.environment.parsed_options.api_key

    @task(9)  # 90 percent single-row traffic
    def score_one(self) -> None:
        self.client.post(
            "/predict",
            headers={"X-API-Key": self.key, "content-type": "application/json"},
            data=json.dumps(_one_subscriber()),
            name="/predict",
        )

    @task(1)  # 10 percent batch traffic
    def score_batch(self) -> None:
        items = [_one_subscriber() for _ in range(200)]
        self.client.post(
            "/predict/batch",
            headers={"X-API-Key": self.key, "content-type": "application/json"},
            data=json.dumps({"items": items}),
            name="/predict/batch",
        )

The two @task weights encode the traffic mix from the target above: nine single-row requests for every batch of 200. Assigning explicit name= values keeps /predict grouped in the report even when the URL contains a query string. Reading the API key from --api-key on the command line avoids hard-coding it into a file that will end up in Git.

Register the option in a small conftest.py next to the file:

# tests/load/conftest.py
from locust import events


@events.init_command_line_parser.add_listener
def _(parser):
    parser.add_argument("--api-key", type=str, required=True, help="X-API-Key value")

Then run the scenario against the container from module 9, never against a raw uvicorn app.main:app on the laptop:

# terminal
docker run --rm -d -p 8080:8080 --name churn \
  -e MODEL_URI=models:/churn-classifier@production \
  -e API_KEY_STORE_URL=redis://host.docker.internal:6379/0 \
  churn-api:1.0.0

locust -f tests/load/locustfile.py \
       --host http://localhost:8080 \
       --users 50 --spawn-rate 5 --run-time 5m \
       --api-key dev-key-do-not-use

Fifty simulated users, ramping up at five per second, for five minutes. The Locust UI (or --headless output) then gives one row per named endpoint with the numbers that matter.

Reading the report

Four columns of the Locust report drive every decision.

• RPS: requests per second the client sent. Compare it to the target 50 QPS.
• Failures: rate of non-2xx responses. Above one percent, stop reading percentiles and look at the server logs from module 8 — the service is broken, not slow.
• p50, p95, p99: median, 95th and 99th percentile of latency in milliseconds. p50 tells the typical user experience; p95 tells the budget; p99 tells whether a small subset of users suffers.
• Median vs 99th gap: a p50 of 30 ms with a p99 of 3 000 ms means something intermittent is very slow — usually a garbage collection pause, a lock contention, or a cold cache. Investigate before adding replicas; a bigger fleet does not make a slow-tail request faster.

An acceptance run for this course looked like: RPS 50, failures 0.0 percent, p50 42 ms, p95 118 ms, p99 210 ms on the single-row route. Under the 250 ms budget, with margin. The batch route landed at p95 1.4 s for 200 rows, well under the 2 s budget.

Where a slow ML API actually spends its time

Before scaling out, know what you would be scaling out. Three bottlenecks show up over and over on ML services and the fix is different for each.

• The model itself. model.predict() on a scikit-learn GradientBoostingClassifier takes about 5 ms per row; a wide RandomForestClassifier can take 30 ms; a small PyTorch model on CPU can take 80 ms. If a single-row prediction spends 90 percent of its wall clock inside predict, no amount of Uvicorn tuning helps: switch to a faster model, quantize, or move to a GPU replica.
• Serialization. A 200-row batch with 40 float features serializes to about 90 KB of JSON. Pydantic v2 parses that in ~2 ms; Pydantic v1 was closer to 15 ms. On a batch route, JSON parse and dump can easily match the model call. orjson as the response class shaves another 30 percent off the dump.
• Thread pool saturation. Every sync route runs in Starlette's AnyIO thread pool, which defaults to 40 threads. A worker that receives more than 40 concurrent long calls queues the rest inside the process. The queue is invisible from outside; it shows up only as a p99 that climbs while p50 stays flat.

Add one profiling middleware for one minute, then remove it — the answer usually falls out on the first run.

Sizing workers and replicas

The sizing rule from module 9 (workers equals cores for CPU-bound scoring, twice cores for I/O-bound) is the starting point. Load testing replaces it with a measured number.

Run the scenario against one replica at increasing user counts (10, 25, 50, 100, 200) and record the highest user count at which p95 stays under budget. Call that number capacity_per_replica. Then the number of replicas needed for the target peak is:

• replicas = ceil(peak_qps / capacity_per_replica_qps) plus one for headroom during a rolling deploy.

For the churn service: one 2-vCPU replica held 50 QPS at p95 118 ms. Peak is 50 QPS. That is one replica, plus one for the rolling deploy, plus one more so a single pod failure never drops the fleet under peak — three replicas total. Autoscaling on CPU (HorizontalPodAutoscaler at 70 percent) then adds a fourth replica if traffic spikes above forecast.

Two rules that make the exercise honest:

• Load-test the whole path, including the auth dependency, the middleware chain and the response serialization. A benchmark that hits _score_one directly reports a number the real service will never reach.
• Test with a realistic mix. A 100 percent single-row load pretends batches do not exist and misses the coupling — one worker busy on a 2 s batch cannot answer a fast single-row request during that time.

Cost, in one back-of-the-envelope

The last number product and finance want is the monthly cost. On a managed container service that bills roughly USD 30 per 2-vCPU replica per month, three replicas plus one autoscale headroom room is:

• 3 base replicas × USD 30 = USD 90 per month.
• Autoscale bursts to 4 for 5 percent of the month: 0.05 × 30 = USD 1.5 per month.
• Egress and log volume: a rough USD 15 per month at this scale.

Round to USD 110 per month for the whole service, delivering the target 50 QPS at p95 under 250 ms, with the certificate-worthy resilience of a rolling deploy and a spare replica. That number goes into the pull request that ships the service.

A tiny p95 with a huge p99 is not a passing test

An RPS of 50 with p50 30 ms and p99 4 000 ms passes an "average" mindset and fails on the ground: one user in a hundred waits four seconds. The fix is upstream (a lock, a garbage collection pause, a cold cache), not one more replica. Read the gap before you read the average.

In summary

• Write the load target before the Locust file: peak QPS, latency p95 budget, error budget. Every decision is measured against these three.
• Run Locust with a realistic mix (single-row and batch, at the traffic weights the service will see) against the actual container from module 9, not a bare Uvicorn.
• Find the bottleneck first: model call, JSON serialization, or thread pool saturation. Fix the right layer; a bigger fleet never fixes a per-request problem.
• Size with replicas = ceil(peak_qps / capacity_per_replica) plus one for rolling deploys and one for failure headroom. Attach the monthly cost to the sizing decision so trade-offs are visible.

Next module: recap and exam — a full production checklist for a model API, and the 40-question assessment that closes the course.