Lesson 3 — Deployment patterns
Two decisions here. How predictions reach the thing that consumes them, and how you release a new model without discovering its problems in front of users.
How predictions get delivered
Batch prediction
Run the model on a schedule over many records and write the results somewhere — a table, a file, a message queue. Whatever consumes them reads the stored result.
| Suits | Churn scores, risk ratings, recommendations, nightly reports |
| Advantages | Simplest by a wide margin; no latency requirement; failures are retryable; cheap |
| Constraints | Predictions are as old as the last run; you compute for records nobody asks about |
This is the right answer far more often than teams assume. If a prediction from this morning is acceptable, a scheduled job writing to a table is a complete solution and needs no serving infrastructure, no autoscaling and no on-call rotation.
Real-time serving
Wrap the model in a service. A request arrives, the model predicts, the response returns in milliseconds.
| Suits | Fraud checks during payment, search ranking, anything reacting to a user action |
| Advantages | Predictions reflect the current moment, including data that did not exist an hour ago |
| Constraints | Latency budget, availability requirement, capacity you pay for while idle, feature computation must also be fast |
The hidden difficulty is rarely the model. It is the features: if a prediction needs a customer's activity over the last thirty days, that has to be retrievable in milliseconds, which is a data engineering problem rather than a modelling one and is where most real-time projects actually stall.
Streaming
The model consumes a stream of events and emits predictions continuously. Sits between the two others, suits sensor data, transaction streams and log analysis, and adds the operational weight of a streaming platform.
Edge deployment
The model runs on the device: phone, camera, vehicle, industrial sensor.
| Suits | No reliable connectivity, privacy requirements, latency below a network round trip, high volume where bandwidth costs |
| Advantages | Works offline; data never leaves the device; no per-request server cost |
| Constraints | Model must be small; updating deployed devices is a genuine problem; you cannot easily observe what is happening |
The last constraint is underrated. Monitoring a model on ten thousand devices you do not control is substantially harder than monitoring one service, and it needs designing in from the start rather than added later.
How much infrastructure do you need?
Genuinely less than the ecosystem suggests. In rough order of complexity:
- A scheduled script. Reads data, loads the model, writes predictions. Runs on one machine under cron. Serves a large number of real production use cases.
- A small web service. The model behind an HTTP endpoint, containerised, one or two instances. Handles moderate traffic without difficulty.
- A managed serving platform. The cloud providers' model endpoints handle scaling, versioning and traffic splitting for you.
- Your own orchestration. Kubernetes and a serving framework. Justified by scale, by many models, or by requirements the managed options do not meet.
Starting at level three or four for a single model is a recognisable way to spend a quarter building infrastructure rather than value. Start at the lowest level that meets the requirement and move up when you feel the constraint.
Releasing safely
A new model version is not a code change with a test suite that proves it correct. Its behaviour is statistical, and offline metrics only partly predict its live effect. So release progressively.
Shadow deployment. The new model receives real traffic and its predictions are logged but not used. You compare against the current model on live data at no risk. This is the highest-value pattern in the list and the most frequently skipped, because it requires no decision — you simply observe.
Canary release. Route a small share of traffic, perhaps one to five percent, to the new model. Watch metrics and errors. Increase gradually. Roll back at the first sign of trouble, which is easy because most traffic never moved.
A/B test. Split traffic deliberately and measure a business outcome rather than a model metric. This is the only way to learn whether a more accurate model actually produces a better result — and the two diverge more often than intuition allows, because a model optimising a proxy can improve the proxy while worsening the thing you care about.
Blue-green. Two full environments, switch traffic between them. Simple rollback, no gradual exposure, and it does not tell you anything before the switch.
| Pattern | Risk | Tells you about |
|---|---|---|
| Shadow | None | Prediction differences on live data |
| Canary | Low | Errors, latency, obvious regressions |
| A/B test | Controlled | Actual business impact |
| Blue-green | Moderate | Nothing until you switch |
A reasonable sequence: shadow to confirm the new model behaves sensibly on live traffic, canary to confirm it operates correctly, then an A/B test if the business outcome is measurable and worth the traffic.
What to check before any release
- The feature computation path is identical to training, verified by test
- Input validation rejects malformed or out-of-range values rather than predicting on them
- A fallback exists for when the model is unavailable — a previous version, a rule, or a graceful refusal
- Predictions are logged with their inputs and the model version, which is what makes later investigation possible
- Latency is measured at the ninety-fifth and ninety-ninth percentiles rather than the mean
- Rollback has been tested, not merely documented
A model deployed without input validation will happily predict on nonsense — a negative age, a missing category encoded as zero, a currency field arriving in the wrong unit — and return confident numbers that flow into downstream decisions. Validate inputs at the boundary and reject rather than guess. This is cheaper than any monitoring that would eventually catch it.
In three sentences
Batch prediction on a schedule is the right pattern far more often than teams assume, and real-time serving's real difficulty is usually computing features fast enough rather than running the model, while edge deployment trades observability and updatability for offline operation and privacy. Infrastructure should start at the lowest level that meets the requirement — a scheduled script or a small containerised service handles a large share of production workloads, and adopting orchestration for a single model is a recognisable way to spend a quarter building nothing of value. Release progressively because a model's behaviour is statistical rather than testable: shadow deployment gives you a live comparison at zero risk and is the most frequently skipped high-value step, canary releases catch operational problems cheaply, and only an A/B test tells you whether a more accurate model actually improves the outcome you care about.
Next — Lesson 4: monitoring →