Lesson 3 — Controlling cost
Cloud AI bills surprise people in a consistent pattern: the thing they budgeted for is not the thing that cost money.
Where the money actually goes
Idle accelerators. A notebook instance with a GPU attached, left running over a weekend, costs the same as one training a model. Multiply by a team of five who each forgot one, and this becomes the largest line on many bills. It is also entirely avoidable.
Serving capacity you are not using. A real-time endpoint provisioned for peak traffic is paid for around the clock. Traffic that peaks at ten times its average means paying roughly ten times what the work requires.
Storage that accumulates. Every dataset version, every checkpoint, every intermediate artefact. Individually trivial, and nobody deletes anything, so it compounds monthly forever.
Data egress. Moving data out of a provider's network is charged per gigabyte. Cross-region and cross-cloud architectures pay this continuously, and it rarely appears in the original design discussion.
Managed service premiums. Platform endpoints and managed notebooks cost more than the equivalent raw instance. Often worth it, and worth knowing the multiple.
Per-token inference at volume. Cheap per request, and a feature used by a hundred thousand users daily is a substantial recurring cost. Estimate volume before launch, not after.
Training compute — the thing everyone budgets for — is frequently among the smaller lines, especially when you fine-tune rather than train from scratch.
Reducing training cost
Use spot or preemptible capacity. Interruptible instances typically cost sixty to ninety percent less. The provider can reclaim them with little warning, which is entirely manageable if your training checkpoints regularly and can resume — and checkpointing is good practice regardless. This is the single largest lever available for training.
Use managed training jobs rather than notebooks. The job provisions, runs, writes the model and shuts down. No forgotten instances, ever.
Right-size before scaling up. Profile a short run. A great deal of training is bottlenecked on data loading rather than the accelerator, in which case a bigger GPU changes nothing but the price. Check utilisation before upgrading.
Start small. Prove the pipeline on a sample and a small model, then scale. Discovering a bug after eight hours on eight accelerators is an expensive way to learn.
Use mixed precision. Lower-precision arithmetic is faster and uses less memory, with negligible accuracy impact for most training. Frequently a straight speedup for one configuration flag.
Fine-tune instead of training from scratch. Two orders of magnitude cheaper, and usually better. Parameter-efficient methods reduce it further.
Reducing serving cost
| Lever | Typical effect |
|---|---|
| Batch instead of real time | Removes idle capacity entirely |
| Smaller or distilled model | Often ten times cheaper, small accuracy loss |
| Quantisation | Three to four times smaller and faster |
| Caching repeated requests | Depends on repetition, frequently large |
| Autoscaling with a sensible floor | Pays for peak only during peak |
| Serverless inference | Scales to zero; adds cold-start latency |
| Batching concurrent requests | Better accelerator utilisation per request |
| CPU instead of GPU for small models | Frequently sufficient and much cheaper |
The last row deserves emphasis. Many models do not need an accelerator to serve. A modest classifier or a small transformer runs perfectly well on CPU at production latency, and the reflex to attach a GPU costs money for nothing.
Serverless inference deserves its own note: you pay per request and capacity scales to zero, which is ideal for low or spiky traffic. The cost is a cold start when a request arrives after quiet — hundreds of milliseconds to seconds, depending on model size — so it suits background and internal workloads better than a user waiting for a response.
Controls that prevent surprises
Budget alerts at fifty, eighty and a hundred percent of expectation. This is five minutes of setup and the highest-return action in this lesson.
Tagging so cost can be attributed. Untagged spend is unmanageable spend, because nobody can be asked about it.
Automatic shutdown of idle notebook instances after a couple of hours. Every platform supports it and it is off by default.
Storage lifecycle rules moving old artefacts to cold storage and deleting them after a retention period you have actually agreed.
Quotas per team or project, so a mistake is bounded. Someone will eventually launch eight of the largest available instances by accident.
Estimating before you commit
A rough model, which is enough to catch the order-of-magnitude errors that matter:
Training: hours × instance rate × number of runs. Then multiply by three, because you will run it more times than you plan.
Real-time serving: instances needed at peak × 730 hours × rate. Not average traffic — peak, unless autoscaling is configured and tested.
Batch serving: records × time per record ÷ parallelism × rate. Usually a pleasant surprise.
Per-token inference: tokens per request × requests per day × 30 × rate. Include input and output separately, since output typically costs several times more.
Storage: gigabytes × monthly rate × 12, and assume it grows.
Egress: gigabytes leaving the provider × rate. Zero if your architecture keeps data in one place, which is a reason to keep it there.
Forgetting an accelerator instance running, and provisioning real-time serving where batch would have worked. Between them they account for the majority of cloud AI bills that get escalated to a finance conversation, and both are prevented by decisions taken in an afternoon.
In three sentences
Cloud AI bills are dominated not by training compute but by idle accelerators, serving capacity provisioned for peak and paid for around the clock, accumulated storage that nobody deletes, and data egress in cross-region architectures. For training, interruptible spot capacity at sixty to ninety percent less is the largest single lever, followed by using managed jobs that shut themselves down, profiling before scaling up because many runs are bottlenecked on data loading rather than the accelerator, and fine-tuning rather than training from scratch. For serving, moving to batch removes idle cost entirely, a smaller or quantised model is often ten times cheaper, and many models serve perfectly well on CPU — while the controls that prevent surprises are budget alerts, resource tagging, automatic notebook shutdown, storage lifecycle rules and per-team quotas, all of which take an afternoon to set up.