Module 1 — Constraints specific to on-device inference

A cloud-served model has almost no ceiling: memory, cores, storage, network — you buy whatever you need. A model that runs on a phone has to fit inside a device someone paid 200 USD for two years ago, run without heating the case, and answer before the user has time to notice it is thinking. The course opens here because every optimisation you will meet later — quantization, pruning, delegates — is a response to one of the constraints laid out in this module.

The four numbers that matter

Before you convert a single model, write four numbers on a sticky note.

Constraint	Order of magnitude on a mid-range phone	Budget for our running example
Model file size on disk	tens of MB fit, hundreds do not	under 8 MB
Peak RAM during inference	100 to 300 MB shared with the app	under 100 MB
Latency for a single image	100 ms feels instant, 500 ms feels slow	under 60 ms on a mid-range device
Energy per inference	tens of millijoules is unnoticeable, hundreds is not	under 40 mJ

These are the constraints that the plant leaf disease classifier — our running example — has to satisfy on a phone owned by a farmer who bought it four years ago. Everything the course teaches is aimed at hitting those four numbers.

Memory: two distinct budgets

The first mistake is to treat "memory" as one number. There are two, and they behave differently.

The file on disk is what the user downloads and what sits in the app's asset directory. It counts against the Play Store install size, and users are ruthless: a 200 MB install is uninstalled routinely for a feature that a 20 MB competitor also offers. This is the number the CI pipeline can enforce.

The RAM peak during inference is what the interpreter allocates when it processes one image: input tensors, activations at every layer, output tensor. A MobileNetV2 in float32 costs around 30 MB of activations for a single 224x224 image, on top of the 14 MB of weights. On a phone that is already running the app UI and the camera pipeline, that peak can push the app past the OS's memory limit and get it killed silently — no crash log, just an app that "closes on its own", which for a user is worse than a crash.

Compute: cores that are not equal

A recent phone has six to eight CPU cores, but they are not identical. Two or four "performance" cores handle short bursts at high frequency, four "efficiency" cores handle sustained load at low frequency. Running inference on efficiency cores costs less energy per inference but is markedly slower; running on performance cores is fast but throttles after a few seconds because the phone heats up. The interpreter chooses by default, and that choice is rarely optimal for your case — module 6 shows how to pick.

The GPU is another option, roughly two to five times faster than the CPU on convolutional networks — but it costs about the same energy per inference and its warm-up (compiling shaders, allocating buffers) can take 100 to 300 ms. For a one-shot inference every few seconds, warm-up dominates and CPU wins.

Energy: the constraint users never articulate

No user files a bug report saying "your app drains my battery". They uninstall. And because energy is not visible in a stack trace, it is the constraint most often forgotten in code review.

A rough order of magnitude: a single inference on a MobileNetV2 costs 30 to 80 millijoules on a phone. A phone battery holds roughly 40 000 joules. If your app runs one inference per photo — say a hundred photos a day — the direct cost is negligible. If it runs continuous detection at 15 FPS — a live camera classifier — you burn several joules per second, and the battery is visibly warm after ten minutes. That regime demands quantization, delegates and frame skipping, not just a "better model".

Offline operation is not a fallback, it is the point

For our farmer running the plant disease classifier, network connectivity is intermittent at best. On-device is not a graceful degradation of a cloud-served model — it is the only mode that ever runs. That decision has consequences that ripple through the whole course.

You cannot A/B test. You cannot roll back a bad model without waiting for the app update to reach every user. You cannot patch a preprocessing bug in production; the version installed today is the one that classifies tomorrow's photos. Model versioning has to be inside the app, updates have to be shipped through the store or through an in-app download, and every release has to be tested on real devices because there is no server-side monitoring to catch the drift.

Perceived latency is not the same as measured latency

The user does not experience the model's runtime, they experience the interval between tapping the shutter and seeing a result. That interval includes camera capture, YUV-to-RGB conversion, resizing, model inference, post-processing, and finally screen rendering. Model inference is often 20 to 40 percent of the total; the rest is preprocessing and the app's own overhead.

That is why module 9 measures end-to-end percentiles, not just the interpreter's invoke call. And why module 7 pushes preprocessing onto a background thread: a 30 ms inference feels 300 ms if it blocks the UI, because the whole animation freezes.

Privacy is a feature, not a side effect

Running inference on the device means the raw input — an image of a plant, a voice sample, a health measurement — never leaves the phone. For a farmer photographing his own crop, that matters less than for a health app, but the two share a code path: no image upload means no server-side leak, no logging pipeline to secure, no cross-border data question to answer. This is why several regulated verticals (medical imaging, banking KYC) actively prefer on-device even when the cloud would be cheaper.

Key takeaways

• Four constraints govern every later choice: file size on disk, peak RAM during inference, single-image latency, and energy per inference — write down a target number for each before you start.
• Memory has two faces: the download size the CI can enforce, and the RAM peak that the OS will silently kill the app for.
• Compute is not one number either: performance cores are fast but throttle, efficiency cores are slow but sustainable, and the GPU wins only when inferences follow one another closely enough to amortise warm-up.
• On-device is the whole app, not a fallback: no A/B testing, no server-side rollback, and privacy comes for free because raw inputs never leave the phone.

Next module: converting a Keras model to the .tflite format, and the operators that survive the trip.