Module 9 — Measuring latency and power consumption
Modules 3 to 8 promised specific numbers: 2.35 MB, 22 ms, 0.939 accuracy, and so on. Every one of those numbers came from a measurement. This module is about how you actually get them, why the naive "start timer, run, stop timer" is wrong, and what to publish when the tech lead asks "how fast is it?".
The benchmark tool: what to run
TensorFlow Lite ships a benchmark tool — a small native binary you push to the device and run against a .tflite file. It handles all the traps the naive approach ignores: warm-up, multiple runs, percentile reporting, delegate selection, thread control.
Download the prebuilt binary (android_aarch64_benchmark_model) for your architecture and push it to the device:
adb push android_aarch64_benchmark_model /data/local/tmp/
adb push leaf_classifier.tflite /data/local/tmp/
adb shell chmod +x /data/local/tmp/android_aarch64_benchmark_model
adb shell /data/local/tmp/android_aarch64_benchmark_model \
--graph=/data/local/tmp/leaf_classifier.tflite \
--num_threads=4 \
--warmup_runs=50 \
--num_runs=200 \
--enable_op_profiling=true
The output ends with something like:
Inference timings in us: Init: 12345, First inference: 34012,
Warmup (avg): 22450, Inference (avg): 22125,
Std deviation: 512, Percentile: 21980, 22120, 22345 (50th, 90th, 95th)
Five numbers, each telling a different story.
Warm-up: why the first inference lies
The first inference is often 20 to 100 times slower than the steady state. Memory allocations happen on the first invoke, the CPU governor scales up frequency in response to the workload, the GPU delegate compiles shaders, the JIT warms up. If you report that first number, you are reporting an artefact of cold start, not the real behaviour.
The benchmark tool runs 50 warm-up inferences before it starts timing. Do the same in any measurement code you write yourself. In an app, the "warm-up" cost happens once at startup and is amortised over every subsequent inference; a benchmark that includes it in the reported latency describes a load-once workload that most apps do not have.
Percentiles beat averages
The average latency is misleading because the distribution is not symmetric. On a phone, latency has a long right tail: most inferences finish in 22 ms, a handful take 40 ms because the CPU governor throttled, one takes 200 ms because the app was suspended for garbage collection.
Report the median (P50), the 90th percentile (P90) and the 95th (P95). These tell the real story: "half of inferences complete in under 22 ms, 90 percent in under 24, 95 percent in under 27". A user perceives the tail — the one frame in ten that stalls — more than the median. Publishing only the average hides the tail entirely.
| Statistic | What it says | What it hides |
|---|---|---|
| Average | Convenient number | Sensitivity to outliers, tail shape |
| P50 | Typical case | Behaviour under load |
| P90 | What one in ten users sees | Rare pathology |
| P95 | Complaint threshold | Absolute worst |
| Max | The horror story | Everything else |
Energy: harder to measure, more honest to report
There is no clean "millijoules per inference" API on a phone. But there are three practical proxies.
Battery Historian (Android) collects battery discharge over time. Run your app in a controlled loop (200 inferences per second for ten minutes), read the discharge, divide. The unit is milliampere-hours per thousand inferences; multiply by the battery voltage (about 3.85 V) to get millijoules.
Instruments (iOS) has an Energy Log template that reports app-level power draw over time. Same protocol: a controlled loop, subtract the idle baseline, divide by the number of inferences.
Thermal throttling as a proxy: after a few minutes of continuous inference, does the CPU frequency drop? If it does, the workload is thermally unsustainable and the app will regress over long sessions. This is not an energy measurement, but it flags the same underlying problem — the model consumes more power than the phone can dissipate.
The numbers below come from Battery Historian and are approximate. Report them with an explicit uncertainty; the reader who takes an "energy per inference" number as gospel is going to be disappointed.
Three phones, one model
The purpose of measuring on three phones — a flagship, a mid-range, an entry-level — is that a model that is fine on the flagship can be unusable on the entry-level, and shipping the same app to both tiers demands that both work.
| Phone tier | Example (2024) | Latency P50 (CPU) | Latency P50 (GPU) | Energy per inference |
|---|---|---|---|---|
| Flagship | Pixel 8 Pro | 12 ms | 6 ms (NNAPI: 4 ms) | 22 mJ |
| Mid-range | Pixel 4a | 22 ms | 18 ms | 38 mJ |
| Entry-level | Redmi 9 | 61 ms | 55 ms | 89 mJ |
Three lessons.
First, the ratio between flagship and entry-level is 5x on latency and 4x on energy. The mid-range is closer to the flagship than to the entry-level — the market is skewed toward the top. But entry-level is still a large fraction of installs, and 61 ms P50 is at the edge of "feels instant".
Second, GPU wins on the flagship (mature drivers, Neural Engine or NPU) and barely wins on entry-level (weak GPU, high driver overhead). On the entry-level, CPU + XNNPACK is often the sweet spot.
Third, the entry-level's energy per inference is 4x the flagship's. That is not a small number — it means a live camera classifier that a flagship user runs for an hour costs about 12 percent of battery; the entry-level user pays 40 percent for the same session. That is unshippable in this form; frame skipping, a smaller model or an off-by-default toggle become mandatory.
In-app measurement
The benchmark tool tells you about the model. It does not tell you about the app. The user-perceived latency includes camera capture, YUV conversion, resize, inference, post-processing, UI update. Measure that too, inside the app:
val start = SystemClock.elapsedRealtimeNanos()
val bitmap = imageProxy.toBitmap()
val results = classifier.classify(TensorImage.fromBitmap(bitmap))
val end = SystemClock.elapsedRealtimeNanos()
telemetry.record("end_to_end_ms", (end - start) / 1_000_000)
Collect these numbers in a rolling histogram; ship them in an internal build first, then anonymously in production once the numbers stabilise. A user's real experience of "how fast is it" is this end-to-end number, and it is nearly always double the interpreter's invoke time.
The trade-off matrix from modules 2 to 5 is not "publish once and forget". Every retrained model needs the whole matrix redone. A pipeline that runs the benchmark tool on three reference devices after every model artifact is the smallest thing that prevents "why did the latency get worse?" from becoming a two-day investigation.
The complete trade-off matrix
| Variant | Size (raw) | Size (gzip) | P50 CPU | P50 GPU | Accuracy |
|---|---|---|---|---|---|
| Float32 baseline | 9.20 MB | 8.10 MB | 88 ms | 42 ms | 0.941 |
| Dynamic range int8 | 2.40 MB | 2.25 MB | 65 ms | — | 0.940 |
| Full-integer int8 (PTQ) | 2.35 MB | 2.20 MB | 22 ms | 24 ms | 0.928 |
| Full-integer int8 (QAT) | 2.35 MB | 2.20 MB | 22 ms | 24 ms | 0.939 |
| QAT + 60% prune | 2.35 MB | 1.30 MB | 22 ms | 24 ms | 0.933 |
| QAT + prune + cluster | 2.35 MB | 0.95 MB | 22 ms | 24 ms | 0.930 |
Latency numbers are from the Pixel 4a mid-range reference. This is the table the mobile team, the training team and the product team all read from, and it is the direct input to the decision in module 10.
Key takeaways
- Use the benchmark tool: it handles warm-up, multiple runs, percentiles and delegate selection out of the box; do not roll your own timing loop.
- Report P50, P90, P95, not averages: the tail is what the user complains about, and the average hides it.
- Energy is measured with Battery Historian on Android, Instruments on iOS, and thermal throttling as a proxy for "will this stay usable over a long session".
- Run on three phone tiers: the ratio between flagship and entry-level is 4x to 5x, and the same model is often shippable on one tier and not the other — that fact belongs in the decision, not in a footnote.
Next module: assembling everything into a complete on-device image classifier, choosing the final variant and defending the trade-offs.