Module 6 — Interpreter and hardware delegates

The .tflite file we spent five modules building is inert. Something has to load it, hand it a tensor, and read a tensor back. That something is the interpreter, and the choice of what runs the graph — CPU cores, GPU, NPU, dedicated accelerator — is made by delegates. This module explains the lifecycle in Python, because the API is the same on Android and iOS, and it introduces the four delegates that matter on real hardware.

The interpreter, step by step

Every framework — the Task Library on Android, the TensorFlowLite pod on iOS — is a thin wrapper around the same C++ interpreter. The pattern is identical everywhere:

import numpy as np
import tensorflow as tf

# 1. Load the model.
interpreter = tf.lite.Interpreter(model_path="leaf_classifier.tflite")

# 2. Allocate the tensors. This is where memory is actually claimed.
interpreter.allocate_tensors()

# 3. Discover the input and output tensor indices.
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# 4. Prepare an input in the right shape and dtype.
image = np.zeros((1, 224, 224, 3), dtype=np.int8)

# 5. Set the input, run, read the output.
interpreter.set_tensor(input_details[0]["index"], image)
interpreter.invoke()
predictions = interpreter.get_tensor(output_details[0]["index"])

print(predictions.shape)   # (1, 38)

The five steps decompose the lifecycle:

• Load parses the file and builds an in-memory representation of the graph.
• Allocate actually reserves the memory for weights and activations. This is where the peak RAM number from module 1 gets set. On a low-memory device, this is where allocation can fail; catching the exception here is much better than crashing on invoke.
• Discover returns metadata: names, shapes, dtypes, quantization parameters. Real apps cache this once at startup, not every inference.
• Set input copies data into the input tensor. set_tensor does one copy; on hot paths, using get_input_tensor and writing directly into the buffer avoids that copy.
• Invoke runs the graph. This is the only expensive step. Read output copies from the output tensor to a NumPy array; the same buffer trick applies.

Delegates: routing the graph to specialised hardware

By default the interpreter runs on CPU. A delegate is a plugin that takes ownership of part of the graph and runs it somewhere else — GPU, NPU, DSP, dedicated ML accelerator. The interpreter partitions the graph automatically: operators the delegate supports go to it, the rest stays on CPU. That is the CPU fallback, and it explains why an unsupported op does not break the model, only slows it down.

Four delegates matter in production.

GPU delegate

Available on Android (OpenCL, OpenGL ES) and on iOS (Metal). It shines on convolutional networks — MobileNet, EfficientNet, U-Net — where the arithmetic dominates and can be parallelised across many threads.

# Android GPU delegate; on iOS, use CoreMLDelegate or the Metal path.
delegate = tf.lite.experimental.load_delegate("libtensorflowlite_gpu_delegate.so")
interpreter = tf.lite.Interpreter(
    model_path="leaf_classifier.tflite",
    experimental_delegates=[delegate],
)

Two things to know before you enable it.

Warm-up is expensive: 100 to 300 ms on the first invoke, because the delegate has to compile shaders and allocate buffers. For a one-shot classification per photo, that is the whole latency budget spent on setup. For a live camera feed at 15 FPS, warm-up is amortised over hundreds of inferences and the average latency drops.

Not every op is supported. If a model contains a non-standard op, the delegate hands only the standard sub-graph to the GPU and leaves the rest on CPU. Every GPU-to-CPU boundary costs a memory transfer, and enough boundaries can make GPU inference slower than pure CPU. Always measure before assuming the GPU wins.

NNAPI delegate

NNAPI is Android's neural network abstraction layer, present on API level 27 and above. It routes to whatever accelerator the manufacturer exposes: dedicated NPU on Pixel, Hexagon DSP on Qualcomm, MediaTek APU, or falls back to GPU or CPU.

delegate = tf.lite.experimental.load_delegate("libnnapi_delegate.so")
interpreter = tf.lite.Interpreter(
    model_path="leaf_classifier.tflite",
    experimental_delegates=[delegate],
)

The good: on phones with a modern NPU, NNAPI beats both CPU and GPU on latency and energy, often by a factor of 2 to 5.

The bad: the abstraction is leaky. Same phone, different driver version, different runtime characteristics. Some ops are supported on paper and buggy in practice, producing wrong results silently. Every release cycle needs a regression check on the top three or four phones your users actually own.

Rule of thumb: NNAPI on Android 10 and above with int8 models is worth trying. Below Android 10, the driver landscape is too messy to be worth the maintenance.

Core ML delegate (iOS)

Apple's counterpart to NNAPI. On an iPhone with a Neural Engine (A11 and above), it delivers dramatic speed-ups and outstanding energy efficiency.

// iOS, Swift
let coreMLDelegate = CoreMLDelegate()
guard let interpreter = try? Interpreter(
    modelPath: "leaf_classifier.tflite",
    delegates: [coreMLDelegate]
) else { return }

The Core ML delegate quietly converts supported subgraphs into Core ML operations at runtime. Unsupported ops stay on the TFLite CPU or Metal path. On an iPhone 12 or newer, expect the Neural Engine to shave the leaf classifier's inference to under 10 ms.

XNNPACK: better CPU by default

XNNPACK is a CPU delegate. It ships enabled by default in recent TFLite versions, and it uses SIMD instructions (ARM NEON, AVX on desktop) and fused kernels to speed up float and int8 convolutions on the CPU itself.

The point of XNNPACK is that "CPU" is not a floor: the same phone, the same model, the same interpreter is 30 to 60 percent faster with XNNPACK than without. When the numbers in module 9 look surprisingly good on plain CPU, XNNPACK is usually why.

Choosing a delegate: the decision table

Situation	First choice	Fallback
Single inference per user action	CPU + XNNPACK	(no delegate warm-up wasted)
Live camera (10 FPS or more)	GPU	XNNPACK on unsupported devices
Highest speed on Android 10+	NNAPI (int8)	GPU
Highest speed on iOS (A11+)	Core ML	Metal / XNNPACK
Wide device coverage, low variance	XNNPACK only	—

The fallback column matters. A delegate that fails to load — driver missing, device too old, model contains unsupported ops — must not crash the app. The correct pattern is: try the delegate, catch the error, log it, run the interpreter without the delegate. The user sees a slower inference; they do not see a crash.

Threading and cores

The CPU delegate exposes a thread count:

interpreter = tf.lite.Interpreter(
    model_path="leaf_classifier.tflite",
    num_threads=4,
)

Set it explicitly. The default is one thread, which leaves performance on the table. Four threads is usually optimal on a mid-range phone with four performance cores; eight is often worse because efficiency cores drag the average down.

A wrong delegate is worse than no delegate

Enabling the GPU on a model with unsupported ops can result in latency higher than the plain CPU baseline, because of the memory transfers between CPU and GPU. Enabling NNAPI on a buggy driver can result in wrong predictions with no error at all. The rule from module 9 applies: measure on a real device, do not guess.

Key takeaways

• The interpreter lifecycle is load, allocate, discover, set, invoke, read; catch memory errors at allocate_tensors, cache tensor metadata once at startup.
• A delegate takes part of the graph and runs it elsewhere; unsupported operators fall back to CPU automatically, and every boundary costs a memory transfer.
• GPU wins on convolutional workloads with sustained inferences, NNAPI on modern Android with int8, Core ML on iPhone with a Neural Engine, XNNPACK everywhere else — and it is the reason plain CPU is often faster than it looks.
• Always measure on the target device and always catch delegate failures to fall back gracefully; a slow inference beats a crash every day of the week.

Next module: integrating the model in a real Android app, including camera preprocessing and the Task Library.