Module 6 — Measuring latency and throughput

Modules 1 to 5 involved numbers — 380 ms, 88 %, 30 tok/s — that we asked you to believe. This module is about the discipline that produces them. In 2026, most public benchmarks for local LLMs are unreproducible: different hardware, warm caches, uncounted prompt tokens, single runs. Any decision that rides on those numbers is fragile. A reproducible protocol is the difference between a project that ships and a project that quietly regresses at the third hardware refresh.

The three numbers that matter

Speak three numbers, not one.

Time to first token (TTFT). Milliseconds from invoke to the first output token. This is what the user perceives as responsiveness. For an interactive assistant, keep it under 500 ms.

Tokens per second, output (TPS-out). How fast the tokens stream after the first one. This dominates the total time on longer generations. For our ticket summaries (around 60 tokens), even 15 tok/s stays under 5 seconds total.

Total time, wall clock. The one the user sees. Roughly TTFT + output_tokens / TPS-out, plus the network round-trip if any.

A model that has fast TTFT but slow TPS-out feels snappy on short answers and painful on long ones. A model with slow TTFT but fast TPS-out feels sluggish on chat but great on batch. The two numbers cannot be replaced by a single "latency" figure.

Why prompt length matters as much as output length

Small-model inference splits cleanly into two phases with very different profiles.

Prefill is the pass over the prompt tokens: highly parallel, dominated by matmul throughput, benefits massively from a GPU. For a 4k-token prompt on a laptop CPU it takes hundreds of milliseconds; on an entry GPU it is under 50 ms.

Decode is the token-by-token generation: sequential by nature, memory-bandwidth-limited. GPU helps but less dramatically. Roughly 20 tok/s on CPU, 60 tok/s on entry GPU for a Q4 3B model.

The consequence: TTFT scales with prompt length, TPS-out scales with hardware bandwidth. A benchmark that reports only "40 tok/s" without saying how long the prompt was is telling you almost nothing.

A reproducible measurement protocol

The protocol has five rules and a script.

Rule 1 — Fix the prompts. A set of 20 real inputs from your task, saved once, versioned in git. For the ticket assistant, 20 anonymised tickets covering the four categories.

Rule 2 — Fix the outputs. Set temperature=0 and max_tokens to a value that reflects the real task (60 for our summaries). Randomness in the outputs makes runs incomparable.

Rule 3 — Warm the cache. Run one throwaway inference before starting the timer. The first call loads weights from disk into RAM and would dominate the mean.

Rule 4 — Report percentiles. Mean is misleading for latencies; give p50 and p95. A CPU under thermal throttling has a long tail that a mean hides.

Rule 5 — Note the machine. CPU model, RAM, OS, GGUF quant, llama.cpp build hash. In three months this is what lets you tell whether "the model got slower" or "the hardware got busier".

The script:

# bench.py -- measure TTFT and TPS on GGUF via llama.cpp
import time, json, statistics
from llama_cpp import Llama

llm = Llama(model_path="qwen-ticket.Q4_K_M.gguf", n_ctx=2048, n_threads=8)
prompts = [json.loads(l)["text"] for l in open("bench_prompts.jsonl")]

# Warm-up
llm("hello", max_tokens=5, temperature=0)

ttft, tps = [], []
for p in prompts:
    t0 = time.perf_counter()
    first = None
    n = 0
    for chunk in llm(p, max_tokens=60, temperature=0, stream=True):
        n += 1
        if first is None:
            first = time.perf_counter()
    t1 = time.perf_counter()
    ttft.append((first - t0) * 1000)     # ms
    tps.append((n - 1) / (t1 - first))   # tok/s on decode phase only

def p(x, q): return sorted(x)[int(len(x)*q)-1]
print(f"TTFT p50 = {p(ttft,0.5):.0f} ms, p95 = {p(ttft,0.95):.0f} ms")
print(f"TPS  p50 = {p(tps,0.5):.1f}   , p95 = {p(tps,0.95):.1f}")

Readings on the target hardware

Running the protocol above on Qwen 2.5 3B Q4_K_M GGUF, 20 tickets, max_tokens=60:

Hardware	TTFT p50	TPS-out p50	Total time (60 tokens)
Laptop CPU (Intel i7-1355U, 16 GB)	210 ms	18 tok/s	~3.5 s
Desktop CPU (Ryzen 7 7700X, 32 GB)	90 ms	34 tok/s	~1.8 s
Entry GPU (RTX 4060, 8 GB VRAM)	40 ms	78 tok/s	~0.8 s
M2 Air, 16 GB (MLX Q4)	60 ms	42 tok/s	~1.5 s

The laptop CPU is the slowest, and it is fast enough. That is the number that matters for the deployment — most agents run on the laptop configuration, and 3.5 s for a ticket summary is well under the acceptable interactive budget.

Batch size: not our problem here, but know it exists

Batching several prompts together amortises the prefill cost across them and multiplies aggregate throughput dramatically. On a GPU, a batch of 8 prompts can go 5× faster than 8 sequential calls. For a single-user desktop assistant, this is irrelevant — there is one user, batch size is 1. But when a server-side deployment enters the picture (module 8 briefly discusses it for the team lead's setup), batching is the first knob to turn.

The trap of comparing readings from two protocols

The trap that costs teams the most: comparing your reading of "18 tok/s on the laptop" to a blog post's "42 tok/s on the same laptop". Different prompt length, different sampler, different threads, different build flags — the numbers are apples and pears. Any comparison worth reporting comes from your own script, on your own machines, with the same version tag. A number without a protocol is not data.

In summary

• Report three numbers: time to first token (TTFT), tokens per second on decode (TPS-out), and total wall-clock time.
• Prefill scales with prompt length and dominates TTFT; decode is memory-bandwidth-bound and dominates TPS-out — no single "latency" number captures both.
• The five rules of a reproducible protocol: fixed prompts, fixed outputs, warm cache, percentiles (not means), machine notes — every reading anyone believes comes from a script that lives in git.
• On the ticket assistant, a laptop CPU hits ~18 tok/s on Q4_K_M, which is fast enough for the deployment target; hardware upgrades pay off but are not required.

Next: fine-tuning the small model cheaply with LoRA — using the distilled dataset from module 3 to close the last accuracy gap that quantization did not open.