Skip to main content

Module 5 — Efficient runtime formats

A safetensors file downloaded from Hugging Face is a good format to distribute weights, but a poor format to run on a CPU. It assumes a Python process, a PyTorch stack, and a compute pattern (padded batches, contiguous tensors) that leaves a lot of speed on the table for a single interactive user. Runtime formats are the specialised repackagings of the same weights for a specific class of hardware. Three matter in 2026: GGUF, ONNX and MLX.

The running example targets three deployment surfaces in the same team: a Windows laptop with an Intel CPU (most agents), a Linux workstation with a low-end NVIDIA GPU (the team lead), and one MacBook Air with an M-class chip (the manager). One base model, three runtimes.

GGUF and llama.cpp — the default for CPU​

llama.cpp is a small, portable C++ inference engine originally written for Llama and now the de facto standard for CPU inference of open LLMs. Its native format is GGUF — a single binary file containing weights, tokenizer, chat template and metadata.

Two properties make it the right default for CPU deployment:

  • Native quantization. GGUF stores Q8_0, Q4_K_M, Q5_K_M, Q6_K and half a dozen more directly, without a runtime dequantization step in Python. The Q4_K_M file from module 4 is what llama.cpp reads and runs as-is.
  • Small dependency footprint. A single executable plus a single file. It ships to a locked-down enterprise laptop where installing PyTorch is a three-week ticket.

Ecosystem context: Ollama wraps llama.cpp with a REST server, a model registry and one-line updates. It is what module 8 uses to install the model on the ticket agents' workstations. Under the hood, it is llama.cpp reading GGUF.

ONNX Runtime — for the Windows and cross-vendor GPU story​

ONNX (Open Neural Network Exchange) is a portable computation-graph format supported by Microsoft's ONNX Runtime. It is the format of choice when:

  • You must run on a Windows CPU with DirectML, or on an Intel GPU with OpenVINO — both of which the ONNX Runtime supports natively, and neither of which llama.cpp supports as well.
  • You need to run alongside a classic ML pipeline (a scikit-learn preprocessor, an XGBoost postprocessor) that also lives in ONNX — one runtime instead of two.
  • You want to call the model from C#, Java or Rust as a library, not as an HTTP service.

Course 37 covers ONNX Runtime in detail; this module only frames the trade-off. For our ticket assistant on the team lead's NVIDIA GPU, ONNX Runtime with the CUDA execution provider gives 2 to 4× the tokens per second of the CPU path — but the CPU path is already fast enough, so we do not bother.

MLX — the Apple silicon path​

MLX is Apple's array framework designed to exploit the unified memory of M-series chips. On a MacBook Air with 16 GB of unified memory, mlx-lm running a 3B model quantized to 4 bits does roughly 25 to 40 tokens per second. That is comparable to a mid-range discrete GPU and dramatically better than what llama.cpp gets on the same machine.

For the one Mac in the team, MLX is the right choice. For everyone else, it is irrelevant — MLX only runs on Apple silicon.

Conversion recipes​

The workflow is always: train and fine-tune in PyTorch, convert to the target format, verify byte-for-byte outputs on a small test set.

Hugging Face → GGUF​

# From a Hugging Face directory with .safetensors weights
python llama.cpp/convert-hf-to-gguf.py ./qwen2.5-3b-ticket \
  --outfile qwen-ticket.f16.gguf --outtype f16

# Then quantize as in module 4
./llama-quantize qwen-ticket.f16.gguf qwen-ticket.Q4_K_M.gguf Q4_K_M

Hugging Face → ONNX​

# optimum-cli, the maintained path in 2026
optimum-cli export onnx --model ./qwen2.5-3b-ticket \
  --task text-generation-with-past ./qwen-ticket-onnx

Hugging Face → MLX​

python -m mlx_lm.convert --hf-path ./qwen2.5-3b-ticket \
  --mlx-path ./qwen-ticket-mlx -q --q-bits 4

The verification step nobody skips twice​

Conversion is a place where silent regressions are common. A tokenizer edge case, a chat-template mismatch, a rope-scaling parameter dropped — any of these produces a model that speaks but answers wrong.

The discipline is a cross-format smoke test: 20 prompts (the same across formats), invoked on each runtime, outputs diffed. Not a benchmark — a diff. If GGUF and ONNX disagree on prompt 7, do not ship; investigate. In practice, 80 % of the time the disagreement is a tokenizer setting, and 20 % of the time it is a bug worth reporting upstream.

Choosing a format for the hardware​

The one-line rule for 2026:

  • CPU-only, any OS: GGUF via llama.cpp or Ollama.
  • NVIDIA GPU: GGUF via llama.cpp with CUDA build for the simple path, ONNX Runtime for the mixed C#/Java stack.
  • Apple silicon: MLX for the fastest tokens, GGUF for portability with the rest of the fleet.
  • Windows CPU with an integrated Intel GPU: ONNX Runtime with OpenVINO, if that hardware happens to be the bottleneck.

For the ticket assistant fleet, we ship GGUF via Ollama on every machine except the one Mac, which runs MLX. Two artefacts to maintain, one base model.

In summary​

  • Runtime formats repackage weights for specific hardware; distributing a safetensors file is not deploying a model.
  • GGUF via llama.cpp / Ollama is the CPU default in 2026 — single binary, native quantization, small dependency footprint.
  • ONNX Runtime wins on Windows-CPU-with-DirectML, on Intel GPU, and when you need to embed the model in a C#/Java/Rust process.
  • MLX is the Apple-silicon path; on a MacBook it beats every alternative and only runs there.
  • Always run a cross-format smoke test after conversion — tokenizer and chat-template edge cases produce silent quality regressions.

Next: measuring latency and throughput with a protocol that survives the next hardware refresh and the next model swap.