Module 7 — Hardware acceleration

Quantization (module 6) shrank the model into RAM. Hardware acceleration is what turns a workable model into a responsive one. This module walks through the three GPU backends Ollama uses, the trick that lets a mid-range card run models it "should not" fit, and the measurement discipline that stops arguments about which machine is faster.

The three backends

Ollama detects the accelerator at service startup and picks the appropriate backend automatically. systemctl status ollama (Linux), the tray icon (Windows) or the menu-bar item (macOS) reports what was picked. The three currently supported paths:

• NVIDIA CUDA — the default on any Windows or Linux machine with a supported GeForce, RTX or Tesla card and a recent driver. Ollama ships the CUDA runtime it needs; no separate CUDA install is required.
• AMD ROCm — supported on Linux and Windows for a subset of Radeon Instinct and RX cards. Coverage is narrower than CUDA; check the runtime's compatibility list before ordering hardware.
• Apple Metal — automatic on all M-series Macs (M1 through M4 and successors). Nothing to configure. Unified memory means "VRAM" and "RAM" are the same pool, which is unusually forgiving on large models.

If none of the three matches, Ollama runs on CPU only. That is a valid fallback — a q4_K_M 7-B model will do 5 to 10 tokens per second on a modern desktop CPU — but the associate typing in the chat window will feel it, and the closet server without a GPU is not the right place for a 70-B model.

What "GPU offload" means

A language model is a stack of transformer layers. Ollama can load some layers into GPU memory and leave the rest on CPU RAM. Generation walks through all layers on every token, so a layer on GPU runs fast and a layer on CPU runs slow — the mix determines the effective speed.

The knob is num_gpu, an integer setting the number of layers offloaded to the GPU. Default is auto: the runtime measures free VRAM and offloads as many layers as fit, with a small safety margin. That default is right most of the time.

To override — say, on a machine where another process claims part of the VRAM — set it explicitly:

ollama run llama3.1:70b-instruct-q4_K_M
>>> /set parameter num_gpu 40

Or via the API, in options. Setting num_gpu 999 asks for full offload and fails cleanly if VRAM is insufficient. Setting num_gpu 0 forces CPU-only inference, useful for a repeatable baseline.

Partial offload — the trick that changes what fits

A 70-B model at q4_K_M weighs 42 GB. A 24 GB RTX 4090 cannot hold it entirely — but it can hold most of it. If 32 of 80 layers fit on the GPU, generation runs at roughly (fraction on GPU × GPU speed) + (fraction on CPU × CPU speed). In practice, offloading half the layers of a 70-B model to a 24 GB card gives 6 to 12 tokens per second — usable for a partner reviewing a contract on a workstation, far better than the 1 to 2 tokens per second of pure CPU.

Partial offload is why a mid-range GPU is worth adding to the office server: not to load the biggest models fully, but to accelerate the layers that do fit.

Measuring tokens per second

The single question that matters for user experience is "how many tokens per second does my machine generate on my model?". The answer is one command:

ollama run <tag> --verbose <<< "Write a 200-word summary of the fair-use doctrine."

The eval rate line at the end is the number. Do it three times, take the median. The typical order of magnitude for the firm's setup:

Machine	Model	eval rate
MacBook Pro M3, 16 GB	qwen2.5:7b-instruct-q4_K_M	25 – 40 tokens/s
MacBook Pro M3 Max, 36 GB	qwen2.5:14b-instruct-q4_K_M	35 – 55 tokens/s
Windows laptop, no GPU	qwen2.5:7b-instruct-q4_K_M	5 – 10 tokens/s
Server, RTX 4070 12 GB	qwen2.5:14b-instruct-q4_K_M	60 – 90 tokens/s
Server, RTX 4090 24 GB	llama3.1:70b-instruct-q4_K_M (partial offload)	6 – 12 tokens/s

Numbers move with driver version, thermal throttling and background load — the point is not the exact figure, it is the ordering. A 7-B model on Apple Silicon beats a 14-B model on the same machine's CPU by a factor of five to ten. The GPU on the server crosses the threshold where the assistant becomes conversational.

CPU-only, on purpose

Sometimes the CPU is the right choice: a partner's older laptop with no supported GPU, an audit that requires deterministic timings, a machine where a video render is already claiming the GPU. On CPU, three levers help:

• Threads: the runtime picks num_cpus by default. On a hyperthreaded chip, capping at physical cores (OLLAMA_NUM_THREADS=8 on an 8-core CPU with 16 logical) is often a few percent faster.
• A smaller quantization — q4_K_M over q5_K_M — pays back more on CPU than on GPU because the CPU is memory-bandwidth-bound.
• AVX2 or AVX512 matter. On a very old CPU without AVX2, generation is one to two tokens per second even for a small model; upgrade the hardware before you upgrade the model.

Diagnosing "the GPU is not being used"

Two symptoms suggest the offload is not happening. First, nvidia-smi (or radeontop, or macOS Activity Monitor's GPU pane) shows the GPU idle during generation. Second, eval rate is exactly what a CPU-only run reports.

Three causes, in order of frequency. The driver is missing or too old — reinstall from the vendor's site and restart the service. Another process (a browser tab, a game, a Jupyter kernel) is using all the VRAM — close it, or lower num_gpu to leave headroom. The model was requested with num_gpu 0 somewhere in a Modelfile or an API call — check ollama show and the code.

Where the GPU pays back the most

For interactive use, the biggest jump is not "no GPU to some GPU" but "some GPU to enough GPU". A card that holds all layers of your production tag runs two to four times faster than a card that offloads half. When budgeting hardware, size to the working tag's weight footprint plus KV cache (module 6), not to the top-of-line card.

Summary

• Ollama picks CUDA, ROCm or Metal automatically and falls back to CPU when none is available.
• num_gpu controls layer offload; the default auto fits as many layers as VRAM allows.
• Partial offload lets a card hold most of a model too large to fit entirely, at the price of CPU-bound layers.
• Measure eval rate with --verbose — the only source of truth on which machine is faster for your model.

Next module: exposing the runtime to existing applications — from a Python script to LangChain to Open WebUI — without opening the wrong port.