Module 6 — Quantization and memory footprint

The partner's MacBook has 16 GB of unified memory. The associate's Windows laptop has 32 GB. The closet server has 64 GB of RAM and a modest GPU. Three machines, one assistant, one recurring question: which tag runs where? Answering it does not require a benchmark — it requires reading the quantization label and doing arithmetic. That is the whole of this module.

What quantization actually is

Model weights are stored in a numeric format. Full precision is fp16 (or bf16) — 2 bytes per parameter. Quantization replaces those 2 bytes with a smaller representation — 1 byte, half a byte, less — grouped in blocks with a shared scale factor. A quantized weight is decoded on the fly during inference, at a small compute cost, for a large memory saving.

Two numbers matter, and they trade against each other. The bits per weight drives the model's memory footprint linearly: a 7-billion-parameter model at 4 bits weighs ~3.5 GB, at 8 bits ~7 GB, at 16 bits ~14 GB. The quality loss is small at 8 bits (indistinguishable on most tasks), noticeable at 4 bits (occasional slips on hard prompts), and painful below 3 bits (the model starts inventing facts).

Reading a tag

Ollama uses the GGUF quantization scheme from llama.cpp. The tags look cryptic and follow a pattern:

Tag	Bits per weight	Typical use
q2_K	~2.5	Tiny footprint, quality drop noticeable
q3_K_M	~3.5	Compromise for very small machines
q4_0	4.5	Simple 4-bit, superseded by K-quants
q4_K_M	4.8	Default sweet spot; the tag most tutorials pick
q5_K_M	5.7	Slightly better than Q4_K_M for slightly more RAM
q6_K	6.6	Quality near Q8, memory close to Q5
q8_0	8.5	Effectively full quality on all common tasks
f16 / bf16	16	Reference; useful for evaluation, rarely for serving

The K denotes the K-quants family, which uses per-block scale factors and beats the older simple quants at equivalent size. _M is the mid-precision variant of the family; _S is smaller and lossier, _L is larger and closer to the next tier.

Rule of thumb: pick q4_K_M first. Move up only if you measure quality problems, move down only if the tag does not fit.

Estimating RAM

The formula the firm's IT lead scribbles on a whiteboard:

Total memory ≈ weight memory + KV cache

Weight memory is parameters × bits per weight / 8. For qwen2.5:14b-instruct-q4_K_M: 14 × 10^9 × 4.8 / 8 ≈ 8.4 GB.

The KV cache is the runtime buffer that stores the model's attention keys and values for every token in the context window. Its size is roughly 2 × num_layers × num_heads × head_dim × num_ctx × bytes_per_value. For a 14-B model at num_ctx=8192, this is about 1 to 2 GB on top of the weights.

Total for qwen2.5:14b-instruct-q4_K_M at num_ctx=8192: roughly 10 GB. On the 16 GB MacBook, that fits — with headroom for the OS, the browser and one extra chrome tab. On the same machine at num_ctx=32768, the KV cache alone reaches 5 to 8 GB and the total pushes past 16 GB. This is why module 3 insisted: set num_ctx deliberately.

For a 70-B tag at q4_K_M, weights alone are about 42 GB, and the KV cache at num_ctx=8192 adds 5 to 8 GB more. The MacBook cannot host it; the 64 GB server can, tightly.

Quality, measured on the firm's questions

Benchmarks in blog posts are not the firm's benchmark. Ten real questions from the case archive are:

for TAG in qwen2.5:14b-instruct-q4_K_M qwen2.5:14b-instruct-q5_K_M qwen2.5:14b-instruct-q8_0; do
  for q in questions/*.txt; do
    ollama run "$TAG" --verbose < "$q" > "responses/$(basename $TAG)/$(basename $q)"
  done
done

Then a human — the partner, thirty minutes — reads the answers and rates each on a three-point scale (correct / partial / wrong). The typical pattern on legal-French tasks:

Tag	Correct	Partial	Wrong	RAM (num_ctx 8k)
q4_K_M	7/10	2/10	1/10	~10 GB
q5_K_M	8/10	2/10	0/10	~11 GB
q8_0	8/10	2/10	0/10	~16 GB

q4_K_M is the sweet spot on the laptop. q5_K_M on the server buys one more correct answer per ten. q8_0 matches q5_K_M at nearly double the RAM — for this family, it does not repay the cost. On another family it might; the point is to measure, not to guess.

Diagnosing an out-of-memory failure

The error surfaces on the /api/generate or /api/chat call as an HTTP 500 with model requires more system memory than is available. Three fixes, in order:

1. Drop num_ctx first. Halving it typically saves a gigabyte or more without touching the weights.
2. Move to a lighter quantization — q4_K_M from q5_K_M, or a smaller size (8b instead of 14b).
3. Load fewer models at once. OLLAMA_KEEP_ALIVE=1m evicts idle models fast; setting OLLAMA_MAX_LOADED_MODELS=1 forces sequential loading on tight machines.

What quantization does not fix

Quantization saves RAM. It does not make the model faster than the base tag at the same GPU. A q4_K_M model runs at roughly the same tokens per second as q8_0 on a GPU that is not memory-bound — sometimes slower because of decoding overhead. Speed lives in module 7.

Summary

• Quantization trades bits per weight for memory on a near-linear scale; K-quants (q4_K_M) dominate at equivalent sizes.
• Total RAM ≈ parameters × bits / 8 + KV cache, which grows linearly with num_ctx.
• q4_K_M is the default; move up only when a measurable quality gap justifies the extra RAM.
• Out-of-memory is fixed first by dropping num_ctx, then by a lighter quantization, then by evicting idle models.

Next module: adding a GPU (or Apple Silicon) to the picture — the change that actually accelerates generation.