Module 8 — Quantization and cost-efficient serving

Modules 3, 4 and 7 gave us a model that behaves. This one makes it affordable to run. Serving a 7B model in full precision requires 14 GB of GPU memory just for the weights, before you have accounted for the KV-cache of a single conversation. Quantization reduces the weight precision from 16 bits down to 8, 4, or even 2 bits, cutting memory by 2x, 4x, or 8x with a small quality loss.

For the customer-support assistant of the running project, this is the module that answers "can we serve this on a $2 000 GPU rather than a$30 000 one?" The answer is yes, and this module explains how.

From 16-bit weights to 4-bit weights

A pretrained model stores its weights in bfloat16 or float16: 16 bits per weight, 2 bytes. Multiply by the parameter count to get the raw memory footprint of the model file: 14 GB for a 7B model, 16 GB for an 8B one, 140 GB for a 70B one.

Quantization stores each weight in fewer bits by mapping the original range into a smaller integer grid, and reversing the mapping on the fly at inference time. Three schemes dominate:

• 8-bit (LLM.int8, bitsandbytes): halves memory with essentially no measurable quality loss.
• 4-bit (GPTQ, AWQ): quarters memory with a small quality drop; the practical default in 2026.
• 2 to 3-bit (SqueezeLLM, some GGUF variants): eighths the memory, with a quality drop that starts to matter for reasoning tasks.

The memory table for the 7B model of the running project:

Precision	Weights	KV-cache per 4k tokens	Fits on
float16	14 GB	0.5 GB	A100 40 GB, H100
int8	7 GB	0.25 GB	RTX 3090 24 GB, L4
GPTQ 4-bit	3.5 GB	0.25 GB	RTX 3060 12 GB, T4 16 GB
GGUF Q4_K_M	4.1 GB	0.25 GB	CPU with 8 GB RAM

The right column is what changes the economics. A 4-bit 7B model runs on hardware that costs a tenth of what full precision needs.

GPTQ, AWQ, GGUF: which to pick

Three formats own the space, and they are not interchangeable.

GPTQ (2022) uses a small calibration dataset to minimise per-layer quantization error. Widely supported, works with vLLM and Hugging Face Transformers, GPU-only.

AWQ (2023) protects the top 1 % most-activated weight channels from quantization. Slightly higher quality than GPTQ at the same bit rate, comparable speed, GPU-only.

GGUF (llama.cpp) is the ecosystem-defining format for CPU and mixed CPU-GPU inference. Ships with many quantization variants (Q4_0, Q4_K_M, Q5_K_S, Q8_0). Not a GPU-first format, but the only serious option when you want a laptop or a CPU-only server.

The choice, in one line:

• Cloud GPU serving with high throughput: AWQ, served with vLLM.
• Cloud GPU serving with an existing Hugging Face pipeline: GPTQ.
• Local demo, laptop, or CPU-only server: GGUF, served with llama.cpp.

vLLM and continuous batching

Naive serving sends one request at a time. A GPU capable of processing 32 tokens in parallel is bored for 31 slots. Continuous batching merges tokens from concurrent requests into the same GPU pass, and reshapes the batch on every step.

vLLM is the reference implementation. Its two key ideas are continuous batching and PagedAttention (a KV-cache manager modelled after virtual memory), which together deliver 10 to 20x higher throughput than a naive HuggingFace loop at the same latency budget.

A minimal launch:

python -m vllm.entrypoints.openai.api_server \
    --model Qwen/Qwen2.5-7B-Instruct-AWQ \
    --quantization awq \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.9

The server speaks the OpenAI Chat Completions API, so your existing client code needs no change beyond a base URL:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

response = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct-AWQ",
    messages=[{"role": "user", "content": "Where is my order?"}],
    temperature=0.3,
)
print(response.choices[0].message.content)

For a single-GPU deployment of a 7B AWQ model, expect 50 to 100 requests per second at a token generation rate of 100 to 150 tokens per second per request.

llama.cpp on CPU

When GPUs are out of the budget, or the deployment target is an on-premise laptop, llama.cpp runs GGUF models on plain CPUs, with optional GPU offload for the layers that fit.

llama-server \
    --model qwen2.5-7b-instruct-q4_k_m.gguf \
    --ctx-size 8192 \
    --host 0.0.0.0 --port 8080

Speed depends heavily on RAM bandwidth. On an M-series Mac or a modern DDR5 workstation, expect 20 to 40 tokens per second on a 7B Q4_K_M model — enough for a single conversation, not for concurrent users. This is the option for internal tools and demos, not for production traffic.

Throughput versus latency: the trade-off you cannot skip

Two metrics matter, and they pull in opposite directions.

• Latency: time from request to first token, and time to full response for a single user.
• Throughput: total tokens served per second across all concurrent users.

Continuous batching maximises throughput and slightly hurts latency of any individual request (waiting for the next batch slot). Speculative decoding trades throughput for latency by predicting several tokens at once with a small draft model and verifying with the big one.

For an interactive chat, the metric that matters is time-to-first-token: users tolerate slow generation once words start appearing, but a two-second silence feels broken. Optimise for it first, then improve total throughput to reduce cost per request.

Quantization is not free on every task

Reported quality drops on standard benchmarks (MMLU, HumanEval) for GPTQ 4-bit hover around 1 to 2 points. On rare-word generation, low-resource languages and multi-step reasoning, the drop can reach 5 points. Evaluate your quantized model on your own test set (module 9) before shipping. Do not rely on the vendor's benchmark row.

Pin your model file, not the format name

A GPTQ or AWQ model is a specific file with a specific calibration. Two files with the same name in different repositories can differ by several quality points. Store the file hash in your service configuration, so a silent upstream change does not silently degrade production.

In summary

• Quantization cuts weight memory by 2x, 4x, or more; GPTQ and AWQ dominate GPU serving, GGUF dominates CPU and laptop inference.
• vLLM with continuous batching and PagedAttention is the reference server for GPU inference and speaks the OpenAI API, so client code is portable.
• llama.cpp runs quantized models on plain CPUs at usable single-user speed, but not at production concurrency.
• Latency and throughput trade off; optimise for time-to-first-token in interactive chat, and always re-evaluate a quantized model on your own test set before shipping.

Next module: evaluation. Before shipping any of the above, we need a way to know whether it actually got better.