What GPU Do You Need to Run an LLM Locally?
The rule that answers most of the question: a model needs roughly its parameter count multiplied by the bytes per parameter, plus about 20% overhead. A 7-billion-parameter model at 16-bit precision is 7 × 2 = 14 GB of weights, so around 17 GB of VRAM in practice. Quantize it to 4-bit and the same model fits in about 5 GB.
Where the memory actually goes
Three things compete for VRAM, and people usually account for only the first.
The weights. This is the big, fixed cost, and the arithmetic is simple:
| Precision | Bytes per parameter | Quality impact |
|---|---|---|
| FP32 | 4 | reference, wasteful for inference |
| FP16 / BF16 | 2 | effectively no loss |
| INT8 | 1 | very small loss |
| 4-bit | ~0.5 | small but measurable loss |
| 3-bit and below | ~0.4 | noticeable degradation |
The KV cache. As the model generates, it stores the keys and values for every token already in the context so it does not recompute them. This grows linearly with context length and can become substantial: a long conversation at a large context window can consume several gigabytes on top of the weights. This is the cost people forget, and it is why a model that loaded fine crashes twenty minutes into a session.
Activations and framework overhead. The working memory of the forward pass, plus whatever CUDA and your runtime reserve. Budget 10–20% on top of everything else.
VRAM by model size
Approximate total VRAM including overhead, for inference at a moderate context length:
| Model size | FP16 | INT8 | 4-bit |
|---|---|---|---|
| 3B | ~7 GB | ~4 GB | ~2.5 GB |
| 7–8B | ~17 GB | ~9 GB | ~5 GB |
| 13B | ~30 GB | ~15 GB | ~8 GB |
| 30–34B | ~75 GB | ~38 GB | ~20 GB |
| 70B | ~150 GB | ~75 GB | ~40 GB |
Read the 4-bit column first, because that is how most people actually run models locally. It changes the conclusion completely: a 7B model on a 6 GB laptop GPU is realistic, and a 70B model on two 24 GB cards is possible.
What this means for real hardware
8 GB of VRAM runs 7–8B models comfortably at 4-bit. This covers the large majority of local use: coding assistance, summarising, drafting, classification.
12–16 GB runs 13B models at 4-bit with room for long contexts, or 7B models at 8-bit if you want the extra quality margin. This is the sweet spot for price against capability.
24 GB runs 30B-class models at 4-bit, and is the practical floor for fine-tuning anything interesting with QLoRA.
48 GB and up, usually two cards, puts 70B-class models within reach at 4-bit.
Apple Silicon deserves a separate note because it behaves differently. Unified memory means the GPU addresses system RAM, so a machine with 32 or 64 GB can hold models that would need an expensive discrete card. Bandwidth is lower than a high-end NVIDIA GPU, so generation is slower, but capacity is often the binding constraint and Apple wins on capacity per dollar.
CPU-only works and is slower than people hope. Expect a few tokens per second for a 7B model at 4-bit — usable for batch jobs you walk away from, frustrating for chat.
The mistake everyone makes: buying for compute
Here is the thing that surprises people who come from training or gaming benchmarks.
When you generate text one token at a time, the GPU must read every weight of the model from memory to produce each token. For a 7B model at 4-bit, that is roughly 4 GB read per token. The compute involved is trivial by comparison. Generation speed is therefore governed almost entirely by memory bandwidth, not by teraflops.
The practical consequences are counter-intuitive and worth internalising:
- A card with more bandwidth beats a card with more compute for single-stream generation.
- Two GPUs do not double your speed for one conversation. They let you load a bigger model.
- Reading the first prompt is compute-bound and fast; generating the reply is bandwidth-bound and slow. That is why time-to-first-token and tokens-per-second behave so differently.
If you are choosing between cards, look up memory bandwidth in GB/s and treat it as the number that predicts your experience.
Quantization: what you actually give up
Quantization stores weights with fewer bits. The loss is real but not linear, and the shape of it matters.
Going from 16-bit to 8-bit is close to free. Going to 4-bit costs a little — slightly weaker on long multi-step reasoning, marginally more likely to drop an instruction — and for most tasks you will not notice in casual use. Below 4-bit, degradation becomes visible quickly: the model stays fluent while getting worse at being correct, which is the worst failure mode because it is hard to spot.
The comparison that decides it: a larger model quantized to 4-bit usually beats a smaller model at 16-bit for the same memory budget. If you have 16 GB, a 13B at 4-bit is generally a better choice than a 7B at 8-bit. Capability comes from parameter count more than from precision.
Getting started without buying anything
Test before you spend money.
Ollama is the shortest path: install it, pull a model, and it handles quantization and memory management for you. llama.cpp underneath gives you finer control and excellent CPU and Apple Silicon support. For serving many concurrent users, vLLM is the right tool, because batching changes the economics — with many requests in flight the GPU becomes compute-bound again and throughput per dollar improves sharply.
Run a 7B model at 4-bit on whatever machine you have now. You will learn more about what you need from ten minutes of that than from any specification table, including this one.
Frequently asked questions
Can I run an LLM without a GPU?
Yes. A 7B model at 4-bit runs on a modern CPU with 8 GB of RAM at a few tokens per second. Fine for scripted jobs, slow for conversation.
Does more VRAM make the model smarter?
No. It lets you load a larger model or a longer context. The intelligence comes from the model you can now fit, not from the memory itself.
Do two GPUs double performance?
Not for a single conversation. Splitting a model across cards adds communication overhead; you gain capacity, not speed. With many simultaneous requests, two cards do roughly double throughput.
How much VRAM do I need to fine-tune rather than just run a model?
Considerably more, because you also store gradients and optimiser state. QLoRA is the workaround: it quantizes the frozen base model and trains small adapters, bringing a 7B fine-tune within reach of a 16 GB card and a 13B within reach of 24 GB.
Why did my model work and then run out of memory?
Almost certainly the KV cache growing with your conversation. The weights are fixed; the context is not. Cap the context length or start a fresh session.
Where to go deeper
Understanding why generation is memory-bound means understanding what the forward pass actually computes, and that pays off well beyond hardware choices. Our free Large Language Models course covers tokens, attention and context windows, and Deep Learning Fundamentals works through the mechanics with code. If your goal is running open models on your own machine, the Ollama and local models course goes through quantization formats and serving in detail.
One number to remember: multiply parameters by bytes per parameter, add 20%. It will get you the right answer far more often than a benchmark chart.