RAG vs Fine-tuning: How to Actually Choose
The short answer, and it is worth stating before anything else: RAG changes what the model knows, fine-tuning changes how the model behaves. Most teams reach for fine-tuning when they actually have a knowledge problem, spend six weeks on it, and end up with a model that sounds right and gets facts wrong.
The distinction that decides everything
A language model has two very different kinds of limitation, and confusing them is the single most expensive mistake in applied LLM work.
A knowledge limitation means the model does not have the information. It never saw your internal wiki, your product catalogue, last Tuesday's pricing change. No amount of training on writing style will fix this.
A behaviour limitation means the model has the information but does not use it the way you need. It answers in the wrong format, ignores your tone, refuses tasks it should accept, produces JSON with inconsistent keys.
Retrieval-augmented generation attacks the first. Fine-tuning attacks the second. They are not competitors on a spectrum; they solve unrelated problems and are frequently used together.
What RAG actually does
RAG puts the relevant information into the prompt at the moment of the request. The pipeline has three stages:
- Index. Your documents are split into chunks and each chunk is converted into a vector — a list of numbers capturing its meaning. Those vectors go into a database built for similarity search.
- Retrieve. The user's question is converted into a vector the same way, and the database returns the chunks closest to it.
- Generate. Those chunks are inserted into the prompt with an instruction along the lines of "answer using only the context below".
The model's weights are never touched. That single fact explains every advantage RAG has.
Updating knowledge is a database write. A price changed this morning? Reindex that document. The next question gets the new answer. With fine-tuning, the same change means assembling training data and running a training job.
You can cite sources. Because you know which chunks you retrieved, you can show the user where the answer came from. This is not a nice-to-have — in legal, medical, financial or support contexts, an answer without a source is often unusable.
Access control still works. Filter the retrieval by the user's permissions and a model physically cannot leak a document that user may not see. Once information is baked into weights, it is available to everyone who can call the model.
What fine-tuning actually does
Fine-tuning continues training on your examples, adjusting the weights so the model's default behaviour shifts towards what you demonstrated.
In practice almost nobody updates all the weights any more. LoRA and its quantized variant QLoRA freeze the original model and train a small set of additional parameters — typically well under 1% of the total. The result is a small adapter file you load on top of the base model. This is what made fine-tuning affordable: a job that once needed a cluster now often runs on a single GPU.
Fine-tuning is the right tool when you need:
- A consistent output format. If you need the same JSON structure every time, a few hundred good examples beat any amount of prompt instruction.
- A specific tone or persona that would otherwise cost you a long system prompt on every single call.
- A narrow task done cheaply. A fine-tuned small model often matches a much larger general model on one specific task, at a fraction of the inference cost. This is the strongest economic argument for fine-tuning and the most underused.
- Behaviour a prompt cannot reach, such as domain-specific classification conventions that took your team years to settle.
Where fine-tuning quietly fails
This is the part that costs teams months, so it deserves to be explicit.
Fine-tuning is unreliable for injecting facts. You can train a model on your documentation and it will learn to sound like your documentation. Ask it for a specific figure and it may produce something plausible and wrong, because gradient descent optimises for likely-looking text, not for retrieval accuracy. There is no mechanism guaranteeing a specific fact survives training intact.
It cannot cite. The information is diffused across billions of weights. There is nothing to point at.
Knowledge goes stale by construction. Every update means another training cycle.
Catastrophic forgetting is real. Training hard on a narrow task degrades general capability. A model fine-tuned aggressively on your support tickets can get worse at ordinary reasoning.
Side by side
| RAG | Fine-tuning | |
|---|---|---|
| Solves | missing knowledge | wrong behaviour |
| Updating information | reindex a document, minutes | new training run |
| Can cite sources | yes | no |
| Respects per-user permissions | yes, filter at retrieval | no |
| Cost per request | higher: retrieval plus more input tokens | lower: shorter prompts |
| Setup cost | pipeline, chunking, evaluation | data collection, training, evaluation |
| Fails at | questions needing whole-corpus synthesis | reliable factual recall |
| Main risk | retrieving the wrong chunk | plausible-sounding fabrication |
Costs, in the shape they actually take
Precise figures date badly, so consider the structure rather than the numbers.
RAG shifts cost to inference. Every request carries retrieval plus the retrieved context as input tokens. A prompt that grows from 500 to 4,000 tokens multiplies your input cost per call by roughly eight. At a million calls a month, that difference dominates your bill.
Fine-tuning shifts cost to preparation. A LoRA run on a mid-sized open model is, today, plausibly a few hours on one rented GPU — often tens of dollars, not thousands. The expensive part was never the compute. It is building and maintaining the dataset, which is human work and recurring.
The decisive question is usually neither: it is evaluation cost. Both approaches need a test set that tells you whether a change helped. Teams that skip this spend months tuning blind, which is far more expensive than any GPU.
A decision path that works
Follow it in order. The order is the point.
- Start with prompting. Write a careful prompt with two or three examples. Measure it. A surprising share of projects stop here, and that is a success, not a shortcut.
- Is the model missing information? Then RAG. Do not fine-tune to teach facts.
- Does it have the information but behave wrong? Try a better prompt first. If the prompt is long, brittle, and still inconsistent, fine-tune.
- Is inference cost or latency the binding constraint? Fine-tune a smaller model on the specific task and compare it against the large general one. This is where fine-tuning pays best.
- Both problems at once? Use both. A fine-tuned model that reliably produces your output format, fed retrieved context at request time, is a common and sound production shape.
Frequently asked questions
Can I use RAG and fine-tuning together?
Yes, and it is often the right answer. Fine-tune for format and tone, retrieve for facts. The two operate at different stages and do not conflict.
Does a large context window make RAG unnecessary?
No. You can put a lot into a long context, but you pay for every token on every call, latency grows, and models still attend unevenly across very long inputs. Retrieval remains the way to send only what matters — and it is what lets you cite and permission the source.
How much data do I need to fine-tune?
Far less than people expect for behaviour, and no amount is enough for facts. A few hundred high-quality, consistent examples often shifts format and tone convincingly. Quality and consistency matter more than volume: contradictory examples teach the model to be inconsistent.
Why did my fine-tuned model start inventing things?
Most likely you trained it on documents to teach knowledge. It learned the register of your documents without a mechanism for accurate recall, so it generates text that fits the pattern. Move the knowledge into retrieval.
Is RAG just semantic search with extra steps?
Retrieval is the search half. The generation half — synthesising an answer across several retrieved passages, in your format, in the user's language — is what search alone does not give you.
Where to go deeper
If you are choosing between these approaches for a real system, the two mechanisms are worth understanding properly rather than by analogy. Our free Large Language Models course covers how transformers, attention and prompting work, and the RAG Systems and Model Fine-tuning courses go through embeddings, vector stores, chunking strategy, LoRA and QLoRA with working code.
The habit worth building is narrower than any of it: before choosing a technique, write down whether your problem is knowledge or behaviour. That one sentence saves more engineering time than any framework.