Lesson 3 — Retrieval: giving it your knowledge
A model knows what was in its training data up to a cutoff date. It does not know your internal documentation, your product catalogue, yesterday's figures or anything behind your login.
Two ways to change that. One works well and one is widely misapplied.
The idea, in one paragraph
Retrieval-augmented generation puts a search step in front of the model. When a question arrives, search your documents for the passages most likely to contain the answer, paste those passages into the context, and ask the model to answer using them.
That is all it is. The model's job shifts from recalling to reading, which is a task it is genuinely reliable at.
Three things this buys you, and they are the reasons it became the default architecture:
Current and private knowledge. Update a document and the next answer reflects it. No retraining.
Citations. You can show which passage the answer came from, which makes verification possible and is often the difference between a system people trust and one they do not.
Less fabrication. Grounded in supplied text, the model has far less need to invent — though it still can, and it will happily contradict a supplied passage on occasion.
How the pieces fit together
Chunking. Documents are split into passages, because you cannot paste a thousand-page manual into the context. This step is far more consequential than it appears. Chunks that are too small lose the context needed to make sense; too large and they dilute the relevant sentence among irrelevant ones and waste tokens. Splitting mid-table or mid-procedure produces chunks that are actively misleading. Respecting document structure — sections, headings, list boundaries — beats splitting at a fixed character count almost every time.
Embedding. Each chunk is converted into a vector by an embedding model, using the geometry-of-meaning idea from the NLP course. Similar meanings land near each other.
Vector storage. The vectors go into a database that can find nearest neighbours quickly. Dedicated vector databases exist, and a Postgres extension handles a surprising amount of real workload — the choice matters much less than teams expect.
Retrieval. The question is embedded the same way, the nearest chunks are found, and the top few are selected.
Generation. The chunks and the question go to the model, with an instruction to answer from the provided material and to say when it cannot.
Why most RAG systems disappoint
The failure is almost always retrieval, not generation. If the right passage is not in the context, no model can answer well — and it will usually produce something plausible anyway, which makes the failure hard to spot.
The common causes, in rough order of how often they are the culprit:
Vector search misses exact terms. Embeddings capture meaning and are mediocre at precise identifiers — a part number, an error code, a person's name. Hybrid search, combining vector similarity with traditional keyword search, fixes a large share of real-world retrieval failures and is the single highest-value addition to a basic pipeline.
Chunking destroyed the answer. The relevant procedure was split across two chunks and neither is retrievable on its own.
The question does not resemble the document. A user asks "why is my card declining" and the document says "payment authorisation failure codes". Rewriting the query with the model before searching, or generating hypothetical answers to search with, helps materially.
Top-k is wrong. Too few chunks and the answer is missing; too many and the relevant one gets buried in the middle of a long context, where models attend least reliably. Reranking — retrieving twenty candidates and using a more accurate model to pick the best three — is the standard remedy.
The answer requires synthesis across many documents. "What are our top five recurring complaints?" cannot be answered from three retrieved passages. This is an analytics question wearing a search question's clothes, and RAG is the wrong tool.
The documents do not contain the answer. No architecture fixes this, and the system should say so rather than improvise.
Before touching prompts or swapping models, log the retrieved chunks for twenty questions that were answered badly and read them. In most cases you will see immediately that the right passage was never retrieved, and you will have saved yourself from optimising the wrong half of the system.
RAG or fine-tuning?
This is the most consequential architectural decision in the area, and it is regularly made wrongly.
| Retrieval | Fine-tuning | |
|---|---|---|
| Teaches | Facts, current information | Form: style, format, vocabulary, tone |
| Updating | Change the document | Retrain |
| Citations | Yes, naturally | No |
| Setup cost | Moderate | Higher, needs curated examples |
| Cost per request | Higher, more tokens in context | Lower, shorter prompts |
| Fails when | Retrieval misses | You needed facts |
The frequent mistake: fine-tuning a model on a corpus of company documents in the hope it will then answer questions about them. It generally will not. Fine-tuning adjusts how the model writes, not what it can look up, and a model fine-tuned on your documents will produce text that sounds like your documents while inventing the specifics. The two are complementary — retrieval for facts, fine-tuning for house style — and if you only do one, do retrieval.
What to try in order
- Prompting alone. Sometimes the model already knows enough. Test this before building anything.
- Paste the document in. If the relevant material fits in the context, skip the entire retrieval pipeline. Long context windows have made this viable for far more cases than it used to be, and it is dramatically simpler.
- Basic retrieval. Chunk, embed, search, generate.
- Hybrid search and reranking. Where most of the remaining quality lives.
- Query rewriting. For when users and documents speak differently.
- Fine-tuning. Only for style, format or a domain vocabulary the model handles poorly.
Most teams that jump to step 6 should have stopped at step 2 or 4. The order is the lesson.
In three sentences
Retrieval-augmented generation puts a search step in front of the model so it reads supplied passages instead of recalling from training, which gives you current and private knowledge, citations that make verification possible, and substantially less fabrication. When these systems disappoint, the cause is almost always retrieval rather than generation — vector search missing exact identifiers, chunking that split the answer, questions phrased unlike the documents, or the relevant chunk buried in a long context — so hybrid search, reranking and query rewriting are where the quality actually lives, and reading the retrieved chunks for twenty bad answers is the fastest diagnostic available. Retrieval teaches facts while fine-tuning teaches form, and the common expensive mistake is fine-tuning on company documents expecting the model to then answer questions about them, when it will instead produce text that sounds right with invented specifics.
Next — Lesson 4: tools and agents →