#rag — LLMs and transformers

RAG: answering from your documents.

What you'll play with

Welcome to #rag. On the left, 14 gray spheres: the employee handbook of a fictional company, cut into chunks of 40 words (with 25% overlap), each turned into a vector and dropped into space by a 3D projection. On the right, two empty frames: the context we will send to the language model, and its answer. An LLM alone answers from memory — and confidently invents when it does not know: that is hallucination. Retrieval-augmented generation (RAG) changes the rule: we first search the chunks closest to the question, put them in the context, and the model answers from them, citing its sources. Here the embedding is a toy (bag of words weighted by TF-IDF, not a real embedding model), but the pipeline is exactly the same.
Let us ask a first question: /example 1 — "How many home office days per week?". Watch the blue sphere drop into the cloud, then the green links reach out to the closest chunks.
What if we kept only one chunk? /k 1.
Widen it: /k 4. The system asks for four neighbours.
Assemble the answer from this context: /answer.
Now a question that has nothing to do with the documents: /off-topic.
Chunking matters as much as retrieval. Switch to 20-word chunks: /chunk 20.
Ask a real question again on these small chunks: /example 2 — "How do I file an expense report?".
Your turn to play. /documents cooking then /example 3 (rescue a mayonnaise); /documents ai then /question what is a hallucination; /threshold 0.4 to see chunks turn gray again; /overlap 50 to double the chunks; /k 8 to stuff the context with noise; /reset to start over. Remember the triptych: chunk well, retrieve few but relevant, refuse when nothing fits. Next stop: #alignment-rlhf, where we teach the model to prefer honest and helpful answers.

Channel commands

/documents <company|cooking|ai> — Switch the document corpus (the question is cleared).
/chunk <20..80> — Chunk size, in words.
/overlap <0..50> — Share of words repeated between consecutive chunks (%).
/k <1..8> — Maximum number of chunks placed into the context.
/threshold <0..1> — Minimum similarity for a chunk to enter the context.
/question <word> — Ask a question (the whole line) and run the search.
/answer — Assembles the answer from the retained chunks, with citations.
/off-topic — Ask a question unrelated to the documents.
/example <1|2|3> — Ask a ready-made question for the current corpus.
/reset — Returns to the employee handbook, 40-word chunks, k = 3, threshold 0.20, no question.

Glossary

RAG (retrieval-augmented generation): Architecture where, before answering, we retrieve from a document store the passages closest to the question, then give them to the language model as context. The answer then relies on fresh and verifiable sources instead of the model's memory alone.
Chunk: Fragment of a document (a few tens to a few hundreds of words) indexed as a unit. Too short and it cuts sentences and loses meaning; too long and it mixes topics and dilutes the context. Chunking is the first knob of a RAG system.
Embedding: Numeric vector that represents a text so that two texts close in meaning are close in space. A real embedding model is learned; here, we use a toy (bag of words weighted by TF-IDF, hashed into 32 dimensions) that only brings identical roots close.
Vector database: Index that stores the chunks' embeddings and quickly answers the question "which vectors are closest to this one?". At scale, it uses approximate structures (HNSW, IVF) to avoid comparing the question against every vector.
Cosine similarity: Cosine of the angle between two vectors: 1 when they point in the same direction, 0 when they are orthogonal (nothing in common). Independent of text length, it is the standard metric to compare embeddings.
k-nearest search: Retrieval step: rank the chunks by similarity with the question and keep the top k. Small k: precise but incomplete context; large k: more complete but noisy and expensive. In practice we retrieve broadly and then re-rank with a finer model.
Overlap: Share of words a chunk repeats from the previous one. It gives sentences cut by a boundary a second chance to appear whole in the next chunk, at the price of a larger index and duplicates in the context.
Context window: Text the language model receives to answer: the question plus the retrieved chunks. It is capped in tokens and every token costs; so we put in few chunks, but well chosen.
Hallucination: Fluent, confident and false answer, produced by a model filling a knowledge gap. RAG reduces it by providing the facts in the context and allowing a refusal when no relevant passage is found.
Grounding and source citation: Attaching each claim of the answer to the passage that supports it, for instance with a number [1], [2]. The user can verify, and we can spot sentences that nothing supports. A grounded system also knows how to say "I do not know from the documents".

Other channels in LLMs and transformers

#tokenization — Tokenization and BPE: how an LLM chops up text.
#attention — Self-attention: each word looks at the others.
#transformer-block — The transformer block: attention, residuals, normalization, FFN.
#generation-temperature — Generation: temperature, top-k and top-p.
#rag — RAG: answering from your documents.
#alignment-rlhf — Alignment and RLHF: learning what humans prefer.