Module 3 — Chunking: size, overlap and structure
Module 2 turned files into long extracted texts, each with its metadata. This module cuts those texts into passages the index can actually search. Chunking is where most RAG quality is made or lost — badly cut passages produce badly answered questions, and the pipeline downstream has no way to recover.
Why not just embed the whole document?
Embeddings compress meaning into a fixed-size vector — typically 384 to 1024 numbers. Compressing a fifty-page manual to 768 numbers dilutes every specific answer inside a soup of averages. A question about the exact retention period for accident reports competes with headers, appendices, glossaries and generic prose, and loses.
Cutting into chunks of a few hundred tokens gives each specific idea its own vector. The retriever brings back the passage that actually addresses the question, and only that. The trade-off is symmetric: chunks that are too small break sentences and lose the local context that made them intelligible.
Four families, from crude to careful
| Strategy | How | When it wins | When it fails |
|---|---|---|---|
| Fixed-size | Every N tokens, no overlap | Prototypes, log lines, chat logs | Splits mid-sentence and mid-table |
| Sentence-based | Split on . ! ? then pack up to N tokens | Long prose paragraphs | Abbreviations, numbered lists, code |
| Section-based | One chunk per ## heading | Well-structured procedures | A single section far exceeds N |
| Hierarchical | Small chunk for retrieval, big chunk for context | Long, uneven documents | Adds implementation complexity |
Fixed-size is the baseline every tutorial shows and nothing else recommends. Sentence-based is a real improvement for prose, and nltk.sent_tokenize or the multilingual pysbd do the segmentation correctly. Section-based is the natural fit when your extractor kept the heading hierarchy of module 2 — a procedure like QUAL-047 divides cleanly into "Purpose", "Scope", "Roles", "Procedure", "Records", each becoming one chunk.
Hierarchical is the strategy production systems converge on, and module 6 revisits it under the name parent document retriever.
Measuring chunk size correctly
Size in characters is wrong. The model that embeds and the model that generates both count in tokens, and a token is roughly a syllable of an English word or a piece of a Chinese character. Use the tokenizer of the model you actually use:
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("intfloat/multilingual-e5-base")
def n_tokens(text: str) -> int:
return len(tok.encode(text, add_special_tokens=False))
A reasonable working range for RAG chunks is 200 to 400 tokens, roughly 800 to 1600 characters in English, less in languages that pack more meaning per character. Below 100 tokens, chunks lose context; above 500, they dilute meaning and start competing for space in the final prompt.
Overlap: the seam nobody sees
A passage cut on a sentence boundary rarely says anything false, but the next passage often starts with a pronoun ("This procedure applies…") that no longer refers to anything. Overlap copies the last few sentences of chunk N as the first sentences of chunk N+1, so a question that lands on the seam still hits a chunk that carries enough context to be understood.
The rule of thumb is 10 % to 20 % overlap. Too little and seams stay broken; too much and the index carries near-duplicates that all match the same queries and drown out other passages. On the red thread's 300 procedures, 15 % has been consistently good.
def sliding_chunks(sentences: list[str], size: int, overlap: int) -> list[str]:
chunks, current, current_len = [], [], 0
for s in sentences:
s_len = n_tokens(s)
if current_len + s_len > size and current:
chunks.append(" ".join(current))
# Keep the tail whose total length is about `overlap` tokens
tail, tail_len = [], 0
for prev in reversed(current):
p_len = n_tokens(prev)
if tail_len + p_len > overlap:
break
tail.insert(0, prev)
tail_len += p_len
current, current_len = tail, tail_len
current.append(s)
current_len += s_len
if current:
chunks.append(" ".join(current))
return chunks
The table-splitting failure
One of the classic RAG diagnoses is a question whose answer sits in a table, and whose retriever brings back a chunk containing only the top half of that table. The header row and half the values live in chunk N; the rest live in chunk N+1. Neither, on its own, answers the question. Both look convincing enough for the reranker to accept, and the language model composes a wrong answer from the visible half.
The fix has three ingredients. First, at extraction (module 2), keep tables as structured objects, not as flat text. Second, at chunking, treat a table as an atomic block — a table shorter than 2 × N tokens becomes one chunk, headers included; a longer one is split by group of rows with the header repeated. Third, at retrieval, oversample near a table hit so both halves have a chance to appear.
def chunk_document(blocks: list[dict], size: int, overlap: int) -> list[dict]:
"""Blocks come from extraction with type in {"para", "table", "heading"}."""
out = []
buffer, buffer_len = [], 0
for block in blocks:
if block["type"] == "table":
if buffer:
out.append({"text": "\n".join(buffer), "kind": "prose"})
buffer, buffer_len = [], 0
out.append({"text": block["text"], "kind": "table"})
continue
text_len = n_tokens(block["text"])
if buffer_len + text_len > size and buffer:
out.append({"text": "\n".join(buffer), "kind": "prose"})
buffer, buffer_len = [], 0
buffer.append(block["text"])
buffer_len += text_len
if buffer:
out.append({"text": "\n".join(buffer), "kind": "prose"})
return out
"Retention: seven years" is meaningless without the enclosing heading "3.2 Accident reports". At chunking time, prepend the trail of parent headings to each chunk's text: "# QUAL-047 > 3. Records > 3.2 Accident reports\n\nRetention: seven years". The embedding then encodes the topic, not the fragment alone.
Hierarchical chunking and parent context
Small chunks are best for retrieval — precise vectors, targeted matches. Bigger chunks are best for the language model — enough context to compose a coherent answer without falling for a fragment. Hierarchical chunking gets both by indexing the small chunks and, at generation time, expanding each hit into its parent block.
CHILD_SIZE, PARENT_SIZE = 200, 800
children = chunk_document(blocks, size=CHILD_SIZE, overlap=40)
parents = chunk_document(blocks, size=PARENT_SIZE, overlap=0)
for i, c in enumerate(children):
c["parent_id"] = find_parent(c, parents) # index of the containing parent
Retrieval matches on children, then step 3 of the pipeline replaces each child with parents[child["parent_id"]] before building the prompt. The retriever wins the precision game, the model wins the context game.
On the red thread: a small comparison
On a sample of 60 annotated questions over the 300 procedures, three strategies produced roughly the following recall at :
- Fixed 500 tokens, no overlap: 0.61
- Sentence packing to 300 tokens, 15 % overlap: 0.78
- Hierarchical (child 200, parent 800), 10 % overlap: 0.84
Numbers depend on the questions, on the extractor and on the embedding model, but the ranking is robust: crude chunking leaves 20 points of recall on the table. Module 8 explains how to build this kind of evaluation set for your own corpus.
In summary
- A whole document dilutes meaning across a single vector; chunks give each specific idea its own embedding at the cost of losing context.
- Count size in tokens, using the tokenizer of the actual model; 200 to 400 tokens with 10 % to 20 % overlap is a solid default.
- Treat tables as atomic blocks, repeat headers when splitting, and prepend the parent-heading trail so the embedding encodes the topic, not just the fragment.
- Hierarchical chunking retrieves on small children and expands to bigger parents at generation time; on the red thread it recovered 6 points of recall over sentence packing.
Next module: turning chunks into vectors, choosing an embedding model, and storing everything in a vector database with the metadata filters citations depend on.