Module 5 — Dense, lexical and hybrid search

Module 4 built an index that answers "which chunks are semantically closest?". This module answers a harder question: "which chunks are actually about what the user asked?". The two are not the same, and the difference between them is where dense-only pipelines lose 15 to 25 points of recall on real users.

Dense is not always enough

Dense retrieval — the cosine search from module 4 — is very good at paraphrase. "How long do we keep accident reports?" and "What is the retention period for incident files?" land close to the same chunk even though they share almost no words. That is the whole point of embeddings.

Dense retrieval is poor at three things: exact identifiers, rare terms, and very short queries. A user typing QUAL-047 expects to find the procedure of that number. An embedding model has probably never seen that identifier during training, and even if it has, it treats it as a small perturbation on the vector — enough for the true chunk to be beaten by half a dozen procedures whose topic is close but whose number is not QUAL-047. The name for this is acronym drowning, and it is the classic dense failure to remember.

BM25 is exactly the right complement

BM25 is a lexical scoring function. For a query $q$ and a document $d$, it computes:

$$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{f(t, d)\,(k_1 + 1)}{f(t, d) + k_1 \left(1 - b + b \cdot \frac{|d|}{\overline{|d|}}\right)}$$

The details do not matter for our purposes; what matters is the shape. BM25 rewards documents that contain the query's rare terms, penalises very long documents so a match feels significant, and cares about exact tokens. It has no idea that "car" and "vehicle" mean the same thing, but it never misses QUAL-047.

from rank_bm25 import BM25Okapi

def simple_tokenize(text: str) -> list[str]:
    return [t.lower() for t in text.split() if t]

corpus_tokens = [simple_tokenize(c["text"]) for c in chunks]
bm25 = BM25Okapi(corpus_tokens)

def bm25_search(question: str, k: int = 20):
    scores = bm25.get_scores(simple_tokenize(question))
    top = sorted(range(len(scores)), key=lambda i: -scores[i])[:k]
    return [(chunks[i], scores[i]) for i in top]

For a production index, replace rank_bm25 with Elasticsearch, OpenSearch or PostgreSQL's tsvector — the algorithm is the same, the indexing scales better, and language-specific analyzers strip stopwords and normalise plurals.

The failing acronym: worked example

On the red thread, one of the annotated questions is:

"What are the accident-reporting steps in QUAL-047?"

Dense retrieval alone returns, at rank 1 to 5, chunks about accident reporting in other procedures, because they contain more synonyms of "accident" and "reporting" than QUAL-047 itself does. The correct chunk sits at rank 12.

BM25 alone returns the correct chunk at rank 1 — QUAL-047 is a rare token that appears in only that document — but ranks poorly on paraphrased questions like "How do employees flag a workplace accident?" where the query and the target share no exact tokens.

Neither alone is enough. Together, they cover both cases.

Reciprocal-rank fusion: merging two orderings

The naive approach — add the dense and lexical scores — fails because the two scales are unrelated. A cosine score sits in $[-1, 1]$, a BM25 score is unbounded and depends on corpus statistics. Normalisation helps, but the tuning is fragile.

Reciprocal-rank fusion (RRF) sidesteps the problem by looking only at ranks:

$$\text{RRF}(d) = \sum_{r \in \text{rankers}} \frac{1}{k + \text{rank}_r(d)}$$

with $k = 60$ as the standard smoothing constant. A document that appears in the top 5 of both rankers gets a much higher score than one that appears only in one, no matter what the raw scores were.

def rrf(rank_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    scores = {}
    for ranks in rank_lists:
        for i, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + i + 1)
    return sorted(scores.items(), key=lambda x: -x[1])


def hybrid_search(question: str, k: int = 20):
    dense = [c["chunk_id"] for c, _ in dense_search(question, k=40)]
    lex   = [c["chunk_id"] for c, _ in bm25_search(question, k=40)]
    fused = rrf([dense, lex])[:k]
    return [chunk_by_id(cid) for cid, _ in fused]

Give each ranker twice the final $k$ so that documents shared between the two make it into the fusion. On the red thread, plain hybrid via RRF lifted recall at $k=5$ from 0.78 (dense only) to 0.89 — the largest single improvement of the course.

Short queries: rewriting the question

A user typing "leave" wants information about paid leave. Dense retrieval returns anything vaguely related to holidays, sabbaticals or absences; BM25 returns any chunk that contains the word. Both are drowning in noise because the query is too short to disambiguate.

Query rewriting uses the language model itself to expand a laconic question into a fuller one before retrieval:

REWRITE = """You rewrite short queries into full search questions.
Keep the user's language. Do not answer. Output only the rewritten query.

User query: {q}
Rewritten:"""

def rewrite(q: str) -> str:
    if len(q.split()) > 6:
        return q
    return llm.generate(REWRITE.format(q=q)).strip()

Rewriting only when the query is short saves cost and prevents the model from over-editing already clear questions. A variant, HyDE (Hypothetical Document Embeddings), goes further: the model writes an imagined answer to the query, and the search runs on the imagined answer's embedding. HyDE helps on very abstract questions, hurts on precise ones — evaluate it on your own set, do not enable it by default.

Multi-query and query decomposition

Some questions have multiple intents packed into one sentence: "What is the retention period for accident reports and who signs them?". Rewriting cannot help; the two topics need two separate searches.

DECOMPOSE = """Split the following user question into 1 to 3 independent
search queries. Return one query per line, in the user's language.

Question: {q}
Queries:"""

def decompose(q: str) -> list[str]:
    out = llm.generate(DECOMPOSE.format(q=q))
    return [line.strip() for line in out.splitlines() if line.strip()][:3]

def multi_query(question: str, k: int = 20):
    sub_queries = decompose(question)
    all_ranks = [
        [c["chunk_id"] for c in hybrid_search(sq, k=40)] for sq in sub_queries
    ]
    return rrf(all_ranks)[:k]

Multi-query is cheap in tokens — the decomposition is a few dozen tokens — and expensive in latency, because it multiplies retrieval calls. Enable it only when the deployed answers show blended sub-answers or "half-empty" responses.

A rewriter that loses named entities is worse than none

Test rewrites on 30 real queries containing identifiers (QUAL-047, 2024-Q3, a person's name). If the rewriter drops or paraphrases even one, adjust the prompt or fall back to the raw query when an identifier is detected. A rewrite that turns "QUAL-047 retention" into "document retention policy" is a regression, not an improvement.

In summary

• Dense retrieval wins on paraphrase; BM25 wins on rare terms, identifiers and very short queries — and the two failure modes are the ones users actually type.
• Reciprocal-rank fusion combines the two orderings by rank, sidestepping the incompatible score scales; on the red thread it added 11 points of recall over dense-only.
• Rewriting short queries and decomposing multi-intent questions before retrieval are cheap fixes that address the two most common query shapes users produce.
• Watch out for the forgotten acronym: a rewriter or HyDE step that paraphrases identifiers turns a solvable question into an unsolvable one.

Next module: taking a hybrid shortlist of 20 candidates and reordering it with a slower but sharper model, so the top 6 sent to the language model are the ones that actually deserve to be there.