Skip to main content

Module 5 — Vector stores and retrievers

Module 4 chunked the reimbursement policy. This module turns those chunks into a retriever — the object the assistant will ask "what does the policy say about a 42 EUR dinner in Berlin?" — and wires it into the first end-to-end chain of the course.

The VectorStore interface​

Every vector store LangChain supports — Chroma, FAISS, PGVector, Qdrant, Weaviate, Milvus and a dozen others — implements the same interface. Two methods are enough for 90 % of the work: from_documents (or add_documents) to index, and as_retriever to expose it to a chain.

from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-mpnet-base-v2",
)

store = Chroma.from_documents(
    documents=policy_chunks,       # from module 4
    embedding=embeddings,
    collection_name="policy",
    persist_directory="./chroma_policy",
)

Two picks to make explicitly, once.

The embedding model decides retrieval quality more than the chunker does. In 2026, all-mpnet-base-v2 is a fair open-model baseline; bge-m3 is a strong multilingual upgrade; OpenAI's text-embedding-3-small is the API baseline. Never mix embedding models in one collection: the vectors live in incompatible spaces and cosine similarity between them is meaningless.

The store depends on scale, not on tutorials. Under a few hundred thousand chunks, Chroma with a persist directory is the pragmatic default: one line of code, on-disk persistence, no server. At millions of chunks or shared across services, prefer PGVector or Qdrant. FAISS remains the reference for pure-Python in-memory speed but does not persist filters and metadata as cleanly as the others.

Similarity search and score thresholds​

Once indexed, the store retrieves.

results = store.similarity_search_with_score(
    "meal ceiling for internal travel", k=4,
)
for doc, score in results:
    print(score, doc.metadata, doc.page_content[:80])

Two parameters decide precision. k, the number of neighbours to return — 4 is a sensible default; going above 8 stuffs the prompt and dilutes attention. And a similarity threshold: without it, a question with no answer in the corpus still returns the four "least bad" chunks, and the model composes a confident wrong answer on top of them.

retriever = store.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.35, "k": 4},
)

The right threshold depends on the embedding model and the corpus — set it empirically on 20 to 50 real questions. It should let obviously relevant chunks through and drop obviously off-topic ones. Setting it too high starves the model; too low is exactly what returns those confident wrong answers.

MultiQueryRetriever: paraphrasing the question​

A user asks "can I expense a business lunch?". The chunk that answers this is titled "Meal ceiling for internal travel". Cosine similarity between the two may not clear the threshold. The fix is to paraphrase the query and search under each paraphrase.

from langchain.retrievers.multi_query import MultiQueryRetriever

mqr = MultiQueryRetriever.from_llm(
    retriever=retriever,
    llm=model,        # a small, cheap model is enough
)

MultiQueryRetriever asks the model for three or four alternate phrasings, runs the underlying retriever on each and deduplicates. It doubles the retrieval latency and triples the vector-store call count, but it lifts recall on user-worded questions substantially. Turn it on when your evaluation set of module 9 shows a recall problem; do not add it prophylactically.

Other retrievers worth knowing by name: EnsembleRetriever (combine a vector retriever with a BM25 retriever for hybrid search — course 18 explained why), ContextualCompressionRetriever (run a re-ranker on top), and ParentDocumentRetriever (retrieve small chunks, then hand the model the larger parent block — the hierarchical trick from course 18).

A minimal RAG chain​

Every retriever in LangChain is itself a Runnable, which means it plugs straight into a chain with the pipe operator.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

def format_docs(docs):
    return "\n\n".join(
        f"[{d.metadata.get('page', '?')}] {d.page_content}" for d in docs
    )

policy_prompt = ChatPromptTemplate.from_messages([
    ("system",
     "You answer questions about the internal reimbursement policy. "
     "Use ONLY the context below. If the context does not contain the "
     "answer, say 'The policy does not cover this' and cite no page. "
     "Cite the pages you used in square brackets, e.g. [4]."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

policy_chain = (
    {"context": retriever | format_docs,
     "question": RunnablePassthrough()}
    | policy_prompt
    | model
    | StrOutputParser()
)

policy_chain.invoke("Is a 42 EUR dinner reimbursable in Berlin?")

Three ideas concentrated in fifteen lines. The retriever fans out to fetch context. RunnablePassthrough keeps the original question available for the prompt. format_docs decides what the model sees — including the page number, so citations become verifiable. The abstention licence in the system prompt — "The policy does not cover this" — is the structural fix from course 18 that turns hallucinations into a detectable failure mode.

Test the retriever alone before the whole chain

Print the passages returned for ten real questions and read them by hand. If the retriever is wrong, the generator cannot save it. This ten-minute check saves the ten-hour debug session that starts with "why does the model invent policies?".

In summary​

  • Every LangChain VectorStore shares an interface; Chroma with a persist directory is the pragmatic default under a few hundred thousand chunks, PGVector or Qdrant at larger scale.
  • Never mix embedding models in one collection; pick the model once and rebuild the index if you change it.
  • A similarity threshold is what turns "no answer found" into a first-class outcome instead of a confident wrong answer over off-topic chunks.
  • MultiQueryRetriever paraphrases the query to lift recall on user-worded questions; add it when evaluation shows a recall problem, not by default.
  • A retrieval chain is the pipe of retriever → format → prompt → model → parser, with RunnablePassthrough keeping the question available for the final prompt.

Next module: giving the assistant memory across turns without letting the context window run away.