Module 4 — Embeddings and vector databases

Module 3 gave us chunks with their metadata. This module turns each chunk into a vector, stores those vectors in an index that answers "give me the closest 20" in milliseconds, and lets us filter the search on the metadata we spent module 2 preserving. Three choices — the model, the index, the filter — decide most of what will succeed later.

What an embedding actually encodes

An embedding model reads a passage and outputs a fixed-size vector of floating-point numbers. Two passages are considered semantically close when their vectors are close, using a similarity measure — almost always the cosine of the angle between them, which we saw in course 03 module 3.

Formally, for two vectors $u$ and $v$ of dimension $d$:

$$\text{cos}(u, v) = \frac{\sum_{i=1}^{d} u_i v_i}{\sqrt{\sum_i u_i^2} \sqrt{\sum_i v_i^2}}.$$

Numbers range from $-1$ (opposite meaning) to $1$ (identical meaning), with $0$ meaning unrelated. Most modern models produce unit-norm vectors, so cosine reduces to a plain dot product, which is cheaper.

Meaning here is what the model was trained on. A model trained on generic web text places two sentences close together when they discuss the same everyday topic; a model trained on scientific abstracts places them close when they discuss the same concept, even in different words. The choice of model decides what "similar" means for you.

Choosing an embedding model on three axes

There is no single best model. Three axes matter:

Axis	Range on the market	What to pick for the red thread
Language coverage	English-only, bilingual, multilingual (100+)	Multilingual — procedures mix languages
Dimension	384 to 3072	768 is the sweet spot for accuracy vs storage
Licence	Open weights, hosted API	Open weights — internal procedures leave the network

For the internal-procedures assistant, intfloat/multilingual-e5-base is a defensible default: 768 dimensions, open Apache licence, works acceptably on 100+ languages, runs on CPU for the corpus size we target. bge-m3 and nomic-embed-text-v1.5 are two other reasonable choices; benchmarks tighten every quarter, so re-measure yearly rather than chasing headlines.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/multilingual-e5-base")

def embed(texts: list[str], is_query: bool = False) -> list[list[float]]:
    prefix = "query: " if is_query else "passage: "
    return model.encode(
        [prefix + t for t in texts],
        normalize_embeddings=True,
    ).tolist()

Note the prefix: E5 was trained to distinguish "query" from "passage" via a leading word. Omitting it degrades recall by 10 to 15 points without any error message. This kind of model-specific detail is exactly what makes the "swap the model" refactor less trivial than it looks.

From vectors to an index: HNSW in one paragraph

Storing 300 000 vectors of 768 floats is 900 MB of RAM. Finding the 20 closest to a query by naive comparison is fast enough at that scale, but grows linearly with the corpus. HNSW (Hierarchical Navigable Small World) is the standard approximate-nearest-neighbour index: it organises vectors as a graph where each node holds pointers to its closest neighbours at several "zoom levels". A query starts at the coarsest level and greedily walks toward the query vector, dropping down levels as it approaches. In practice, sub-millisecond queries on tens of millions of vectors, at 95 to 99 % recall of the true top-k.

Two knobs control the trade-off: M (average number of neighbours per node — higher gives better recall and more memory) and ef_construction at build time / ef at search time (how many candidates to visit — higher gives better recall and slower queries).

Chroma: the file-backed default

import chromadb

client = chromadb.PersistentClient(path="./chroma-procedures")
coll = client.get_or_create_collection(
    name="procedures",
    metadata={"hnsw:space": "cosine"},
)

coll.add(
    ids=[c["chunk_id"] for c in chunks],
    documents=[c["text"] for c in chunks],
    embeddings=embed([c["text"] for c in chunks]),
    metadatas=[
        {
            "source_name": c["source_name"],
            "page": c["page"],
            "section": c["section"],
            "language": c["language"],
            "access_class": c["access_class"],
        }
        for c in chunks
    ],
)

Chroma stores everything in a local directory, is trivial to set up, and scales comfortably to a few million chunks on a laptop. It is the right choice for the red thread and for the exercises of this course.

pgvector: the "we already have Postgres" default

When the organisation already runs PostgreSQL, adding the vector extension avoids introducing a new database.

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
    chunk_id     TEXT PRIMARY KEY,
    document_id  TEXT,
    section      TEXT,
    language     TEXT,
    access_class TEXT,
    text         TEXT,
    embedding    vector(768)
);

CREATE INDEX chunks_hnsw
  ON chunks
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

The retrieval query becomes plain SQL, with the filters expressed in WHERE:

SELECT chunk_id, section, text
FROM chunks
WHERE language = 'en' AND access_class <= 2
ORDER BY embedding <=> $1
LIMIT 20;

The operator <=> is cosine distance; smaller is better. Below a few million vectors, pgvector's HNSW is comparable to Chroma in latency, and the joins with your existing tables (users, permissions, audit) become trivial.

Metadata filters: the classic forgotten step

The most common failure diagnosed after "chunk splits a table" is a metadata filter that was never applied. A user asks a question in the context of the sales department; the retriever brings back a passage from the HR handbook that is close in the embedding space but not applicable. The passage is factually accurate, the model composes a fluent answer, and the human reads a policy that does not apply to them.

The two fixes work together. First, at ingestion, attach every filterable attribute — language, department, access class, effective date, document status — to the chunk's metadata. Second, at every query, compute the filter from the user's context and pass it explicitly:

def retrieve(question: str, user_context: dict, k: int = 20):
    q = embed([question], is_query=True)[0]
    return coll.query(
        query_embeddings=[q],
        n_results=k,
        where={
            "language": user_context["language"],
            "access_class": {"$lte": user_context["clearance"]},
            "status": "in_force",
        },
    )

A pipeline that omits the where= clause "just to test" is a pipeline that ships that omission to production. Wrap the retriever in a small helper that refuses to run without a user_context, and the failure mode disappears.

Updates and deletions without rebuilding the index

Corpora change: a procedure is revised every few weeks, one is retired, a new note is added. The naive approach — drop the collection and reindex everything — is 2 to 3 hours on a laptop for 300 documents and grows linearly.

Both Chroma and pgvector support incremental updates:

coll.upsert(
    ids=["QUAL-047#3.2"],
    documents=[new_text],
    embeddings=embed([new_text]),
    metadatas=[{"source_name": "QUAL-047.pdf", "revision": "v3"}],
)

coll.delete(where={"source_name": "PROC-obsolete.pdf"})

The key is a stable chunk ID derived from the source: for example, document_id + "#" + section + "#" + hash(text[:64]). When a document is re-extracted, unchanged chunks keep their ID (no work), changed chunks are upserted (re-embedded and rewritten), and removed chunks are deleted. Module 9 comes back to this under the name "incremental reindexing".

Test the filter before you trust the answer

Before evaluating the assistant end to end, evaluate the retriever in isolation: for each of 20 questions with an obvious right document, check that the top-5 hits belong to that document and satisfy the filter. If they do not, no reranker or prompt engineering downstream will save the answer.

In summary

• An embedding maps a passage to a vector; cosine similarity compares meaning, and modern models return unit-norm vectors so cosine reduces to a dot product.
• Choose the model on language coverage, dimension and licence; for internal content prefer open weights, and respect the model's expected query/passage prefix.
• Store vectors in an HNSW index — Chroma when there is no database, pgvector when Postgres is already in the stack — with metadata attached to every chunk.
• The most common silent failure is a forgotten metadata filter; wrap retrieval in a helper that requires a user_context, and update the index incrementally using stable chunk IDs.

Next module: turning "close in vector space" into "answers the actual question" — the dense, lexical and hybrid search that makes retrieval robust to short queries and acronyms.