Lesson 3 — Embeddings: meaning as geometry
Tokenisation gave us numbers, and those numbers are meaningless — token 3797 has no relationship to token 3798. Embeddings are where meaning enters, and they are arguably the most important idea in modern NLP.
Before embeddings: counting
The first workable numerical representation was the bag of words. Build a vocabulary, and represent a document by how many times each word appears. "Bag" because order is discarded entirely.
It worked better than it deserved to. Spam filters built this way were genuinely effective, because spam really does use distinctive vocabulary, and it remains a reasonable baseline for topic classification today.
Its limits are structural:
Order is gone. "The dog bit the man" and "The man bit the dog" produce identical representations.
Synonyms are strangers. "Car" and "automobile" are as unrelated as "car" and "asparagus", because they are simply different columns.
The vectors are enormous and almost entirely zero. A vocabulary of fifty thousand words gives a fifty-thousand-dimensional vector with perhaps forty non-zero entries.
No notion of similarity. Every word is equally distant from every other, which means the representation carries no semantic information at all.
A refinement called TF-IDF improves the weighting by down-weighting words that appear everywhere, and it does not touch the fundamental problem: the representation knows nothing about meaning.
The idea that changed everything
The breakthrough came from a linguistic observation predating computers: a word is characterised by the company it keeps. Words appearing in similar contexts tend to mean similar things.
"Cat" and "dog" both occur near "pet", "feed", "vet" and "adopt". "Coffee" and "tea" both occur near "drink", "cup", "hot" and "morning". You can learn a great deal about a word purely from its neighbours, and crucially you can learn it without any labelled data — the text supplies its own supervision.
Word2vec, in 2013, made this practical. Train a small model on the task "predict the surrounding words from this word", and the internal representations it develops turn out to place semantically related words near each other. The prediction task itself is throwaway; the representations are the point.
Each word becomes a vector of a few hundred numbers, and — as the maths course described — that vector is a point in space. Similar meanings, nearby points.
The structure nobody put there
The resulting space has properties that were not designed:
- Related words cluster: countries together, colours together, verbs of motion together.
- Similarity becomes arithmetic: the dot product measures relatedness directly.
- Directions acquire meaning. There is roughly a direction for plural, for tense, for gender.
Hence the much-quoted example: the vector for "king", minus "man", plus "woman", lands near "queen". Nobody encoded that relationship; it emerged from co-occurrence statistics alone.
That result deserves a caveat that is usually omitted. These analogies work on a curated set of examples and fail on many others, and the same mechanism reproduces harmful associations from the training text with equal fidelity — the "doctor is to man as nurse is to woman" pattern is a documented, reproducible property of these spaces, not an anomaly. The ethics course treats this properly.
The problem with fixed embeddings
Word2vec gives each word one vector, permanently. Which is plainly wrong:
"I sat on the river bank." "I deposited it at the bank."
Same vector for both, so the representation has to average two unrelated meanings into one point that fits neither.
More broadly, a fixed embedding cannot represent that meaning shifts with use: "light" as weight versus illumination, "run" as movement versus operating a business.
Contextual embeddings
The fix, arriving with BERT in 2018 and now universal, is to compute the vector from the sentence, not from a lookup table.
A transformer reads the whole passage and produces a representation for each token that depends on its context. "Bank" near "river" gets one vector; "bank" near "deposit" gets a different one. Ambiguity is resolved by construction rather than left to be averaged away.
This is the attention mechanism from the deep learning course doing its job: each token attends to the others and incorporates what is relevant.
| Fixed embeddings (word2vec) | Contextual embeddings (BERT and later) | |
|---|---|---|
| Vector per word | one, always | one per occurrence, from context |
| Ambiguity | averaged into a single point | resolved by the surrounding text |
| Computation | a table lookup, instant | a forward pass through a model |
| Storage | one small table | the whole model |
| Quality | useful | substantially better |
What embeddings are used for
Beyond feeding a model, embeddings are a product in themselves, and the applications share one mechanism: similarity is a dot product.
Semantic search. Embed the query and every document, return the nearest. Unlike keyword search, this finds "how do I reset my password" when the document says "credential recovery procedure", because the meanings are close even though no word matches.
Retrieval-augmented generation. The retrieval half of RAG is exactly this: embed a question, find the relevant passages in your own documents, and hand them to a language model as context. It is how you get a model to answer from your data rather than from its training. The premium RAG course builds one.
Clustering and deduplication. Group similar support tickets, find near-duplicate documents, detect plagiarism.
Recommendation. Embed articles and users, and match by proximity.
Classification with very little data. Embed your few examples, embed the new item, and assign the nearest label. This works surprisingly well when you have twenty examples and no budget to train anything.
Vector databases exist to make this fast at scale: specialised stores that find the nearest vectors among billions in milliseconds.
Semantic search, RAG, recommendation, deduplication and few-shot classification are all the same operation: embed, then compare by dot product. Recognising that they are one idea rather than five saves a great deal of confusion when reading about them separately.
In three sentences
Counting words gave usable but meaning-blind representations, and embeddings replaced them by learning, from raw text alone, to place words used in similar contexts near each other in space. Fixed embeddings gave each word one vector regardless of use, which contextual embeddings from transformers fixed by computing a different vector per occurrence based on the surrounding sentence. Because similarity becomes a dot product, one mechanism underpins semantic search, RAG retrieval, clustering, recommendation and few-shot classification alike.
Next — Lesson 4: the tasks →