Module 4 — Word2Vec, GloVe and the geometry of meaning

Module 3 left us with words as isolated columns: phone and handset are as far apart in the TF-IDF matrix as phone and banana. This module gives every word a dense vector in a low-dimensional space, learned from raw text, in such a way that related words end up close together. It is a shift that changed everything in the field between 2013 and 2017, and it is still the correct answer for many production systems where a transformer is overkill.

From sparse column to dense vector

A TF-IDF column for phone is a 25 000-dimensional vector: a 1 wherever the word appears, 0 elsewhere. It carries no information about the word itself, only about where it lives in the corpus.

A Word2Vec vector for phone is a 300-dimensional dense vector of small real numbers, learned so that words appearing in similar contexts end up with similar vectors. phone, handset, smartphone and mobile land close together because they are used in similar sentences. banana lands far away.

Concretely you go from a 25 000-column matrix per word to 300 numbers per word. Storage drops by a factor of a hundred, and — the real prize — comparisons between words now say something.

Skip-gram: predict the neighbours from the word

The Skip-gram model is a shallow neural network with one hidden layer and a very specific task: given a word, predict the words that appear around it in a small window (typically 5 words on each side).

Take the sentence the battery of the phone lasts twelve hours. If the target word is phone, the model is asked to raise the probability of the neighbours the, battery, of, lasts, twelve, hours and to lower everything else. Do this on a corpus of billions of words and the hidden layer weights become the word embeddings — a matrix where row i is the vector for word i.

The network itself is thrown away at the end. Only the weights matter.

CBOW: predict the word from the neighbours

Continuous Bag of Words is the reverse task: given the surrounding words, predict the missing centre word. Same architecture, same training set, symmetric objective. On small corpora, CBOW is faster to train and slightly better on frequent words; Skip-gram tends to win on rare words and larger corpora.

Both share a scaling problem. Predicting a probability over a 100 000-word vocabulary at every training step requires a softmax over 100 000 outputs. On a billion-word corpus this becomes the bottleneck of training. Negative sampling replaces the softmax with a much cheaper trick.

Negative sampling in one paragraph

Instead of asking the model to raise the probability of the true neighbour among 100 000 candidates, negative sampling asks it to distinguish the true neighbour from a handful of randomly drawn wrong ones, typically 5 to 20. The problem becomes binary classification: is this pair (word, context) a real one from the corpus, or did I make it up?. The loss involves only $k+1$ output units instead of the full vocabulary, and the words drawn as negatives are sampled with a probability proportional to their frequency raised to the power $0.75$ — a value picked empirically to slightly downweight the most common words.

The result is a training procedure that runs on a laptop for a corpus that used to require a cluster.

GloVe: a matrix factorisation with the same output

GloVe (Global Vectors) reaches similar embeddings from a different angle. It builds the co-occurrence matrix across the whole corpus — how often word $i$ appears within a window of word $j$ — and finds low-dimensional vectors $u_i$ and $v_j$ such that

$u_i \cdot v_j + b_i + b_j \approx \log \mathrm{count}(i, j)$

for the frequent pairs. In practice GloVe and Word2Vec produce vectors of similar quality on the same downstream tasks; GloVe is often faster because it works on the pre-aggregated matrix rather than streaming through the corpus.

Loading and using pretrained vectors

For 99 % of applications you do not train these vectors yourself: you download embeddings pretrained on a corpus much bigger than yours and use them directly.

import gensim.downloader as api

vectors = api.load("glove-wiki-gigaword-300")   # ~ 400k words, 300 dims

print(vectors.similarity("phone", "handset"))    # ~ 0.6
print(vectors.similarity("phone", "banana"))     # ~ 0.05
print(vectors.most_similar("battery", topn=5))

The last line returns the five closest words in the space: on a Wikipedia-trained model, expect batteries, charger, lithium, charging, cell. This is what you use to enrich a search bar, expand a query, or spot near-duplicate customer reviews.

Analogies and their limits

The most-photographed property of these embeddings is the analogy: king - man + woman ≈ queen. Geometry captures a relation. Paris - France + Italy ≈ Rome works too. It is genuinely surprising, and it is the one demo everyone remembers.

It is also less robust than it looks. Beyond the twenty analogies in the original paper, the accuracy drops sharply. plumber - man + woman ≈ nurse on many pretrained embeddings — not because the model reasons about occupations, but because the training corpus wrote about male plumbers and female nurses often enough for the geometry to encode the stereotype.

Biases baked in the space

This is the single most important point of the module. Word embeddings inherit the statistics of their training corpus, including its social biases, and they do so in ways that are hard to see until you look for them.

Reproducible on public embeddings:

• doctor - man + woman ≈ nurse on glove-wiki-gigaword-300
• Female names cluster closer to words about family and appearance; male names closer to words about career and money.
• Ethnic names carry sentiment signals that reflect the sentiment of their appearances in text.

The vector geometry does not create these correlations; it exposes them, in a numerical form that a downstream classifier will happily amplify. A CV screener built on top of these embeddings will penalise female names for engineering roles, not because anyone wrote a rule but because the geometry encoded the historical over-representation of one group in the training text.

Do not use raw pretrained embeddings for high-stakes decisions

Debiasing techniques exist (subtracting the projection onto a gender direction, hard debiasing, counterfactual data augmentation), and none of them fully remove the problem. For hiring, credit, healthcare and any decision that materially affects a person, embeddings are a diagnostic tool, not a decision layer.

Static embeddings still have their moment

For similarity search, query expansion, clustering of short texts and any task where the sentence is short and the word does not change meaning with context, Word2Vec or GloVe are cheaper, faster and often good enough. The move to contextual embeddings (module 5) matters most when polysemy does — bank in a finance review versus a river review, apple the company versus the fruit.

In summary

• Word2Vec and GloVe replace a 25 000-column sparse vector with a 300-dimensional dense vector, learned so that words used in similar contexts end up close together.
• Skip-gram and CBOW are two symmetric training tasks; negative sampling and matrix factorisation are two ways to make the training tractable at billion-word scale.
• Analogies work impressively on the classic examples and less well beyond; the geometry captures a piece of meaning, not a full reasoning capability.
• Biases in the corpus become biases in the space, in a form a downstream classifier will amplify; static embeddings are unsafe for high-stakes decisions without an explicit audit.

Next module: contextual embeddings, where the same word finally gets a different vector depending on the sentence around it.