#embeddings-3d — Neural networks

A word becomes a vector: close in space = close in meaning, and you can do math on them.

What you'll play with

Welcome to #embeddings-3d. Each point is a word, and its position is not arbitrary: a language model turns each word into a vector (a list of numbers), and two words with similar meaning end up close in space. This is a simplified 3D projection, built to be readable — real embeddings have 768 dimensions or more. The scene rotates on its own; type /rotate to stop it.
Let's start with one word. Type /word cat: I highlight it and compute its 5 nearest neighbours (cosine similarity).
Change family: /word Paris. Its neighbours are the other capitals… and notice France just below, in the blue cloud of countries.
Type /links to draw a line from each country to its capital. The lines are all parallel: the relation "capital of" is ONE direction in space. Same for king → queen, man → woman, prince → princess.
Since "capital of" is an arrow, we can copy it onto another country. Type /analogy Paris France Italy: I compute Paris − France + Italy and look for the closest word to the result.
This is arithmetic on meaning. The historical example from word2vec (Mikolov, 2013): /analogy king man woman. The model never read a dictionary, it just saw billions of sentences.
To isolate one cloud, filter: /cluster emotions. The other families fade. Then explore freely: /word joy, /distance joy sadness, /distance cat Paris…
You have seen it all: a word = a vector, close = similar meaning, a relation = a direction. LLMs do exactly this, but with thousands of dimensions and whole sentences — it is the very first layer of every transformer, the one that feeds the neurons of the #neuron channel. Keep playing: /analogy Tokyo Japan Canada, /word doctor, /cluster all, /reset.

Channel commands

/word <word> — Highlights a word and shows its 5 nearest neighbours (cosine).
/analogy <a> <b> <c> — Computes a − b + c and finds the closest word to the result.
/cluster <animals|cities|countries|royalty|emotions|jobs|all> — Keeps only one family of words in colour.
/links — Shows or hides the country→capital and male→female lines.
/rotate — Starts or stops the scene's auto-rotation.
/distance <a> <b> — Cosine similarity between two words, with interpretation.
/reset — Clears selection, analogy and filter; restarts rotation.

Glossary

Embedding: Representation of a word (or a token, an image…) as a vector of numbers, learned so that items close in meaning have close vectors.
Vector space: The set of all possible vectors. A 300-dimensional embedding lives in a 300-dimensional space; here we project into 3D so we can see it.
Cosine similarity: Cosine of the angle between two vectors: 1 = same direction, 0 = orthogonal, −1 = opposite. Standard measure of semantic closeness, insensitive to length.
Vector analogy: king − man + woman ≈ queen: relations (gender, capital, plural) become constant directions in the space.
Word2vec: Method (2013) that learns embeddings by predicting a word from its neighbours (CBOW) or the reverse (skip-gram). "You shall know a word by the company it keeps."
Distributional hypothesis: Two words that appear in similar contexts have similar meanings. This is the principle that lets us learn meaning without a dictionary.
Dimension: Number of coordinates in a vector. More dimensions = more nuance representable, but more data needed. Typically 100 to 4,096.
Dimensionality reduction: Projecting high-dimensional vectors into 2D or 3D so we can visualize them (PCA, t-SNE, UMAP), losing as little information as possible.
Nearest neighbors: The words whose vector is closest to a given word. Listing them is the simplest test of an embedding's quality.

Other channels in Neural networks

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.