Lesson 2 — Linear algebra
If you learn one branch, learn this one. Linear algebra is the language in which both data and models are written, and understanding it resolves a large share of the confusion — and the errors — you will meet in practice.
Everything becomes a vector
A vector is an ordered list of numbers. That is the whole definition. What makes it powerful is the second reading: a list of four numbers is also a point in four-dimensional space, and a list of a thousand numbers is a point in a thousand-dimensional space.
This dual reading is the key idea of the lesson, because it turns questions about data into questions about geometry.
- A customer becomes
[34, 2, 1520, 7]— age, number of orders, total spend, months since signup. A point in four-dimensional space. - A grayscale image of 28 by 28 pixels becomes 784 brightness values. A point in 784-dimensional space.
- A word becomes a list of a few hundred numbers, learned rather than chosen. A point in that space, positioned so that related words sit near each other.
Once every object is a point, "which customers resemble each other" becomes "which points are close together", and that is a question with a precise answer.
Nobody visualises 784 dimensions, and nobody needs to. Every operation you use is defined identically in two dimensions and in seven hundred, so you can reason in two and trust the arithmetic in the rest. This is why every diagram in this field is drawn in 2D.
Matrices as transformations
A matrix is a rectangular grid of numbers, and it has two useful readings too.
As data: a table. Rows are examples, columns are features. A thousand customers with four attributes each is a 1000 by 4 matrix. This is the reading pandas gives you.
As a transformation: a matrix is a machine that takes a vector in and gives a different vector out. Multiplying by a matrix can rotate space, stretch it along some axes, squash it, or project it into fewer dimensions. This is the reading that explains neural networks.
Here is why that second reading matters so much. A neural network layer is a matrix multiplication followed by a simple non-linear function. That is essentially the whole architecture:
output = activation(W · input + b)
W is a matrix of learned weights, b a vector of learned offsets, and activation a simple function applied element by element. Stack a few dozen of those and you have a deep network. Training means adjusting the numbers inside W and b.
This also explains the hardware. Matrix multiplication is thousands of independent multiply-and-add operations, and a graphics processor exists precisely to do thousands of independent arithmetic operations at once. GPUs did not become AI hardware by coincidence; deep learning is matrix multiplication, and GPUs are matrix multipliers.
The dot product: why similarity is arithmetic
The dot product of two vectors multiplies them element by element and sums the results. A tiny operation with an outsized role.
Its geometric meaning is what matters: the dot product tells you how much two vectors point in the same direction.
- Pointing the same way: large positive value
- At right angles, unrelated: zero
- Pointing opposite ways: negative
Because it measures alignment rather than distance, and because vector length varies for uninteresting reasons, the usual practical form is cosine similarity: the dot product divided by both lengths, giving a score between -1 and 1 that depends only on direction.
This single operation is behind a surprising amount of modern AI:
- Recommendation: represent users and films as vectors, and a high dot product predicts a good match
- Semantic search: represent your query and every document as vectors, and return the highest cosine similarity
- Attention in transformers: every position computes dot products against every other position to decide what to pay attention to
- Vector databases: the entire product category exists to compute these similarities quickly over billions of vectors
If you take one concrete thing from this lesson: similarity is a dot product. When you later read that a retrieval system "embeds documents and finds the nearest neighbours", you are reading about this operation, at scale.
Embeddings: learned geometry
An embedding is a vector representation that was learned rather than designed, and it is where the geometry becomes genuinely interesting.
Suppose you want to represent words numerically. Assigning arbitrary numbers achieves nothing, because the numbers carry no relationships. Instead, a model is trained on enormous quantities of text to place each word so that words used in similar contexts end up nearby.
The resulting space has structure nobody put there deliberately. Related words cluster. Directions acquire meaning: there is a direction that roughly corresponds to plural, another to grammatical tense, another to gender. The much-quoted example is that the vector for "king", minus "man", plus "woman", lands near "queen" — arithmetic on meaning, emerging from nothing but co-occurrence statistics.
The same principle applies far beyond words. Products, users, images, molecules and audio clips are all routinely embedded, and once they are, similarity search and clustering work identically for all of them.
Dimensionality reduction, in one paragraph
Data often has many features, most of them redundant. Principal component analysis finds the directions along which the data varies most and keeps only those, discarding directions where nothing much happens. Fifty correlated features become five that preserve most of the information.
This is the practical use of eigenvalues and eigenvectors — the directions a transformation stretches without rotating. It is worth understanding conceptually. It is not worth computing by hand.
The vocabulary, and what each thing is
| Term | What it is | Where you meet it |
|---|---|---|
| Scalar | a single number | a learning rate, a loss value |
| Vector | a list of numbers, a point in space | one example, one embedding |
| Matrix | a 2D grid | a dataset, a layer's weights |
| Tensor | the same idea in any number of dimensions | a batch of colour images: 4 dimensions |
| Shape | the size along each dimension | the thing to print when confused |
| Transpose | flipping rows and columns | making shapes line up for multiplication |
| Dot product | element-wise multiply then sum | similarity, attention, every layer |
| Norm | the length of a vector | regularisation, gradient clipping |
Matrix multiplication requires the inner dimensions to match: an m × n matrix can only multiply an n × p one. Shape mismatch is far and away the most common error in deep learning code, and the fix is nearly always to print the shapes and find where your mental model diverged from reality.
In three sentences
A vector is both a list of numbers and a point in space, which turns questions about data into questions about geometry, and a matrix is both a table and a transformation of that space. A neural network layer is a matrix multiplication plus a simple non-linear function, which is why deep learning runs on graphics processors built to multiply matrices. The dot product measures how much two vectors point the same way, and that single operation underpins recommendation, semantic search, attention and every vector database.
Next — Lesson 3: calculus →