Skip to main content

Module 3 — Norms, distances and cosine similarity

Whenever a model must say "these two things are alike" — recommend a product, group customers, retrieve a document — it measures a distance or a similarity between vectors. This module gives the three tools that underpin vector search, clustering and modern embeddings.

The norm: the size of a vector​

The norm measures the "length" of a vector. The most common, the Euclidean norm (called L2L_2), generalizes the Pythagorean theorem:

∥x∥2=x12+x22+⋯+xd2\|x\|_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_d^2} 
import numpy as np
x = np.array([3, 4])
np.linalg.norm(x)        # 5.0 — the classic 3-4-5 triangle

The norm is used to normalize a vector (bring it to length 1 by dividing by its norm), an omnipresent operation: it lets you compare directions regardless of scale. You also meet the L1L_1 norm (sum of absolute values), which will play a key role in model regularization.

Euclidean distance: "as the crow flies" closeness​

The distance between two observations is the norm of their difference:

d(a,b)=∥a−b∥2d(a, b) = \|a - b\|_2 
a = np.array([72, 3, 15])
b = np.array([90, 4, 8])
np.linalg.norm(a - b)    # distance between two homes

This is the distance used by k-nearest-neighbors and by k-means clustering. It has a major pitfall: it is dominated by large-amplitude features. Here area (tens) crushes the number of bedrooms (units): the distance reflects almost only area. Hence the need to bring features to the same scale before any distance computation — a point module 6 and the feature-engineering course return to at length.

Manhattan distance: by the streets, not the crow's flight​

The Manhattan distance (L1L_1) adds up absolute differences, like a taxi following the grid of streets:

d1(a,b)=∑i∣ai−bi∣d_1(a, b) = \sum_i |a_i - b_i|

Less sensitive to extreme values than Euclidean distance, it is sometimes preferred in high dimension. The Euclidean/Manhattan choice is a genuine tuning lever depending on the data.

Cosine similarity: comparing directions, not lengths​

Often what matters is not distance but orientation. Two texts about the same topic have vectors pointing in the same direction, even if one is much longer (a wordier document). Cosine similarity measures the angle, ignoring lengths:

cos⁡(θ)=a⋅b∥a∥ ∥b∥\cos(\theta) = \frac{a \cdot b}{\|a\|\,\|b\|} 
def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

It equals 1 for two vectors of the same direction, 0 for two perpendicular ones (unrelated), −1 for opposite directions. The numerator a⋅ba \cdot b is the dot product, the building block that measures how much two vectors "go the same way."

Why cosine rules over embeddings

Modern models — semantic search, RAG systems, recommendation — represent text and images as high-dimensional vectors (embeddings). We almost always compare them by cosine similarity, because only direction carries meaning; vector length merely reflects artifacts like text size. When you read that a vector database "finds the nearest neighbors," this heavily optimized computation is what runs underneath.

Summary​

  • The L2L_2 norm measures a vector's length; normalization brings it to length 1 to compare directions.
  • Euclidean distance measures "as the crow flies" closeness; it is dominated by large-amplitude features, hence the importance of scaling.
  • Manhattan distance (L1L_1) sums absolute differences, more robust to extreme values.
  • Cosine similarity compares directions while ignoring lengths; it is the go-to measure for embeddings and semantic search.

Next module: eigenvalues and dimensionality reduction — how to compress data with hundreds of features while keeping the essential.