#pca — Unsupervised learning

Principal component analysis: find the axes where data varies most, project, compress — and measure what is lost.

What you'll play with

Welcome to #pca. On screen, 200 blue points form an elongated cloud shaped like a cigar, tilted in space. Three arrows leave its centre: pink, yellow, green. These are its principal components: the direction in which the cloud varies the most, then a bit less, then almost not at all. The length of each arrow tracks the standard deviation of the points along that direction (√λ). Starting question: in which direction does this cigar vary the most? PCA answers with three axes and three numbers, without any labels. Drag with the mouse to see the cloud in relief.
Each arrow carries a share of the cloud's total variance: λ1 / (λ1 + λ2 + λ3) for the first, and so on. Show these percentages next to the arrow tips: /variance.
Keep only the first two components: /components 2. An indigo plane appears: this is the (PC1, PC2) plane, the one that contains the most variance among all possible planes. That is where we will flatten the cloud.
Flatten the cloud: /project. Each point slides toward its shadow on the plane: from 3 coordinates, it keeps only 2.
Where did the lost information go? Show it: /reconstruct. Grey segments link each original point to its reconstruction.
Move the cloud away from the origin: /shift 2. The arrows follow the cloud without changing direction: centered PCA does not depend on where the cloud sits.
Change the shape of the cloud: /stretch z 3. The z standard deviation goes from 0.35 to 3: the "thinnest" direction of the cigar becomes the longest.
Maximum compression: /components 1. The plane becomes a line (PC1) and each point now has a single coordinate.
Your turn: /rotate 90 (the cloud pivots, the arrows follow), /noise 0.6 (noise inflates λ3: the "useless" component is not quite useless anymore), /seed 42 (another draw, almost identical axes), /shift 3 then /center (the non-centering mistake, in full), /components 3 (back to three dimensions: zero error), /reset to start over. Next: #t-sne-umap, where dimensionality reduction becomes non-linear, and #embeddings-3d, where vectors of hundreds of dimensions are brought down to three so we can look at them.

Channel commands

/components <1|2|3> — Number of components kept: line (1), plane (2) or all (3).
/project — Slides each point to its projection (or brings it back).
/reconstruct — Show or hide the point ↔ projection segments (the error).
/variance — Show or hide the explained variance next to each arrow.
/stretch <x|y|z> <factor=0.2..4> — Change the standard deviation of the cloud along one axis (before rotation).
/rotate <angle=0..180> — Rotates the cloud around the axis (1, 1, 0).
/shift <distance=0..3> — Moves the centre of the cloud away from the origin (upward).
/center — Turn data centering on or off before computing the covariance.
/noise <0..1> — Standard deviation of the isotropic noise added to each coordinate.
/seed <1..99> — Redraws the 200 points with another seed, same parameters.
/reset — Back to the initial cloud: 3 components, centered, no projection.

Glossary

Principal Component Analysis (PCA): Unsupervised method that finds the orthogonal directions along which data vary the most, then ranks them by decreasing variance. Used to reduce dimension, compress and visualise data while keeping as much information as possible.
Principal component: Direction (unit vector) in data space along which the variance is maximal, subject to being orthogonal to previous components. In the scene: the pink (PC1), yellow (PC2) and green (PC3) arrows, of length proportional to √λ.
Explained variance: Share of the total variance carried by a component: λi / (λ1 + λ2 + λ3). Cumulative shares tell you how much information you keep with k components — this is what drives the choice of k.
Covariance matrix: Square matrix (3×3 here) whose (i, j) entry measures how coordinates i and j vary together around the mean; its diagonal contains the variances. PCA diagonalises it: its eigenvectors are the principal components.
Eigenvalue and eigenvector: An eigenvector v of a matrix C satisfies C·v = λ·v: the matrix only stretches it by a factor λ. For the covariance, eigenvectors are the components and each eigenvalue is the variance of the data along that axis. We compute them here with Jacobi rotations; in practice, the singular value decomposition (SVD) of the centered matrix is preferred, as it is more numerically stable.
Projection: Replacing a point with the nearest point of a subspace (line, plane): you keep its coordinates along the retained components and discard the rest. In the scene, /project slides each point onto the indigo plane or line.
Dimensionality reduction: Going from d coordinates per point to k < d while preserving the structure of the data as well as possible. PCA is the reference linear reduction; t-SNE and UMAP are its non-linear cousins.
Data centering: Subtracting the mean from each point before computing the covariance. Without centering, we decompose XᵀX / n and the first "component" points toward the mean instead of describing the shape of the cloud: the classic mistake shown by /center.
Reconstruction error: Mean squared distance between each point and its reconstruction from k components. For centered PCA, it equals exactly the sum of the discarded eigenvalues (λ3 if k = 2). The grey segments of /reconstruct show it point by point.
Lossy compression: Reducing the size of a piece of data while accepting to lose some of it, measurably. Keeping k components out of d means storing k numbers per point instead of d: PCA is a lossy compression whose loss is exactly the reconstruction error.

Other channels in Unsupervised learning

#k-means — Grouping without labels: centroids that move, inertia that drops, the choice of k — and the shapes where k-means fails.
#pca — Principal component analysis: find the axes where data varies most, project, compress — and measure what is lost.
#hierarchical-clustering — Merge the points two at a time until only one is left: the dendrogram, the linkage criteria, and the cut height that decides the number of clusters.
#dbscan — Group by density: epsilon, MinPts, core, border and noise points — the algorithm that finds arbitrary shapes and ignores intruders.
#anomaly-detection — Spot what fits nothing: z-score / Mahalanobis, Isolation Forest, LOF — three ways to say 'this point is odd'.
#t-sne-umap — Map the high dimensions: t-SNE and UMAP unfold 10-dimensional data into a readable 2D map — perplexity, neighbors, and reading pitfalls.