Module 6 — Principal component analysis

A change of family: we leave clustering for dimensionality reduction. The question is no longer "which groups?" but "can I describe these data with fewer variables without losing the essentials?". PCA is the reference answer, and one of the most widely used tools in all of data science.

The problem: too many variables, and redundant ones

Fifty variables describing a customer raise three concrete difficulties: you cannot visualize anything beyond three dimensions; distances lose their discriminating power (the curse of dimensionality, which undermines k-means and DBSCAN); and many of these variables say the same thing — height and weight, income and spending, floor area and number of rooms.

That redundancy is precisely the opening. If two variables are strongly correlated, a single well-chosen direction almost suffices to summarize both.

What PCA does: finding the dominant directions of variation

PCA looks for new axes, called principal components, such that:

1. the first component is the direction in which the data vary the most;
2. the second is the direction of greatest remaining variation, perpendicular to the first;
3. and so on.

Each component is a linear combination of the original variables — the matrix product of the mathematics course applied to a precise goal. The perpendicularity constraint guarantees that each new axis brings information not already captured: the components are decorrelated from one another, and redundancy has vanished.

By keeping only the first components, you describe the data along far fewer axes while retaining most of their variability. Technically, PCA diagonalizes the covariance matrix: the eigenvalues from the mathematics course measure the variance carried by each axis.

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X_s = StandardScaler().fit_transform(X)     # indispensable: see the warning
acp = PCA().fit(X_s)
acp.explained_variance_ratio_               # share of variance per component
acp.explained_variance_ratio_.cumsum()      # cumulative variance: the curve to read

Standardize first, no exceptions

PCA maximizes variance, and variance depends on units. A variable in euros (enormous variance) would monopolize the first component against a variable in years, not through importance but through choice of unit. Without prior standardization, a PCA is uninterpretable — and the error is silent, since the computation completes regardless.

Choosing the number of components

Explained variance is the central criterion: each component carries a share of it, and the cumulative total indicates what you retain. Three ways to decide, chosen according to the objective:

Criterion	Rule	Use
variance threshold	keep enough components for 90 or 95%	compression, preprocessing
scree plot elbow	curve of variance explained per component, cut at the drop-off	exploratory analysis
visualization	2 or 3 components, whatever the variance says	graphical display

The scree plot reads exactly like the elbow of module 3: beyond the drop-off, components capture only noise. And scikit-learn lets you state the objective directly: PCA(n_components=0.95) automatically keeps the number of components needed for 95% of the variance.

Interpreting the axes

A component has no name: you must give it one, by examining the weights of each original variable in its definition (acp.components_). If the first component loads heavily on income, home floor area and spending, you will name it "standard of living". This interpretive work turns a mathematical result into business knowledge; it is what gives PCA its explanatory value, beyond compression.

Limits to know

PCA is linear: it captures only straight directions of variation, and a coiled structure escapes it (hence t-SNE and UMAP in the next module). The components are combinations of all the variables, hence less directly readable than the original variables. Finally, PCA privileges variance, which is not always the useful information: a low-variance variable can be decisive for a given task.

Summary

• PCA exploits redundancy between variables to describe the data along fewer axes.
• Principal components are linear combinations of successively maximal variance and perpendicular to one another, hence decorrelated.
• You choose their number by cumulative variance threshold, scree plot elbow, or a visualization constraint.
• Standardization is mandatory; interpretation goes through the variable weights, and the method remains linear.

Next module: t-SNE and UMAP, which visualize non-linear structures — with strict reading precautions.