Skip to main content

Module 4 — Eigenvalues and dimensionality reduction

Real data often has hundreds of features, many of them redundant. Dimensionality reduction compresses that information into a few well-chosen axes. The mathematical tool behind it — eigenvalues and eigenvectors — has an intimidating reputation; this module gives the intuition and the concrete use, without proofs.

Eigenvectors: the directions a transformation preserves​

Recall module 2: a matrix transforms vectors, generally changing their direction. But for certain matrices, a few special directions are only stretched or shrunk, not deflected. These are the eigenvectors, and the associated stretch factor is the eigenvalue:

Av=λvA v = \lambda v

The eigenvector vv keeps its direction; the eigenvalue λ\lambda says by how much it is stretched. A large eigenvalue flags an "important" direction of the transformation — one where the action is strong.

From covariance to principal axes​

Apply this to data. The covariance matrix of a dataset summarizes how features vary together: high covariance = related features. Its eigenvectors point toward the directions where the data spreads out most, and the eigenvalues measure that spread (the variance) along each direction.

The idea of principal component analysis (PCA) fits in one sentence: keep the few directions of largest variance, discard the rest. You replace 200 correlated features with, say, 10 axes that capture 95% of the variance.

PCA in practice​

from sklearn.decomposition import PCA
import numpy as np

# X: (n_observations, 200 features)
pca = PCA(n_components=10)
X_reduced = pca.fit_transform(X)     # (n_observations, 10)

pca.explained_variance_ratio_.sum()  # share of variance kept, e.g. 0.95

Explained variance is PCA's dashboard: it tells how much information you keep. Going from 200 to 10 dimensions while retaining 95% of the variance is a common and highly profitable trade-off.

What dimensionality reduction is actually for​

  • Visualization: project high-dimensional data into 2D or 3D to look at it.
  • Speed: fewer features = faster training and lighter models.
  • Denoising: low-variance directions often contain mostly noise; removing them can improve generalization.
  • Fighting the curse of dimensionality: in very high dimension, distances lose meaning (module 3); reduction helps.
What PCA does not do

PCA maximizes variance, not class separation: it ignores labels. A low-variance direction may still be decisive for telling two classes apart; PCA then risks discarding it. Also, the principal axes are combinations of the original features, often hard to interpret. Finally, PCA requires pre-scaled data, otherwise large-amplitude features monopolize the first axes — the same pitfall as module 3.

Summary​

  • An eigenvector is a direction a transformation merely stretches; its eigenvalue measures the stretch.
  • The eigenvectors of the covariance matrix point toward the directions of largest data variance.
  • PCA keeps the k axes of largest variance and discards the rest, compressing hundreds of features into a few dimensions.
  • Explained variance quantifies retained information; PCA serves to visualize, speed up and denoise — but ignores labels and requires pre-scaling.

Next module: derivatives and gradient — moving from linear algebra to calculus, the language of learning itself.