Module 9 — Gaussian mixture models

Back to clustering, with a tool that repairs the two most inconvenient limits of k-means. First, a point no longer belongs to a cluster in clear-cut fashion: it belongs with a probability. Second, clusters are no longer constrained to be spherical. The Gaussian mixture is the natural generalization of module 2, and it retrospectively illuminates what that module assumed.

The model: data as a mixture of Gaussians

The assumption is explicit, which is a strength: the data come from several superimposed normal distributions (Gaussians), each corresponding to a cluster. Each component is described by three elements — its mean (its center), its covariance matrix (its shape and orientation), and its weight (its proportion in the mixture).

Clustering then becomes an estimation problem: recovering the parameters of the Gaussians that best explain the observed data. This is a generative model, and that has a welcome consequence: once fitted, it can generate plausible new data, and evaluate the likelihood of any point.

Probabilistic assignment, and what it changes

k-means assigns each point to a single cluster (hard assignment). A Gaussian mixture provides, for each point, its probability of belonging to each cluster (soft assignment): 70% cluster A, 25% cluster B, 5% cluster C.

The benefit is very concrete. A customer sitting on the border of two segments is common, and knowing it beats assigning them arbitrarily. These probabilities identify the ambiguous cases, those deserving special attention or a cautious decision — information that hard assignment destroys outright.

from sklearn.mixture import GaussianMixture
gmm = GaussianMixture(n_components=4, covariance_type="full", random_state=42).fit(X_s)
gmm.predict(X_s)         # most probable cluster
gmm.predict_proba(X_s)   # probabilities per cluster: the GMM's specific contribution

Free shapes thanks to covariance

This is the second contribution, and it follows from the covariance_type parameter. Since each Gaussian has its own covariance matrix, a cluster can be elongated, tilted, wider than another. Where k-means carved space into cells around centers, a Gaussian mixture hugs oriented ellipses.

covariance_type	Shapes allowed	Parameters
spherical	spheres	few
diag	axis-aligned ellipses	moderate
tied	same shape for all clusters	moderate
full	arbitrary ellipses, free orientation	many

full is the most expressive, but also the most data-hungry: estimating a full covariance per cluster demands many observations, especially in high dimension. The usual compromise is diag when data are scarce relative to the number of variables. This is exactly the bias-variance trade-off of the supervised course, here in the form of a parameter count.

This grid also illuminates an instructive equivalence: spherical components of equal weight, with hard assignment, essentially recover k-means. Module 2 was therefore a special case of this one.

The EM algorithm and choosing the number of components

Fitting is done by expectation-maximization (EM), whose structure will recall module 2: you alternate estimating the membership probabilities (E step) and updating the Gaussian parameters (M step), until the likelihood stabilizes. Like k-means, EM converges to a local optimum and depends on initialization — hence the use of several restarts (n_init).

For the number of components, the Gaussian mixture offers a clear advantage over module 3: being a genuine statistical model, it admits principled criteria. The BIC (Bayesian information criterion) adds up goodness of fit and a penalty on the number of parameters; you keep the $k$ that minimizes BIC. This is more objective than reading an elbow, the penalty automatically ruling out needlessly complex models.

bics = [GaussianMixture(n, random_state=42).fit(X_s).bic(X_s) for n in range(2, 11)]

When to prefer a Gaussian mixture

Three signals argue for it: clusters visibly elongated or of unequal sizes; an explicit need to identify border cases; or the usefulness of a generative model (simulation, likelihood computation, anomaly detection by low likelihood — a direct bridge to module 8). Conversely, on very large volumes in high dimension, k-means remains faster and more robust, and the Gaussian assumption must stay plausible: strongly skewed data, or data multimodal within a cluster, break it.

Summary

• A Gaussian mixture models the data as a superposition of Gaussians, each defined by mean, covariance and weight.
• Assignment is probabilistic: it reveals the border cases that the hard assignment of k-means erases.
• covariance_type frees the shape of clusters (oriented ellipses); full is the most expressive, diag the compromise when data are scarce.
• Fitting uses EM (local optimum, several restarts) and the number of components is chosen by minimizing BIC.

Next module: the customer segmentation project, where all the course's methods come together in a complete workflow.