Module 3 — Choosing the number of clusters: elbow and silhouette score

k-means demands a $k$ that nobody knows. This module gives the two tools that let you choose it defensibly — and above all, how to read them without telling yourself stories. It is the most frequently asked question in clustering, and the one where people go wrong most readily.

Why inertia alone cannot answer

Recall from module 2: inertia always decreases as $k$ grows. With as many clusters as points, it equals zero. Minimizing inertia would therefore mechanically lead to one cluster per observation — a perfect and perfectly useless result.

Inertia is not to be discarded, though: it is its rate of decrease that carries the information. That is the idea behind the elbow method.

The elbow method: reading a slowdown

You plot inertia as a function of $k$ and look for the point where the curve stops falling and flattens out — the "elbow". The interpretation is intuitive: up to that point, each additional cluster reveals real structure and drives inertia down; beyond it, you are merely subdividing already-homogeneous clusters, for marginal gain.

inerties = []
for k in range(2, 11):
    inerties.append(KMeans(n_clusters=k, n_init=10, random_state=42).fit(X_s).inertia_)

The method's flaw is its honesty: the elbow is often ambiguous. On real data the curve bends gently and two people will see 3 and 5 clusters in it. The elbow method delimits a plausible range; it does not settle the matter. Hence the next tool, more demanding.

The silhouette score: compactness and separation

The silhouette score evaluates, for each point, whether it is in the right cluster, by comparing two average distances: the one to its own cluster ($a$) and the one to the nearest neighboring cluster ($b$).

$$s = \frac{b - a}{\max(a, b)} \in [-1, 1]$$

The reading is direct, and that is what makes the indicator valuable:

Value	Interpretation
close to 1	point well placed, clusters well separated
close to 0	point on the border between two clusters
negative	point probably misassigned

Averaging over all points gives a global score comparable across several $k$ — and this time, you can keep the maximum.

from sklearn.metrics import silhouette_score, silhouette_samples
silhouette_score(X_s, km.labels_)     # global score, comparable across k
silhouette_samples(X_s, km.labels_)   # per-point score: far more informative

The second call is worth the detour. A global score of 0.55 can hide three very clean clusters and a fourth incoherent one whose points sit near zero or below. Examining the distribution per cluster reveals exactly where the partition holds and where it cracks — something an average conceals.

Arbitrating: the criteria do not decide alone

Elbow and silhouette delimit a space of reasonable solutions; the final choice incorporates two considerations that geometry ignores:

• stability (module 1): rerun with other random seeds, or on 90% of the data drawn at random. If the clusters recompose deeply, that $k$ is not reliable, whatever its score;
• business usefulness: 4 interpretable and actionable segments beat 7 top-scoring segments that nobody knows what to do with. A clustering you cannot name will not be used.

When no k works

A silhouette score that plateaus low (say under 0.25) for every $k$ is not a tuning failure: it is a result. It says there probably are no clean spherical clusters in these data. Three possible next steps, in this order: revisit the variables and the scaling (the signal may be drowned), try a method that does not assume spheres (DBSCAN in module 5, Gaussian mixtures in module 9), or accept that the structure is continuous rather than clustered — in which case the dimensionality reduction of module 6 is the right tool.

Summary

• Inertia always decreases with $k$: what you read is its slowdown, via the elbow method, which delimits a range without settling it.
• The silhouette score compares compactness and separation in $[-1, 1]$ and is comparable across several $k$; the maximum is a serious candidate.
• The per-point score reveals fragile clusters that a global average masks.
• The final choice incorporates stability and business usefulness; a low score for every $k$ is information, not failure.

Next module: hierarchical clustering, which builds a tree of clusters without fixing $k$ in advance.