Skip to main content

Module 8 — Anomaly detection: isolation forest and statistical methods

The course's third and final family. We no longer seek to describe the data as a whole, but to isolate the minority that departs from it: bank fraud, sensor failure, manufacturing defects, network intrusion. The reversal is total — here it is the exceptions that carry the value.

What an anomaly is, and why this is hard​

An anomaly is an observation that departs markedly from the normal behavior of the data. This simple definition hides three difficulties that structure the whole field.

First, anomalies are rare by nature: a few tenths of a percent, often. Second, they are diverse: normal cases resemble each other, anomalies rarely resemble one another — a new fraud does not look like previous ones. Third, they are little or not at all labeled, which rules out treating the problem as plain classification. These three traits explain why we generally model the normal in order to flag what does not fit, rather than learning the anomalies themselves.

Two useful distinctions: an anomaly can be point-wise (a single aberrant value) or contextual (an ordinary value at the wrong time — 25 °C is normal in summer, not in January).

Statistical approaches: simple and often sufficient​

On a single variable, two classic tools frequently suffice. The z-score measures the deviation from the mean in standard deviations, and you flag beyond 3. Its weakness is well known: mean and standard deviation are themselves contaminated by the anomalies present. The interquartile range (IQR) escapes this by relying on quartiles, robust by construction: you flag whatever falls outside [Q1−1.5 IQR,  Q3+1.5 IQR][Q_1 - 1.5\,\text{IQR},\; Q_3 + 1.5\,\text{IQR}] — the whisker rule of the box plot.

Their common limit is being univariate: they examine each variable separately and miss combination anomalies. A height of 1.50 m is normal, a weight of 100 kg is normal, but the pairing of the two is atypical. No per-variable rule will see it, which makes multivariate methods necessary.

The isolation forest: an anomaly is easy to isolate​

The isolation forest starts from a remarkably simple idea, running counter to density-based approaches. You cut the space at random: pick a variable at random, a threshold at random, and repeat, building a tree of random cuts. How many cuts does it take to fully isolate a given point?

  • a normal point, buried among its peers, requires many cuts;
  • an abnormal point, off to the side, ends up isolated in very few cuts.

The average depth needed for isolation, measured over a forest of random trees, directly gives the anomaly score. This is the inverse of density logic: instead of modeling the normal, it exploits the fact that the abnormal is structurally easier to separate.

from sklearn.ensemble import IsolationForest
iso = IsolationForest(contamination=0.01, random_state=42).fit(X_s)
iso.predict(X_s)          # -1 anomaly, 1 normal
iso.score_samples(X_s)    # continuous score: preferable for prioritizing

Its strengths explain its popularity: fast, effective in high dimension, no shape assumption about the data. The contamination parameter is the expected proportion of anomalies — a business decision (what alert volume can we handle?) more than a statistical setting. And it is almost always preferable to use the continuous score rather than the binary decision: it lets you rank cases and handle the most suspicious first.

Two approaches usefully round out the toolkit. LOF (local outlier factor) compares a point's local density to that of its neighbors, detecting anomalies relative to their region — valuable when densities vary. The one-class SVM learns a boundary enveloping the normal data and flags whatever falls outside.

Note too that module 5 already provides a detector: points labeled noise (-1) by DBSCAN are, by construction, natural candidates for anomaly status. And PCA reconstruction (module 6) works on the same principle — a point poorly reconstructed by the principal components is a point that does not fit the dominant structure.

Evaluating without labels, then with the few you have

Without labels, you validate by expert inspection of the flagged cases, the only way to establish whether the alerts are relevant. As soon as you have a few confirmed anomalies, the framework of module 9 of the supervised course applies — with its decisive caveat: on classes this imbalanced, accuracy is misleading and you reason in precision and recall, or even PR AUC. In production, complement this by monitoring alert volume: a sudden drift usually signals a change in the data rather than a wave of anomalies.

Summary​

  • Anomalies are rare, diverse and barely labeled: we model the normal to flag what departs from it.
  • z-score and IQR (more robust) handle one variable at a time, and miss combination anomalies.
  • The isolation forest exploits the fact that an atypical point isolates in few random cuts: fast and effective in high dimension.
  • contamination is a business decision; prefer the continuous score for prioritizing, and evaluate by expertise then precision/recall.

Next module: Gaussian mixture models, which generalize k-means into probabilistic assignments.