#anomaly-detection — Unsupervised learning

Spot what fits nothing: z-score / Mahalanobis, Isolation Forest, LOF — three ways to say 'this point is odd'.

What you'll play with

Welcome to #anomaly-detection. On screen, a cloud of 200 ordinary points (a Gaussian blob, in blue) into which we slipped 8 intruders. Nobody tells us which: no labels, this is unsupervised learning. The current detector (Mahalanobis, contamination 5%) gives each point an 'oddness' score and declares abnormal the top 5%: in red, with a halo. An intruder it missed would stay in yellow. Bank fraud, factory sensor, network intrusion: this is exactly the setup — spot what fits nothing.
Let's look at the score itself rather than the verdict: type /scores. Each point takes a color based on its score: blue = very ordinary, pink = borderline, red = far from center.
The Mahalanobis score has a geometric shape: type /envelope. The indigo ellipsoid is the frontier of the normal: the set of points whose Mahalanobis distance equals the threshold. Anything outside is called abnormal.
The threshold is the contamination: the share of anomalies expected. Be stricter: /threshold 0.02. Watch the ellipsoid grow and red halos go out.
Let's change population: /shape two-groups. The 'normal' now consists of two well-separated clusters, and half the intruders are planted exactly between the two.
Maybe the threshold is just too strict? There are 8 intruders out of 208 points, close to 4%: raise contamination with /threshold 0.05 and see what Mahalanobis gains.
We need a local detector: /method lof. The Local Outlier Factor compares the density around each point with that of its 10 nearest neighbors: a lone point in a void, surrounded by tightly packed neighbors, gets a high LOF.
A third way to say 'odd': /method isolation. An Isolation Forest cuts the space at random, again and again; a point isolated in two cuts is abnormal, one buried in the crowd needs seven or more.
Your turn: /shape elongated then /method zscore (the Mahalanobis cigar catches points outside the ellipse even if they are close to an axis), /inject 15 for more intruders (the 5% contamination is no longer enough), /threshold 0.1 to see precision drop, /point 204 to read a point's score (autocomplete suggests detected ones), /seed 12 for another draw, /reset to start over. Next steps: #dbscan (the 'noise' of density-based clustering, LOF's cousin) and #classification-metrics (precision, recall and their trade-off).

Channel commands

/method <zscore|isolation|lof> — Change the detector: Mahalanobis distance, Isolation Forest or LOF.
/threshold <contamination=0.01..0.3> — Contamination: expected share of anomalies; a point is detected if its score exceeds the 1 − threshold quantile.
/inject <n=1..20> — Number of intruders in the cloud (they drop from above).
/shape <blob|elongated|two-groups> — Shape of the normal population: blob, stretched and rotated Gaussian, or two separated groups.
/envelope — Show or hide the Mahalanobis ellipsoid at the current threshold (zscore method).
/scores — Color every point by score (blue → pink → red) instead of the binary verdict.
/point <id> — Highlight a point and show its score, rank and status.
/seed <n=1..99> — Redraw the cloud and intruders with another random seed.
/reset — Return to the blob, 8 intruders, Mahalanobis, contamination 5%.

Glossary

anomaly detection: Unsupervised task: without labels, learn what 'normal' looks like then flag points that deviate. Bank fraud, sensor failure, network intrusion: anomalies are rare, varied and often unknown in advance.
outlier: A point that does not resemble the rest of the data: too far from the center, too isolated, or placed where density is zero. In this channel, injected intruders are known so we can evaluate detectors, but the detectors never see the label.
z-score: Deviation from the mean measured in standard deviations: z = (x − μ) / σ. Beyond 3, a point is rare if the variable is Gaussian. Its generalization to several correlated dimensions is the Mahalanobis distance.
Mahalanobis distance: d = √((x − μ)ᵀ Σ⁻¹ (x − μ)): distance to the center of the cloud in units of its spread, direction by direction. A point 1 unit along the narrow axis of a stretched cloud is more abnormal than a point 3 units along the long axis. Iso-distance surfaces are ellipsoids.
covariance matrix: Table (3×3 here) of variances on the diagonal and covariances off the diagonal: it encodes the size, stretch and orientation of the cloud. Its inverse weights the Mahalanobis distance; its square root turns a sphere into the cloud's ellipsoid.
Isolation Forest: Ensemble of trees built with random cuts (one dimension, one value at random). An anomaly ends up alone in its leaf after few cuts; a point buried in the crowd needs many. No distance, no shape assumption, linear cost.
isolation depth: Number of cuts needed to isolate a point in a tree, averaged over the forest. The score 2^(−h(x) / c(n)) maps it to 0..1: near 1 = anomaly, around 0.5 = ordinary.
Local Outlier Factor: Ratio of the average local density of a point's k nearest neighbors to its own density (inverse of the average reachability distance). LOF ≈ 1: as dense as its surroundings; LOF ≫ 1: much more isolated than its neighbors. Being local, it spots a lone point between two clusters that a global envelope would call normal.
contamination: Proportion of anomalies expected (here 1% to 30%). The decision threshold is the matching quantile of scores: with 5%, the top 5% of scores are called abnormal. Too low, we miss intruders; too high, we accuse normal points.
false positive: Normal point called abnormal. Together with true positives (intruders correctly detected) and missed intruders (false negatives), it defines precision (correct detections / detections) and recall (found intruders / intruders). The threshold trades one against the other.

Other channels in Unsupervised learning

#k-means — Grouping without labels: centroids that move, inertia that drops, the choice of k — and the shapes where k-means fails.
#pca — Principal component analysis: find the axes where data varies most, project, compress — and measure what is lost.
#hierarchical-clustering — Merge the points two at a time until only one is left: the dendrogram, the linkage criteria, and the cut height that decides the number of clusters.
#dbscan — Group by density: epsilon, MinPts, core, border and noise points — the algorithm that finds arbitrary shapes and ignores intruders.
#anomaly-detection — Spot what fits nothing: z-score / Mahalanobis, Isolation Forest, LOF — three ways to say 'this point is odd'.
#t-sne-umap — Map the high dimensions: t-SNE and UMAP unfold 10-dimensional data into a readable 2D map — perplexity, neighbors, and reading pitfalls.