#knn — Supervised learning

k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.

What you'll play with

Welcome to #knn. On the ground, two already-labelled point clouds: blues and pinks. The yellow sphere is a new, unlabelled point sitting between the two. k nearest neighbours does not "train": to classify this point, it looks at its k nearest neighbours and lets them vote. Here k = 1: one yellow segment, the class of the nearest point, nothing else. A single stray pink is enough to tip the verdict… that is exactly the problem we are going to fix.
What would this classifier say at every point of the plane? Show the decision map: /boundary. Each square takes the colour of the class k-NN would predict at that spot.
Ask seven neighbours instead of one: /k 7. Watch the map, the segments, and the label above the yellow sphere.
Move the query into the zone where the two clouds mix: /query 0.2 -0.4. The seven segments reorient, and the yellow circle on the ground is the neighbourhood "ball": its radius is the distance to the seventh neighbour.
Change the very definition of "near": /distance manhattan. Manhattan distance adds the gaps in x and y instead of combining them under a square root, like a taxi that cannot cut through buildings.
Push k very high: /k 25. Twenty-five neighbours is a fifth of the dataset.
A trade-off: keep many neighbours, but give more weight to the closer ones. Type /weighted — each neighbour votes with weight 1/d.
Your turn. /dataset moons then /k 1 to watch k-NN wrap around a curved boundary without ever learning it (the map stays visible; /boundary hides it); /dataset overlap for a case where no k will work miracles (watch the leave-one-out); /classes 3 on /dataset blobs for a three-way vote; /distance chebyshev for the square ball; /noise 0.9 and /seed 42 for other draws; /reset to start over. Next: #svm-margins, where the boundary is learned rather than deduced from neighbours, and #classification-metrics to judge these predictions beyond accuracy.

Channel commands

/k <1..25> — Number of neighbours consulted for the vote.
/query <x=-2..2> <y=-2..2> — Moves the yellow sphere to be classified.
/distance <euclidean|manhattan|chebyshev> — Changes how "near" is measured.
/boundary — Shows or hides the decision map (40 × 40 grid).
/weighted — Toggles between majority vote and 1/d weighted vote.
/dataset <blobs|moons|overlap> — Changes the point set.
/classes <2|3> — Two or three Gaussian clouds (blobs dataset only).
/noise <0..1> — Spread of the clouds: 0 = crisp, 1 = fully mixed.
/seed <1..99> — Another random draw of the points, same settings.
/reset — Returns to blobs, k = 1, Euclidean distance, query (−0.4, 0).

Glossary

k nearest neighbours (k-NN): A classification method that assigns to a point the most frequent class among its k nearest neighbours in the training set. No model is fit: the data are the model. Works well in low dimension; in high dimension all distances look alike (curse of dimensionality) and the notion of neighbour loses meaning.
Euclidean distance: The straight-line distance: √(Δx² + Δy²). Points at distance r from a center form a circle. Sensitive to variable scale: normalize your data before using it.
Manhattan distance: Sum of absolute gaps: |Δx| + |Δy|, like a taxi following the street grid. Points at distance r form a diamond. Less sensitive to large deviations on a single variable than the Euclidean one.
Chebyshev distance: The largest of the absolute gaps: max(|Δx|, |Δy|), the number of moves of a king in chess. Points at distance r form a square. Only the farthest coordinate matters.
Majority vote: k-NN's decision rule: each neighbour casts a vote for its class, the class with the most votes wins. An odd k avoids ties in two-class problems; otherwise a tiebreak — for example the nearest neighbour — settles it.
Weighted vote: A variant where each neighbour votes with a weight decreasing with its distance, often 1/d. Very close neighbours weigh more than distant ones: you can pick a large k without the global majority class drowning the local structure.
Decision boundary: The line (or surface) in the plane where the predicted class changes. For k-NN it is never computed explicitly: it is revealed by classifying every point of a grid. Jagged for k = 1, smoother and smoother as k grows.
Hyperparameter k: The number of neighbours consulted, fixed before training and not learned. Small k: low bias, high variance (the map hugs the noise). Large k: smooth boundary, but high bias (underfitting). Choose it by validation, for example leave-one-out.
Leave-one-out: Extreme cross-validation: each point is classified by the model built on all the others, then the correct answers are counted. For k-NN it is free (just exclude the point from its own neighbours) and is used to choose k.
Lazy learning: A family of methods, including k-NN, that build no model at training time: they store the examples and defer all computation to prediction time. Instantaneous training, costly prediction (one distance per stored example).

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.