#logistic-regression — Supervised learning

Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.

What you'll play with

Welcome to #logistic-regression. On the scene: a horizontal (x, y) plane, two point clouds — blue (class 0) and pink (class 1) — and a yellow line, the model's decision boundary. Its starting weights are arbitrary: the boundary cuts the blue cloud in two, and the points ringed in red are misclassified. A logistic regression does only one thing: compute z = w1·x + w2·y + b, then turn that number into a probability p = σ(z) of belonging to the pink class. It is the simplest classifier there is — the one you always try first, from medical screening to spam filtering.
What the boundary does not show is the probability. Show it: type /surface. A translucent surface will rise above the plane: at each point its height is p(y=1 | x, y), from 0 (blue) to 1 (pink).
The weights set the orientation of the boundary and the steepness of the surface. Change them by hand: /weights 2 -1.
Setting weights by hand does not scale. Let gradient descent handle it: /train 50. At each epoch, the model measures its log-loss — it punishes a confident but wrong probability very hard — then moves w1, w2 and b in the direction that lowers it.
So far we classify "pink" as soon as p ≥ 0.5. This threshold is a choice, not a law. Demand more certainty: /threshold 0.8.
A new point arrives, whose class is unknown. Ask its verdict: /probability 0.5 0.5.
Now, the trap. Load a set where pinks occupy two opposite corners: /dataset xor.
Your turn: /dataset overlap then /train 200 (some errors are unavoidable: the log-loss stops falling toward zero), /noise 0.8 to blur the clouds, /lr 0.05 to see training crawl, /threshold 0.3 for the other trade-off, /seed 12 for another draw, /reset to start over. Next: #classification-metrics (precision, recall, ROC curve: what the threshold really moves) and #svm-margins (another way to pick the line).

Channel commands

/weights <w1=-5..5> <w2=-5..5> — Sets both weights: orientation of the boundary, steepness of the surface.
/bias <-5..5> — Sets the bias b: shifts the boundary without rotating it.
/threshold <0..1> — Sets the decision threshold: predict pink if p ≥ threshold.
/train <1..200> — Batch gradient descent on the log-loss for n epochs.
/lr <0.01..3> — Sets the learning rate used by /train.
/probability <x=-2..2> <y=-2..2> — Drops a query point and reads its predicted probability.
/surface — Shows or hides the probability surface p(y=1 | x, y).
/dataset <separable|overlap|xor> — Changes the point set (weights kept, epochs reset).
/noise <0..1> — Sets the spread of the clouds (Gaussian standard deviation, XOR clump spread).
/seed <1..99> — Changes the random seed of the point draw.
/reset — Returns to the separable set, starting weights, threshold 0.5, no surface.

Glossary

Logistic regression: A classification model that computes a weighted sum of the inputs z = w·x + b, then turns it into a probability p = σ(z). Despite its name, it predicts a class, not a quantity: it is the "regression" of a probability.
Binary classification: A task where each example belongs to one of two classes (0 or 1, blue or pink, spam or ham). The model learns a rule that picks one of the two for any new input.
Sigmoid: The function σ(z) = 1 / (1 + e^−z), S-shaped: it maps every real number into ]0, 1[, is 0.5 at z = 0, and saturates at both ends. It gives the surface its shape and lets the output be read as a probability.
Decision boundary: The set of points where the model changes its mind: here the line w1·x + w2·y + b = logit(threshold), drawn in yellow. For a logistic regression it is always a line (a hyperplane in higher dimension) — that is why it fails on XOR.
Decision threshold: The probability above which class 1 is predicted (0.5 by default). Changing it does not change the model but shifts the boundary: false positives are traded for false negatives, depending on the cost of each mistake.
Log-loss: The loss −[y·ln p + (1 − y)·ln(1 − p)], averaged over the examples: zero for a perfect probability, very large for a confident but wrong probability. This is what gradient descent minimizes.
Predicted probability: The model's output p(y=1 | x) for a given input: a confidence between 0 and 1, not a bare yes/no. The height of the surface (and of the yellow line under the query point) shows it.
False positive / false negative: False positive: a class-0 example predicted 1 (a blue classed pink). False negative: a class-1 example predicted 0. Raising the threshold reduces the former and increases the latter; classification metrics (precision, recall) count them separately.
Linear separability: A property of a dataset whose two classes can be separated by a line (a hyperplane). Two Gaussians can be; XOR cannot: no logistic regression will pass ~50% on it, a nonlinear model is needed.
Epoch: One full pass over the training data. Here each epoch is one batch gradient step: the log-loss gradient is computed on all 200 points, then w1, w2 and b move one step lr in the opposite direction.

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.