#linear-regression — Supervised learning

Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.

What you'll play with

Welcome to #linear-regression. On the left of the scene: 40 blue points (x, y) and a pink line y = a·x + b that misses them completely (a = −1, b = 1). On the right: the error bowl. At each (a, b), the height of the surface is the MSE, the mean squared error of the matching line; the yellow ball is our current line, perched high up the wall. Fitting a line means sliding this ball to the bottom of the bowl. A neural network does exactly the same thing… in a bowl with millions of dimensions.
Make the error visible: type /residuals. Each yellow segment ties a point to the prediction of the line right above or below: that is the residual y − ŷ. The MSE is the mean of the squares of these lengths.
Fix the slope by hand: /slope 1.5. The line pivots, the yellow segments shorten, and the ball drops sharply on the surface: MSE plunges. The bias b has not moved.
Tuning two numbers by hand is fine. A million, no. Let the algorithm work: /step 10. At each step, gradient descent computes the slope of MSE with respect to a and b, then moves in the opposite direction by a length lr × gradient (here lr = 0.1).
For a line, we do not even need to descend step by step: least squares has a closed form. a* = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)² and b* = ȳ − a*·x̄. Type /solution: the ball jumps straight to the bottom of the bowl.
The formula has an Achilles heel. Type /dataset outliers: three red points, far off, join the cloud and the bowl deforms. Then run /solution to see where the "best" line lands: those three points pull it toward them.
One last trap: /dataset curve, then /solution. The points follow a parabola, and the best possible line stays off to the side: the residuals are all the same sign at the extremes and the other in the middle. That is underfitting: the model is too simple for the data. You would need an x² term… or a neural network.
Your turn: /dataset linear then /slope -2, /lr 0.3 and /step 20 (right at the edge of instability: the ball zigzags from one wall to the other); again /slope -2, then /lr 0.5 and /step 5 (each step amplifies the error: it diverges); /noise 1.5 (the bottom of the bowl rises); /points 200 (the optimal line stabilizes); /seed 12 for another draw; /reset to start over. Next: #logistic-regression (the same line, but for classifying) and #gradient-descent (the same bowl, in far more twisted shapes).

Channel commands

/slope <a=-3..3> — Sets the slope a of the line (the bias does not move).
/bias <b=-3..3> — Sets the bias b (the y-intercept) of the line.
/residuals — Shows or hides the residuals (yellow segments from point to line).
/step <n=1..50> — Runs n gradient-descent steps on (a, b) with the rate lr.
/lr <0.001..1> — Sets the learning rate for descent.
/solution — Jumps to the exact least-squares solution (bottom of the bowl).
/dataset <linear|curve|outliers> — Changes the dataset (the current line is kept).
/noise <0..2> — Sets the standard deviation of the Gaussian noise added to y.
/points <n=10..200> — Sets the number of points in the cloud.
/seed <1..99> — Changes the random seed (different cloud, same settings).
/reset — Returns to the 40-point linear cloud and to the line a = −1, b = 1.

Glossary

Linear regression: A model that predicts a numeric value as a linear combination of its inputs: here ŷ = a·x + b. It is the simplest brick of supervised learning, and a neural network with no activation is nothing else.
Least squares: The method that picks the line minimizing the sum of squared residuals. For a line, the solution has a closed form: a* = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)², b* = ȳ − a*·x̄.
Residual: The gap y − ŷ between the observed value and the prediction, at one point. On the scene, it is the vertical yellow segment from a point to the line.
MSE (mean squared error): The mean of the squared residuals: MSE = (1/n) Σ (yᵢ − ŷᵢ)². This is the loss the regression minimizes; its square root (RMSE) is in the units of y.
R² (coefficient of determination): The share of y's variance explained by the model: R² = 1 − SSE / SST. Equals 1 for a perfect fit, 0 when the line does no better than the mean ȳ, and turns negative when it does worse.
Slope and bias (a, b): The two parameters of the line. The slope a (or coefficient) says by how much ŷ changes when x goes up by 1; the bias b (y-intercept) is the predicted value at x = 0. In a network, these are a weight and a bias.
Gradient descent: An optimization algorithm that repeats parameter ← parameter − lr × ∂loss/∂parameter. On the MSE bowl, each step follows the local slope to the bottom. Essential when there is no closed form.
Learning rate: The factor lr that sets the length of each descent step. Too small: slow convergence; too large: zigzags, then divergence. For MSE, descent is stable only if lr < 2 / λmax of the Hessian.
Outlier: A point far from the rest of the data (measurement error, exceptional case). Because MSE squares the residuals, a handful of outliers is enough to pull the least-squares line toward them.
Underfitting: A situation where the model is too simple for the structure of the data: a line on a parabola. The error stays high even at the optimum, and the residuals show a visible pattern (same sign across whole regions). The fix is a richer model, not a better optimizer.

Other channels in Supervised learning

#live-training — Six algorithms learning before your eyes, like a video: REC, timecode, subtitles, live metrics. Watching is free; touching the model is Premium.
#linear-regression — Fit a line: least squares, residuals, MSE, R² and gradient descent — the first brick of every supervised model.
#logistic-regression — Classify into two categories: sigmoid, decision boundary, threshold and log-loss — and why a line is not always enough.
#decision-trees — A tree that carves the plane into rectangles: Gini, entropy, depth, pruning — and the overfitting you can see with your own eyes.
#knn — k nearest neighbours: classify by resemblance, pick k, change the distance — and watch the boundary smooth out or shatter.
#svm-margins — Support vector machines: the widest possible margin, the C parameter, and the RBF kernel that curves the boundary.
#classification-metrics — Precision, recall, F1, confusion matrix, ROC and AUC: reading a classifier honestly, especially when classes are imbalanced.