#overfitting — Neural networks

A big model on few data points: the decision boundary twists until it memorizes the noise.

What you'll play with

Welcome to #overfitting. On the stage: 30 noisy points — pink for class 0, blue for class 1 — and a 2-32-32-1 MLP with about 1,200 parameters, way too big for thirty examples. The decision boundary is the line where the model hesitates (p = 0.5): pink on one side, blue on the other. For now the background is uniform: nothing has been learned yet.
Type /train 30. The boundary appears, still smooth and reasonable: it roughly separates the two moons without lingering on the exceptions.
Keep going: /train 200. The model has enough capacity to chase every noisy point: the boundary twists, tentacles appear. Train accuracy climbs toward 100% while val stalls or drops.
Type /val to show the 300 validation points (small, translucent) that the model has never seen. Many fall on the wrong side of the tentacles: this is the cost of memorization.
Fix #1: more data. Type /points 200, then relaunch /train 200 and compare. With 200 examples, the noise averages out: the same model no longer has enough leverage to memorize every exception.
Fix #2: a smaller model. Go back to few points with /points 30, then type /size 2-4-1 and /train 200. Four neurons cannot memorize thirty points: the boundary stays simple.
Fixes #3 and #4: dropout (channel #dropout) and early stopping — stop as soon as val loss climbs again; the dashed line in the right panel marks that moment. Play freely: /noise 0.8, /dataset circles, /size 2-64-64-64-1, /reset. Next up: #cnn-filters, where the learned patterns become visible.

Channel commands

/train <epochs=1..300> — Continues training the current model (lr 0.05, ReLU, sigmoid output).
/size <2-…-1> — Changes the architecture (e.g. 2-4-1, 2-32-32-1, 2-64-64-64-1) and starts from scratch.
/points <10..300> — Changes the number of training points (validation stays at 300) and starts from scratch.
/noise <0..0.8> — Changes the data noise (train and val regenerated) and starts from scratch.
/dataset <moons|circles|xor> — Changes the 2D dataset and starts from scratch.
/val — Shows or hides the validation points in the scene.
/reset — Back to the initial state: moons, 30 points, 2-32-32-1, no training.

Glossary

Overfitting: The model memorizes the training examples (down to the noise) instead of learning the rule: great on train, poor on new data. Boundary twists around every point.
Underfitting: The model is too simple for the data's structure: bad everywhere, boundary too smooth. Fix: more capacity, more training, better features.
Generalization: A model's ability to predict well on data it has never seen. This is the only goal that matters; training loss is only a hint.
Bias-variance tradeoff: Bias = error from a too-simple model; variance = sensitivity to the training data (too flexible a model). More capacity lowers bias and raises variance: we look for the balance.
Validation set: Data set aside to measure generalization during training and pick hyperparameters. Distinct from the test set, reserved for the final verdict.
Early stopping: Stop training when the validation loss stops dropping, even if the training loss keeps falling: free regularization.
Capacity: The richness of functions a model can represent (number of parameters, depth). More capacity = risk of overfitting if data is scarce.
Noise: The part of the data that follows no rule (measurement errors, randomness). A model that learns it overfits by definition.

Other channels in Neural networks

#neuron — Let us break a neuron apart: inputs, weights, sum, activation.
#dropout — Dropout regularization: the best friend of deep networks.
#activation — Why an activation? ReLU, sigmoid, tanh, Leaky ReLU, GELU and the vanishing gradient.
#gradient-descent — Loss is a landscape. The gradient says which way is up, so we step the other way.
#backpropagation — The computation graph replayed backwards: each node receives ∂L/∂(itself) and the chain rule does the rest.
#overfitting — A big model on few data points: the decision boundary twists until it memorizes the noise.
#cnn-filters — An image is a grid of numbers. A 3×3 filter slides over it, multiplies, adds up: that is a convolution.
#is-it-einstein — Is it Einstein? Two faces go through the scanner of a network that learned only Einstein: Haythem → NO, Einstein → YES. Watching is free; touching is Premium.
#embeddings-3d — A word becomes a vector: close in space = close in meaning, and you can do math on them.