Module 5 — Overfitting and underfitting
The previous module showed the mechanics: minimizing a loss over examples. This module deals with the question that decides a project's fate: will the model be good on cases it has never seen? That ability is called generalization, and it fails in two opposite ways.
The real objective is not the one being optimized
Reread the training procedure with a critical eye. We minimize the error on the training data. But what we want is a model that is right on future data — tomorrow's customers, tomorrow's images. Training optimizes a stand-in for the real objective, not the objective itself.
The whole craft of machine learning lies in managing this gap. The two typical failures are its two sides.
Underfitting: the model missed the structure
A model underfits when it is too simple — or too badly tuned — to capture the real regularities in the data. The unambiguous symptom: it is already bad on the training data. A straight line trying to fit a bell-shaped relationship; a churn model reduced to two features when the phenomenon involves twenty.
The usual causes: a model too simple for the problem, poor input features that do not carry the necessary information, training stopped too early, or a learning rate so badly set that the descent never converged.
The remedies follow: a more expressive model, better features (often the most profitable lever — the feature engineering course is devoted to it), longer or better-tuned training.
Overfitting: the model memorized instead of learning
The opposite failure, and the more insidious one. A model overfits when it hugs the training examples so tightly that it captures their noise — the accidental particularities of that specific sample — instead of only the regularities that will repeat.
The characteristic symptom: excellent on training, mediocre on new data. A large gap between the two performances is the signature of overfitting.
The student analogy is exact: the one who understood the course solves new exercises; the one who memorized the answer keys only succeeds on questions already seen. Both get excellent grades on past exams. Only one is ready for the real one.
The risk grows when the model is very expressive (millions of parameters can memorize a lot), when data is scarce (fewer examples = easier to memorize them all), and when training runs long (the model ends up learning the noise after learning the structure).
The measuring device: three datasets
You can neither prevent nor even notice these failures without an honest measuring device. The field's universal practice splits the data into three watertight parts:
| Set | Typical share | Role |
|---|---|---|
| Training | 60–80% | Adjust the model's parameters |
| Validation | 10–20% | Compare variants, tune hyperparameters, decide when to stop |
| Test | 10–20% | Final measurement, used once, never to choose anything |
Why three and not two? Because the validation set, after being used to choose among dozens of variants, ends up being indirectly "learned" as well: you keep the variant it likes, which is a soft form of overfitting. The test set, untouched by any decision, gives the only credible estimate of future performance.
No information from the test set may influence development: not the model choice, not the features, not the thresholds, not the preprocessing (normalization statistics are computed on the training set only). Any breach — even unintentional — inflates the measured performance, and the gap is paid back in production. This family of mistakes has a name: data leakage, the leading cause of models that are "excellent in development, disappointing in production".
The standard remedies for overfitting
More data. The most reliable remedy: the more numerous and varied the examples, the harder memorizing becomes and the more profitable generalizing gets. When collection is possible, it often beats any algorithmic trick.
Regularization. A family of techniques that penalize model complexity during training — adding to the loss a cost on parameter magnitudes, or randomly switching off parts of the network (dropout). The effect: forcing the model to keep robust regularities rather than details.
Early stopping. Watch the validation error during training: as long as it goes down, continue; as soon as it rises while training error keeps falling, the model has started memorizing — stop and keep the best checkpoint.
Simplify. Reduce model size or feature count. Less spectacular than stacking complexity, often more robust.
Diagnosis in practice
The two-way check every practitioner runs mentally:
Key takeaways
- Training optimizes the error on known examples; the real objective is performance on future cases. Everything plays out in that gap.
- Underfitting: bad everywhere — model too simple or underfed. Overfitting: brilliant on training, disappointing elsewhere — the model memorized the noise.
- Three watertight sets (training, validation, test); the test set is used once. Any data leakage corrupts the measurement.
- Remedies for overfitting, by reliability: more data, regularization, early stopping, simplification.
Next module: the three learning regimes — supervised, unsupervised, reinforcement — and the criterion for recognizing which one your problem belongs to.