Skip to main content

Module 10 — The bias-variance trade-off, shown on an example

Here is the module where the whole course converges. Overfitting and underfitting, described intuitively in the introductory course, find here their exact mathematical explanation: a model's error decomposes into two antagonistic terms, bias and variance, and knowing how to learn is knowing how to balance them.

Decomposing the error​

The error a model makes on new data splits into three parts:

Error=Bias2+Variance+irreducible noise\text{Error} = \text{Bias}^2 + \text{Variance} + \text{irreducible noise}
  • Bias measures the error from overly simple assumptions: the model is too rigid to capture the true structure of the data.
  • Variance measures the model's sensitivity to the specific training data: change the sample slightly, and the model changes a lot.
  • Irreducible noise is the randomness inherent in the data, which no model can remove.

We can act only on the first two — and, this is the whole point, they pull in opposite directions.

The target example: four situations​

The classic image is a dartboard. The center is the truth; each dart is the model trained on a different sample.

  • Low bias, low variance: darts clustered at the center. The ideal.
  • High bias, low variance: darts clustered, but far from center. The model is consistent… and consistently wrong: underfitting.
  • Low bias, high variance: darts centered on average, but scattered. The model aims right overall but depends too much on the sample: overfitting.
  • High bias, high variance: the worst, scattered and off-center.

The U curve: the heart of the trade-off​

The more complex a model (more parameters, more flexibility), the lower its bias — it fits the data better. But its variance rises — it clings to the details and noise of the sample. Total error follows a U curve:

The optimal point is neither the simplest model nor the most complex, but the one that minimizes the sum. This is exactly what the gap between training error and validation error reveals, as described in the introductory course: a large gap betrays excess variance (overfitting); two high, close errors betray excess bias (underfitting).

The levers to balance​

This trade-off is not a fate you suffer; you steer it:

  • Reduce variance: more training data, regularization (L1L_1/L2L_2, which penalizes overly large weights — a direct use of the module 3 norms), ensemble models such as random forests.
  • Reduce bias: a more expressive model, better features (the feature-engineering course), longer training.

Regularization deserves a special mention: it adds a penalty on weight size to the cost, forcing the model to stay simple. It is the most direct control of variance, and one of the most-used tools in practice.

The thread of the whole course

This trade-off is the reason these mathematics matter. Linear algebra builds the model (modules 1-4); calculus trains it (modules 5-6); probability and statistics evaluate its uncertainty and spread (modules 7-9). The bias-variance trade-off is the synthesis: it says why a more powerful model isn't always better, and why judging a model always requires data it has never seen.

Summary​

  • Error decomposes into bias² (too-simple assumptions), variance (sensitivity to the sample) and irreducible noise.
  • Bias and variance oppose each other: complexity lowers bias but raises variance, hence a U curve of total error.
  • High bias = underfitting; high variance = overfitting; the optimum balances the two, revealed by the training/validation gap.
  • You steer the trade-off: more data and regularization against variance; a more expressive model and better features against bias.

Next module: the general recap and the exam that certifies your mastery of the mathematics of AI.