Module 9 — Learning curves: reading overfitting

Plotting training loss and validation loss against epochs is the most profitable diagnostic gesture in deep learning. Two curves, and most of what is going wrong becomes readable. This module teaches how to read them.

The five situations to recognize

What the curves show	Diagnosis	What to do
Both losses fall and converge	healthy training	continue, then monitor
Both stay high	underfitting	enlarge the network, train longer, raise the rate
Training falls, validation rises	overfitting	regularize, early stopping, more data
The loss oscillates violently	rate too high or batch too small	lower the rate, enlarge the batch
The loss does not move at all	technical error	check data, labels, zero_grad()

The last two rows deserve careful distinction, as they are often confused. A loss that oscillates means optimization is progressing, but too abruptly. A loss that is flat from the start is almost never an optimization problem: it is a bug. Gradients not propagating, a zero learning rate, labels misaligned with inputs, or a forgotten optimizer.zero_grad().

Overfitting is not the enemy

Counter-intuitive but important: seeing validation rise is not a failure, it is information. It proves the network has the capacity to learn the data. The point where the two curves separate marks the optimal moment, which the early stopping of module 7 captures automatically.

The genuinely worrying situation is the opposite: two curves that stay high and glued together. There, the network learns nothing, and diagnosis is harder because the possible causes are many — insufficient capacity, a badly set rate, uninformative data, or a preparation defect.

The test to run before any long training

Here is the most useful habit in this course, and it takes two minutes: deliberately overfit a very small sample.

Take about ten observations and train the network on them, with no regularization, until the loss reaches practically zero.

small = torch.utils.data.Subset(train_data, range(10))
loader = torch.utils.data.DataLoader(small, batch_size=10)

for epoch in range(300):
    for X_batch, y_batch in loader:
        optimizer.zero_grad()
        loss = criterion(model(X_batch), y_batch)
        loss.backward()
        optimizer.step()
    if epoch % 50 == 0:
        print(f"epoch {epoch}: loss {loss.item():.5f}")

The interpretation is binary and conclusive. If the loss tends to zero, the whole chain works: data arrives correctly, gradients flow, the optimizer acts. You can launch full training with confidence. If it stalls, there is a bug, and it is pointless to hope the problem resolves itself on more data. Look at label alignment, tensor shapes, the learning rate, or a missing zero_grad().

This test regularly saves hours of training on a broken pipeline.

Two signals that should worry you

A validation loss below the training loss is surprising, but it usually has a benign explanation: dropout and batch normalization penalize training and are disabled at validation, and the training loss is averaged over the epoch while validation is measured at its end. If the gap is large and persistent, however, suspect a validation set that is too easy, or data leakage.

Excellent performance from the very first epoch should trigger exactly the reflex of the previous course: look for leakage. Module 8 of the feature engineering course describes the four families to review.

What to log

Beyond the two losses, three series prove valuable in diagnosis. The effective learning rate, to verify the schedule does what you think. The gradient norm, which reveals vanishing and explosion as in module 6. And the business metric — accuracy, area under the ROC curve — because a loss that falls while the metric stalls often signals a calibration or class-imbalance problem.

Summary

• Two curves suffice to tell apart healthy training, underfitting, overfitting, rate too high and technical bug.
• A loss flat from the start is a bug, not an optimization problem; a loss that oscillates is a step-size problem.
• Deliberately overfitting a small sample validates the whole chain in two minutes and should precede any long training.
• Validation better than training is often explained by dropout and measurement timing; an excellent score from the outset calls for a leakage hunt.

Next module: putting it into practice, with a first network trained end to end.