Module 7 — TensorBoard: tracking losses, weights and histograms

The logs from fit give you two numbers per epoch. That is enough to know a run is going badly, and nowhere near enough to know why. TensorBoard answers the second question.

Wiring up monitoring

The callback joins the list from module 6, with a separate directory per run.

import datetime
from tensorflow import keras

stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
logdir = f"logs/{stamp}-adam-lr1e3"

monitor = keras.callbacks.TensorBoard(
    log_dir=logdir,
    histogram_freq=1,
    write_graph=True,
    update_freq="epoch",
)

model.fit(dataset, validation_data=val_dataset, epochs=50, callbacks=[monitor])

Then, from a terminal or a notebook cell:

tensorboard --logdir logs

The directory name is the only mechanism identifying a run. A name that describes the configuration — optimiser, rate, architecture — saves you from comparing twelve anonymous curves a week later. It is a ten-second investment that always pays off.

update_freq="batch" is expensive

Writing on every batch produces very detailed curves and enormous logs, and the writing itself becomes a bottleneck: on a small model it can double epoch duration. Reserve this setting for a targeted diagnosis over a few hundred steps, never for a full run.

Compare runs rather than viewing them one by one

This is where most of the value lies. With one directory per configuration under a shared parent, TensorBoard overlays the curves and makes comparison immediate.

logs/
  20260810-1420-adam-lr1e3/
  20260810-1455-adam-lr1e4/
  20260810-1530-sgd-momentum/

Three readings then appear at a glance. The gap between training and validation loss gives you overfitting, exactly as in module 9 of course 07. The initial slope tells you whether the learning rate is too low. And an abrupt validation collapse pinpoints the epoch where training diverged.

Weight histograms, the underused tool

histogram_freq=1 records the distribution of weights and activations at each epoch. It is the only way to watch the pathologies described in module 6 of course 07 while they happen.

What the histogram shows	Diagnosis
distribution flattening toward zero layer after layer	vanishing gradient
values spreading across several orders of magnitude	exploding gradient
activations concentrated at zero in a ReLU layer	dead neurons
distribution frozen from epoch to epoch	a layer that is not learning

The last case deserves particular attention because it is silent. A layer whose weights no longer move may indicate a layer frozen by mistake, a tf.Variable created without add_weight as in module 4, or a gradient that no longer reaches that depth. None of these three causes produces a message.

Logging your own quantities

Any scalar can join TensorBoard, which is useful for the business metrics Keras knows nothing about.

import tensorflow as tf

writer = tf.summary.create_file_writer(logdir + "/business")

with writer.as_default():
    tf.summary.scalar("false_positive_cost", cost, step=epoch)
    tf.summary.histogram("predicted_scores", scores, step=epoch)
    tf.summary.image("errors", misclassified_images, step=epoch, max_outputs=8)

The step argument is mandatory and serves as the horizontal axis. Image logging is especially instructive: displaying the eight worst-classified examples at each epoch often reveals a labelling problem rather than a modelling one.

The profiler answers "why is this slow"

The profiler measures how time splits between data preparation and computation, settling the question left open in module 5.

monitor = keras.callbacks.TensorBoard(
    log_dir=logdir,
    profile_batch="10,20",   # profiles batches 10 through 20
)

Profiling from the first batch mostly measures initialisation and tracing, not steady state — hence the offset interval. The profiler tab then reports device wait time. Above a few percent, the data pipeline is at fault and module 5 applies.

The diagnostic sequence

Faced with a run that will not converge, look in this order: the curves to locate when the problem appears, the histograms of the layer involved to identify its nature, then the graph to check that the architecture you built is the one you think you wrote. That last check surprises people more often than they admit, especially with the functional API where a branch can stay disconnected without raising an error.

Key takeaways

• One directory per run, named after its configuration, is what makes comparison usable; it is the only identification mechanism available.
• Overlaid curves reveal overfitting, learning rate quality and the epoch of a divergence at a glance.
• Histograms make vanishing gradients, exploding gradients, dead neurons and frozen layers visible — four pathologies that produce no error message.
• The profiler settles data bottleneck versus compute bottleneck, provided you profile in steady state rather than from the first batch.

Next module: transfer learning, which delivers strong results without training a network from scratch.