Skip to main content

Module 10 — Interpretation: Grad-CAM and saliency maps

The fine-tuned ResNet50 from module 9 sits at 91 % validation accuracy on the waste dataset. Ship it? Not yet. A 91 % accuracy number tells you what the model does on average; it says nothing about why it decides. The same headline accuracy can be produced by a model that looks at the object and by one that has learned to recognise the studio background of the training photos. The two behave identically on validation and drastically differently in production. This module opens the model with Grad-CAM, shows the pattern, and closes with the honest limits of visual interpretation.

The Grad-CAM idea in one paragraph​

Take the last convolutional feature maps of the network — for ResNet50, those are the 7 by 7 by 2048 activations right before the global average pooling. Each of the 2048 channels is a spatial map of where a particular feature fired. If we could weight each channel by how much it contributes to the predicted class, we would obtain a spatial heatmap that says "the network looked here to decide this class". Grad-CAM does exactly that: the weight of a channel is the average of the gradient of the predicted class score with respect to that channel's activations.

Grad-CAM in code​

The recipe is short. Compute the gradient of the class score with respect to the last convolutional feature maps; average over the spatial dimensions to obtain a per-channel weight; combine the channels; apply a ReLU (only positive contributions matter for a heatmap); upsample to the input resolution.

import numpy as np
import tensorflow as tf

def grad_cam(model, image, class_index, last_conv_layer_name):
    """Return a Grad-CAM heatmap for one image and one predicted class."""
    grad_model = tf.keras.Model(
        inputs=model.inputs,
        outputs=[model.get_layer(last_conv_layer_name).output, model.output],
    )
    with tf.GradientTape() as tape:
        conv_out, predictions = grad_model(image[tf.newaxis, ...])
        class_score = predictions[:, class_index]

    # Gradient of the class score with respect to the feature maps.
    grads = tape.gradient(class_score, conv_out)                # (1, H, W, C)
    weights = tf.reduce_mean(grads, axis=(1, 2))                # (1, C)

    # Weighted combination of the feature maps, then ReLU.
    cam = tf.einsum("bc,bhwc->bhw", weights, conv_out)
    cam = tf.nn.relu(cam)[0].numpy()
    cam = (cam - cam.min()) / (cam.max() - cam.min() + 1e-8)    # normalise
    return cam

# On the waste-sorting model of module 9, the last conv layer is "conv5_block3_out".
heatmap = grad_cam(model, sample_image, class_index=3, last_conv_layer_name="conv5_block3_out")

To overlay the heatmap on the input image, resize it to the input size, apply a colormap (matplotlib.cm.jet is the classic choice) and blend at 40 % opacity:

import cv2
import matplotlib.pyplot as plt

heatmap_resized = cv2.resize(heatmap, (224, 224))
heatmap_colored = plt.cm.jet(heatmap_resized)[:, :, :3]
overlay = 0.6 * sample_image / 255.0 + 0.4 * heatmap_colored
plt.imshow(overlay); plt.axis("off"); plt.show()

Saliency maps: a gradient, without the trick​

An even simpler interpretation method is a saliency map: the absolute value of the gradient of the predicted score with respect to the input pixels. It answers a slightly different question — "which pixels, if slightly perturbed, would most change the score?" — and produces a per-pixel map at the full input resolution, which is finer than Grad-CAM's coarse 7 by 7 map.

def saliency(model, image, class_index):
    image_var = tf.Variable(image[tf.newaxis, ...], dtype=tf.float32)
    with tf.GradientTape() as tape:
        predictions = model(image_var)
        class_score = predictions[:, class_index]
    grads = tape.gradient(class_score, image_var)[0]
    return tf.reduce_max(tf.abs(grads), axis=-1).numpy()

Saliency maps are noisier than Grad-CAM because raw input gradients are noisy. Improvements like SmoothGrad average the map over slightly perturbed copies of the input to reduce the noise, and Integrated Gradients integrates the gradient along a straight path from a baseline (usually a black image) to the input — both are one-liners on top of the code above.

The Clever Hans effect on our waste dataset​

Take the model of module 9 and pick ten validation images the model got wrong. Compute Grad-CAM on each of them. If the model is looking at the object, the heatmap will be centred on the piece of waste. On our dataset, roughly one in three misclassifications shows a heatmap centred on the studio backdrop or on the shadow of the object. The model has learned a spurious correlation: photographs taken on a green backdrop tend to be one particular class in the training split, and the model shortcut its way to the label through the backdrop, not the object.

This is the Clever Hans effect: a network that appears to solve the task by exploiting a background artefact. In production, where photographs are taken on a conveyor belt with a metallic background, the model would fail dramatically. Grad-CAM makes it visible in a single figure.

The two responses are complementary. Short term: retrain with augmentation that includes background replacement (paste the object on random backgrounds). Long term: collect more diverse training photographs, and check the class distribution across background types.

A heatmap is not a proof

Grad-CAM says "the class score depends on these regions". It does not say "the model reasoned like a human about these regions". Two networks can produce nearly identical heatmaps and different failure modes. Use Grad-CAM as a hypothesis generator, not a verdict. A model that highlights the object is more trustworthy than one that highlights the background, but the correct conclusion is "the object heatmap is reassuring", not "the model has understood the object".

Faithfulness: does the heatmap match the model?​

Two follow-up questions turn a heatmap into evidence.

  • Deletion test. Blank out the pixels the heatmap highlights; the prediction score should drop sharply. If it does not, the heatmap does not describe the true reasoning of the model.
  • Insertion test. Start from a fully blurred image; progressively add back the pixels ordered by heatmap intensity. The score should rise fast if the heatmap is faithful.

Both tests are cheap and worth running on any explanation you publish. They catch the case where a visualization looks pretty but is actually decorrelated from the model.

The limits of visual interpretation​

Even a perfectly faithful heatmap has structural limits.

  • Class-agnostic backgrounds. If every class shares the same salient background, Grad-CAM will always highlight it, whether or not it drives the decision.
  • Fine-grained categories. Grad-CAM's 7 by 7 resolution on ResNet50 cannot distinguish a "bird's beak" from a "bird's eye". Higher-resolution variants (Grad-CAM++, HiResCAM) help.
  • Text and structured inputs. Grad-CAM was designed for images; on OCR, on documents, on non-image inputs, adaptations exist but the intuition weakens.
  • Multi-object images. Grad-CAM shows one heatmap per class. With two objects of two classes, the two heatmaps overlap and can look misleading.

PyTorch equivalent​

In PyTorch, the equivalent is torch.autograd.grad(class_score, feature_maps, retain_graph=True) followed by the same average, weighting and ReLU. Third-party libraries like pytorch-grad-cam package the whole pipeline; using them is fine for a quick check, but implementing it by hand once — as in the code above — is what fixes the concept.

Closing the red thread​

The course started with a Sobel filter slid over a synthetic image and ends with a heatmap that says "the model looked at the studio backdrop". The distance is 500 layers of CNN history and one common thread: every step of computer vision, from a hand-designed kernel to a self-supervised ViT, needs to be verified, not just measured. The 40-question exam of module 11 assesses whether you can perform that verification on your own.

In summary​

  • Grad-CAM weights the last convolutional feature maps by the gradient of the predicted class, producing a coarse heatmap that says where the decision looks.
  • Saliency maps are per-pixel gradients, higher resolution but noisier; SmoothGrad and Integrated Gradients denoise them.
  • Grad-CAM regularly exposes the Clever Hans effect: a model that appears accurate on validation but decides through a background artefact.
  • A heatmap is a hypothesis, not a proof; deletion and insertion tests measure whether an explanation is faithful, and even faithful heatmaps have limits on fine-grained, class-agnostic-background, and multi-object cases.

Next module: the recap and 40-question exam that closes the CNN course, with the running threads pulled explicitly across the ten modules.