Class activation mapping methods are explainable AI (XAI) techniques used to visualize the regions of an input image that are the most relevant for a particular task, especially image classification, in convolutional neural networks (CNNs). These methods generate heatmaps by weighting the feature maps from a convolutional layer according to their relevance to the target class.
CNNs are designed to process spatially structured data, such as images, exploiting a series of convolution, non-linear activation, and pooling operations to extract relevant features contained in the so-called feature maps from input data. CNNs have proven to be highly effective in a variety of computer vision and image processing tasks. Despite their strengths, CNNs (and deep-learning models more broadly) are described as black boxes due to their complex and non-transparent internal layers of representation. The need to interpret decision-making processes gave birth to XAI techniques.
Class-activation mapping methods were originally developed for class-discriminative scenarios to visualize which parts of the input image influenced the classification decision, namely to visually highlight the regions of those feature maps that contribute most strongly to the prediction of a given class. More advanced versions of these methods are not limited to image classification tasks, but have been extended also to several vision-related tasks, such as object detection, image captioning, visual question answering, image segmentation, and medical image interpretation.
Background The following methods laid the groundwork for the class activation maps approaches, forming the conceptual basis of using gradients to highlight class-discriminative regions.
Class model visualization and saliency maps for convolutional neural networks The class model visualization and image-specific saliency maps approaches have been presented in the foundational work "Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps" by Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman; and it generalizes the DeconvNet method by Zeiler and Fergus.
Class model visualization synthesizes an artificial input image that strongly activates the output neurons associated with a target class. Given a trained, fixed model, this method starts with a zero-initialized image, backpropagates the gradients from the class score to the image pixels, updates the image pixels, increasing the specific class scores; and it repeats the pixel updating process, showing an encoded (idealized version) prototype of the class of interest. Image-specific class saliency visualization method provides a visual explanation by highlighting the most relevant pixels in an image for predicting a certain class C of interest. This is done by computing the gradient of the class score with respect to the input image, approximating the model locally (around ) as linear, using a first-order Taylor expansion: . The magnitude of , the gradient, indicates the importance of the pixels: larger gradients suggest greater influence on the prediction. Once the gradient is known, the saliency map is defined as the maximum absolute gradient across the color channels: resulting in a saliency map (i.e. heatmap).
Guided backpropagation The concept of guided backpropagation can be traced to the paper by Springenberg et al. "Striving For Simplicity: The All Convolutional Net", and this method builds upon the work by Zeiler and Fergus "Visualizing and Understanding Convolutional Networks".
Understanding guided backpropagation is to understand what a CNN is learning, by visualizing the patterns that more strongly activate individual neurons (or filters) in architectures that do not rely on a max-pooling layer.
When propagating gradients back through a rectified linear unit (ReLU), guided backpropagation passes the gradient if and only if the input to the ReLU was positive (forward pass) and the output gradient is positive (backward signal), tackling inactive neurons, negative gradients, and noise suppression. The result is sharper, high-resolution visualizations of what each neuron is responding to.
Guided backpropagation is a practical method for model interpretability to support understanding of how and where neural networks detect semantic concepts across layers. This method can be applied to any network architecture.
Base versions
Class-activation mapping and gradient-weighted class-activation mapping are the original, and most widely used, methods for visual explanations in convolutional neural networks. These methods serve as the foundation for many later developments in explainable AI.
Notation: In this article, the symbols i and j represent integer indices that disappear inside sums or averages, while x and y are the continuous (or up-sampled integer) coordinates of the final heatmap that is plotted.
Class activation mapping (CAM)
Class activation mapping (CAM) was the first, and the original, version of CAM methods; and it gave its name to the whole category. The approach was first introduced by Zhou et al. in their seminal work "Learning Deep Features for Discriminative Localization".
This approach achieves class-specific heatmaps by modifying image classification CNN architectures, replacing fully-connected layers with convolutional layers and a final global average pooling layer.
Its main scope is to localize and highlight discriminative regions of an input image that a CNN uses to identify a particular class, without needing explicit bounding-box annotations.
Global average pooling (GAP) Global average pooling (GAP) represents the key element in the original CAM approach.
It is a dimensionality reduction technique; and, similarly to other pooling layers, it allows the downsampling of the feature maps, calculating representative values for a specific region of the feature map. The distinction of GAP is that it calculates a single value for an entire feature map, significantly reducing the model dimensions.
Mathematical description The mathematical description considers as its key the combination of convolutional and GAP layers.
In CAM, it is mandatory to have the GAP layer after the last convolutional layer and before the final linear classifier layer. This last element of the architecture connects the output logits (the network predictions) , to the GAP values, with its respective fine-tuned weights, .
Considering as the last feature maps of the last convolutional layer, GAP produces one value for each feature map, by averaging all the matrix elements (i, j) of the feature map:
with
Namely, in the GAP layer, each feature map is reduced to a single scalar via GAP, producing k values, hence reducing the dimensionality of the network. A kmn tensor is reduced to k scalars, shrinking the parameter count for the linear classifier head.
The final output logits are calculated as the linear sum of the GAP values, weights, and bias:
The localization map is computed as follows:
namely, is the activation of node k in the target layer of the model, and is the class-specific weight, for the channel k, in the linear classifier layer.
Advantages and drawbacks The use of the GAP layer represents an example of an interpretability by design (IBD) approach. IBD refers to a technique which uses the model's own architecture to help explain its predictions.
The main drawback of CAM is that it is highly model-specific, being applicable to CNN architectures whose layer before the softmax one is a GAP.
Since the approach relies on the post-GAP weights for the overall evaluation, the method can't be applied to intermediate layers.
The choice of dealing with an IBD approach restricts the possibility to generalize the model architecture. Moreover, IBD methods often require re-training of the model.
Gradient-weighted class activation mapping (Grad-CAM) Gradient-weighted class activation mapping (Grad-CAM) is a generalized version of CAM and it overcomes the latter's architectural limitations. Grad-CAM computes the gradient of a target class score, the pre-softmax logit, with respect to the feature maps of a convolutional neural network. The gradients are global-average-pooled to obtain importance weights, which are used to compute a class-specific localization map by linearly weighting the feature maps. The result is a heatmap that highlights the regions in the input image that are the most influential for predicting the target class.
The main advantage of Grad-CAM, with respect to the standard CAM, is that it is model agnostic (provided that the network is still differentiable), meaning that it generates a visual explanation for any CNN-based network without architectural changes or re-training, making it broadly applicable to pre-trained models.
Mathematical description Considering: y the logits (i.e. the pre-softmax activated neurons responsible for a certain class prediction) of interest; A the feature-activated map for a specific convolutional layer; L ∈ the class-discriminative localization map, of width u and height v for any class c. Grad-CAM, employing backpropagation, computes the logit gradient with respect to the feature map A as
highlighting the importance of a certain class-discrimination decision-process of the logit. These gradients are global-average-pooled over each element of the feature map (hence, highlighting the "importance" of the elements of a feature map k for a target class C):
So, to account for the total number of feature maps, each of them is multiplied by its weight (via dot-product) and element-wise summation:
It can be observed that, due to the intrinsic nature of the gradient operation, some elements of the weighted feature map will have negative values; so, since only elements that have increased the logit of the predicted class are of interest, a ReLU activation function is applied:
Lastly, the output heatmap image dimensions are upsampled to the original image size to match the input dimensions.
Advantages and drawbacks Grad-CAM addresses the most important CAM limitations. It makes CAM free from the need for a GAP layer, generalizing its behavior and enabling visual explanation at intermediate layers.
However, Grad-CAM focuses on the most discriminative region when contributing to classification. If multiple similar objects are present, Grad-CAM often highlights only one of them, or part of one, while providing coarser maps and lower localization accuracy.
Moreover, Grad-CAM retrieves backwards information (the gradients), without taking into consideration how the activation flowed forward during prediction (unless combined with the guided backpropagation technique), resulting in a certain probability of missing patterns highlighted in the forward signal. On top of that, Grad-CAM heat maps are low-resolution when choosing a very deep layer.
Lastly, false emphasis in the heatmap may be present when large gradients are computed for low activation values. Grad-CAM assumes that gradient implies importance, ignoring the activation features value.
Grad-CAM and CAM comparison + CAM and Grad-CAM differences Feature CAM Grad-CAM Architecture Requires CNN whose last convolutional layer is followed by GAP Works with any CNN architecture that can be backpropagated Flexibility Only works with GAP-based networks Generalized, gradient-based version, it works with any pre-trained CNN at any layer Working principle Applies weights (class-specific linear classifier learned after GAP) from the output layer to feature maps Compute importance by calculating the gradients of the class score with respect to the feature maps Mathematical description of the working principle
Fine-tuned versions