Attribute : Neural Network Interpretability in Pytorch and TensorFlow Plaksha Tech Leaders Fellowship Capstone Project Praveen Sridhar Mentored by Nikhil Narayan, Lead Data Scientist, mfine.co

Introduction Predictions of Deep Learning models are difficult to explain and interpret Regulated industries like medical and finance industries often fall short from using Deep learning techniques due to this. Techniques for Interpretability help users to : ● Trust and understand why predictions are made in a certain way ● Provides accountability for usage in important decision making

Interpretability in the Medical Industry [Mukundhan et al. Google Research]

Objective 1. Research and implement algorithms for Neural Network Interpretability in the domain of Computer Vision

Literature Review

Pixel-space Attribution [Srinivas et al. NIPS 2019]

Saliency Heatmaps Grad CAM Full Grad Integrated Gradients Original Image [Srinivas et al. NIPS 2019]

Neuron Visualization Gradient Ascent Integrated Gradients SUMMIT [Hohman et al.]

Implementation

Attribute Python Library for Neural Network Interpretability in PyTorch and TensorFlow

Design ● Unified API for different types of techniques ● Benchmarking for comparisons

Details of Algorithms implemented

Gradient Attribution CEO In this, the gradient of output with respect to input image is taken as the attribution map

Smooth Gradients CEO Smoothed version of gradient attribution. Random noise is added to the input image and averaged to get the final attribution

Integrated Gradients CEO

Integrated Gradients (continued) CEO

GradCAM CEO Generalization of CAM (Class Activation Maps) to arbitrary architectures. CAM works only for networks with last layer as an Global Average Pooling layer Alpha = Weights[:, class_index] # (512,) FeatureMaps = getLastConvLayer() # (7,7,512) CAM = Alpha * FeatureMaps # (7,7) Upsample to original image size and overlay In Grad CAM, the equivalent to global average pooling is performed on the gradients of the output with respect to the feature maps Aij

FullGrad CEO FullGrad saliency uses a decomposition of the neural network into input sensitivity and neuron sensitivity components to give an exact representation of attribution

FullGrad CEO Official implementation requires modification of the network definition to get intermediate gradients and biases

FullGrad CEO In the attribute library, the implementation is flexible and automatically extracts the required intermediate gradients and bias values using internal PyTorch APIs

Sample Outputs

Sample Outputs GradCAM

Benchmarks

Conclusion The required Neural Network Library was implemented in both PyTorch and TensorFlow supporting the following algorithms ● Gradient Attribution ● Integrated Gradients ● Smoothed Gradients ● GradCAM ● FullGrad The library features a consistent API across different techniques as well as a benchmarking utility

Future Scope The project can be expanded to ● More techniques for interpretability, especially neuron visualisation ● Support other modalities like Text and Speech A possible line of research was found while testing FullGrad technique, where it was discovered that heat maps produced were not very class discriminative. A combination of ideas from GradCAM could potentially solve this.

Thank you