𝐶!-LRP : Visual Explanation Generation based on Layer-Wise Relevance Propagation for ResNet Felix Doublet, Seitaro Otsuki, Tsumugi Iida, Komei Sugiura Keio University

- 2 - Summary : We adapted LRP to handle ResNet models • A new Layer-wise Relevance Propagation (LRP) rule to handle residual connections of ResNet • 𝑪𝟑-LRP that selects the most noteworthy area based on the generated relevance regions. We introduce : LRP [Bach+, PLoS15] GradCAM [Selvaraju+, ICCV17] Ours

- 3 - Background : The need for an eXplainable AI Deep Neural Networks (DNNs) = black box nature : • Lack of trust by end-users (medical analysis, autonomous vehicles) • Legal need (RGPD, AI Act) to explain decisions Anastasia Grivia : https://impact.universityofgalway.ie/articles/to- black-box-or-not-to-black-box/ ⇒ Explainable AI (XAI) Layer-Wise Relevance Propagation [Bach+, PLoS15] : • Strong theoretical background • Transparency in Decision-Making • Interpretable Relevance scores ⇒ White Box Analysis

- 4 - Background : Generating visual explanations for ResNet with LRP Input Image n Layer-Wise Relevance Propagation (LRP) fails to generate insightful explanations for ResNet models LRP Output LRP FAIL !

- 5 - n Layer-Wise Relevance Propagation (LRP) fails to generate insightful explanations for ResNet models LRP Output Residual connections ResNet[He+, CVPR16] No designed rule ✕ Background : Generating visual explanations for ResNet with LRP ???

- 6 - n Layer-Wise Relevance Propagation (LRP) fails to generate insightful explanations for ResNet models LRP Output Residual connections ResNet[He+, CVPR16] Tailored rule ✓ Our method Background : Generating visual explanations for ResNet with LRP

- 7 - GradCAM [Selvaraju+, ICCV17] Related work : Back-Propagation methods GradCAM [Selvaraju+, ICCV17] Gradients flowing to produce a coarse localization map. Input x Gradient [Shrikumar+, ICLR18] Multiplying the input image by the gradient of the output. LRP [Bach+, PLoS15] Propagating the prediction backward through the network layers.

- 8 - Related work : Layer-Wise Relevance Propagation - an established theoretical framework LRP [Bach+, PloS15] n Layer-wise Relevance Propagation [Bach+, PLoS15] n Established theoretical framework n Transparent computational processes

- 9 - : the relevance score of neuron 𝑘 at layer 𝑙 + 1 : activation of neuron 𝑘 : weight of the connection between neuron 𝑖 and neuron 𝑘 Related work : Layer-Wise Relevance Propagation - an established theoretical framework LRP [Bach+, PloS15]

- 10 - : the relevance score of neuron 𝑘 at layer 𝑙 + 1 : activation of neuron 𝑘 : weight of the connection between neuron 𝑖 and neuron 𝑘 Related work : Layer-Wise Relevance Propagation - an established theoretical framework LRP [Bach+, PloS15] A rule must be defined for each type of layer

- 11 - Proposed Method : The need for a new rule for the ResNet models ResNet50 architecture :

- 12 - Proposed Method : The need for a new rule for the ResNet models ResNet50 architecture :

- 13 - One bottleneck block: ResNet50 architecture : Proposed Method : The need for a new rule for the ResNet models Need for a tailored backpropagation rule

- 14 - IDEA : Considering each bottleneck block as a single dense layer Proposed Method : Relevance backpropagation for Bottleneck layers

- 15 - : Relevance of the layer L IDEA : Considering each bottleneck block as a single dense layer Proposed Method : Relevance backpropagation for Bottleneck layers

- 16 - : Input of the Bottleneck block : Output of the Bottleneck block 𝐶 : Output channels 𝑈 : Height 𝑉 : Width : Avoiding a zero division IDEA : Considering each bottleneck block as a single dense layer : Relevance of the layer L Proposed Method : Relevance backpropagation for Bottleneck layers

- 17 - Choice Contour Component (𝐶") : Removing unnecessary information Relevance map (LRP output) Proposed Method : Choice Contour Component : 𝑪𝟑

- 18 - Choice Contour Component (𝐶") : Removing unnecessary information Relevance map (LRP output) Proposed Method : Choice Contour Component : 𝑪𝟑 1st Contour (findContours) 1st Component (connectedComponents) C1C [Iida+, SIG-AM23] TRUE Two binary masks are created

- 19 - Choice Contour Component (𝐶") : Removing unnecessary information Relevance map (LRP output) Proposed Method : Choice Contour Component : 𝑪𝟑 1st Contour (findContours) 1st Component (connectedComponents) C1C [Iida+, SIG-AM23] AND TRUE

- 20 - Choice Contour Component (𝐶") : Removing unnecessary information Relevance map (LRP output) Proposed Method : Choice Contour Component : 𝑪𝟑 1st Contour (findContours) 1st Component (connectedComponents) C1C [Iida+, SIG-AM23] AND Final Relevance map Only the pixels from BOTH the 1st contour and the 1st component are kept TRUE

- 21 - Experimental setup : The CUB-200-2011 dataset n The CUB-200-2011 dataset [Wah+, 11] : n 11,788 images of birds n 200 classes n Split : 5,794:200:5,794 (train:val:test) Laysan Albatros Least Auklet

Method Acc. ↑ Insertion↑ Deletion↓ ID score↑ RISE [Petsuik+, RMVC18] 0.815±0.001 0.371±0.015 0.043±0.004 0.328±0.004 GradCAM [Selvaraju+, ICCV19] 0.815±0.001 0.466±0.019 0.156±0.008 0.310±0.020 LRP [Bach+, PLoS15] 0.815±0.001 0.063±0.007 0.051±0.006 0.011±0.001 ABN [Fukui+, CVPR19] 0.642±0.009 0.282±0.052 0.075±0.011 0.207±0.054 Ours 0.815±0.001 0.685±0.015 0.017±0.001 0.668±0.015 - 22 - Quantitative results : Our method outperformed the baseline methods

- 23 - +𝟎. 𝟐𝟏𝟗 Quantitative results : Our method outperformed the baseline methods

- 24 - −𝟎. 𝟎𝟐𝟔 Quantitative results : Our method outperformed the baseline methods

- 25 - +𝟎. 𝟑𝟑𝟎 Quantitative results : Our method outperformed the baseline methods

- 26 - Qualitative results : Our method provided the most insightful explanations Original Ours ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 27 - Ours Qualitative results : Our method provided the most insightful explanations § Only one wing is attended § The attended area is vague Original ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 28 - Ours Qualitative results : Our method provided the most insightful explanations § The bird is correctly attended § The attended area is vague Original ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 29 - Ours Qualitative results : Our method provided the most insightful explanations § Almost no pixel is attended § Not an insightful explanation Original ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 30 - Ours Qualitative results : Our method provided the most insightful explanations § The bird is correctly attended § The background is also attended Original ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 31 - Ours Qualitative results : Our method provided the most insightful explanations § The bird is correctly attended § The shape of the bird’s body is precisely attended Original ABN [Fukui+, CVPR19] LRP [Bach+, PloS15] GradCAM [Selvaraju+, ICCV17] RISE [Petsuik+, BMVC18]

- 32 - Ablation study : 𝑪𝟑 effectively generated high quality explanations +𝟎. 𝟒𝟑𝟔

- 33 - Ablation study : 𝑪𝟑 effectively generated high quality explanations +𝟎. 𝟎𝟔𝟖

- 34 - Conclusion : We successfully generated high quality explanations Contributions: • A method for calculating LRP in models with residual connections • 𝐶!-LRP, which improves the quality of explanations 𝑪𝟑-LRP LRP [Bach+, PloS15] output 𝐶!-LRP output

- 35 - Appendix : Failure case example – All the different methods failed to generate an insightful explanation Original Ours ABN [Fukui 19] LRP [Bach 15] GradCAM [Selvaraju 18] RISE [Petsuik 18] ⇒ None of the different methods correctly attend the bird

- 36 - Appendix : Error analysis - The method focusing on an insufficient part of the image Error Type IA OA WA #Error 63 21 16 Insufficiently Attended (IA) Over-Attended (OA) Wrongly-Attended (WA) • IA : The area of attention is too small. • OA : The area of relevance is excessively large • WA : The relevance is given to pixels that do not directly contribute to the classification.