What is Trustworthy AI and How It Contributes in Diagnosis? Masahiro Oda1, Zhou Zheng1, Jie Qiu1, Yuichiro Hayashi1, Kensaku Mori1,2, Hirotsugu Takabatake3, Masaki Mori4, Hiroshi Natori5 1Nagoya University, Nagoya, Japan 2National Institute of Informatics, Tokyo, Japan 3Sapporo-Minami-Sanjo Hospital, Sapporo, Japan 4Sapporo-Kosei General Hospital, Sapporo, Japan 5Keiwakai Nishioka Hospital, Sapporo, Japan

AI in Diagnosis Assistance • Deep learning is commonly used as diagnosis assistance AI ‒ For research ‒ For commercial system • Interpretation of decision process in AI ‒ Interpretable AIs: • Rule-based decision, decision tree ‒ Non-interpretable AI (black box AI): • Deep learning-based AI COVID-19 AI : Normal region : Infection region EndoBRAIN-EYE Ali-M3

Trustworthy AI • What is trustable AI for radiologists – Provide reason of its decision that can interpretable by radiologists and patients – Provide confidence rate of its decision • Radiologists will decide he/she follow or ignore the decision by AI based on confidence rate • Clarification including the reason of decision by AI and what AI doesn’t know is necessary • Important research fields in trustworthy AI – Explainable AI, Uncertainty in AI

Elements in Trustworthy AI • Explainable AI ‒ (Mainly) Try to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function GradCAM visualization of chest X-ray image classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020

Elements in Trustworthy AI GradCAM visualization of chest X-ray image classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 • Explainable AI ‒ (Mainly) Try to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function

Explainable AI • What is explainable AI? – Approaches to make decision process of non-interpretable AI (black box AI) interpretable for human • Approaches in explainable AI[2] – Outcome explanation – Model explanation – Model inspection – Transparent box design [2] Guidotti R., et al. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, 51(5), 1-42, 2018

Approaches in Explainable AI • Outcome explanation ‒ Provide explanations for AI outcomes ‒ Methods: LIME, CAM, Grad-CAM • Model explanation ‒ Provide interpretable model that approximates black box model ‒ Method: Born again trees[4] • Model inspection ‒ Understand model from its inputs and outcomes • Transparent box design ‒ Make parts of black box model interpretable [3] https://jp.mathworks.com/help/deeplearning/ug/understand-network-predictions-using-lime.html [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 [4] Leo B. Nong S. Born Again Trees. University of California, Berkeley, Berkeley, CA, Technical Report, 1996 GradCAM visualization of chest X-ray image classification model[1] Visualization by LIME[3]

Outcome Explanation Method: LIME[5] • LIME (local interpretable mode-agnostic explanations) – Visualizes regions on image that contributed to making decision of AI – Applicable to any image processing AI • How LIME works – Separate an image into small components – Make images for testing by randomly removing some of the small components – Feed the test images to AI and check its responses – If response from AI largely changed, removed small component has highly contributed to AI decision [5] Ribeiro M.T., et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD ‘16, 1135-1144, 2016 Images from https://theblue.ai/blog/lime-models-explanation/ Separation of image into small components Identification of small component that contributes to AI decision

Outcome Explanation Method: CAM[6] • CAM (class activation mapping) – Visualizes regions on image that contributed to making decision of AI – Applicable to CNN-based image processing AIs • How CAM works – Input an image to CNN-based AI – AI makes a decision by data propagation in its network – Get feature map and weight from network – Calculate heatmap (highlights regions on image that contributed to decision) from feature map and weight [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 [6] Zhou B., et al. Learning Deep Features for Discriminative Localization. Proceedings of CVPR, 2921-2929, 2016 Input image Conv Conv Pooling Feature extraction part Classification part Classification result G A P w1 w2 Class 1 Class 2 Class 3 w3 w4 * w1 + * w2 + * w3 + * w4 = CAM Heatmap Weighted sum of feature maps Contrib. High Low Feature map Weight Grad-CAM visualization (heatmap)[1]

Elements in Trustworthy AI GradCAM visualization of chest X-ray image classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 • Explainable AI ‒ (Mainly) Try to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function

Uncertainty in AI • What is uncertainty – Ambiguity and volatility of AI judgments resulting from various changes • Uncertainty can be utilized to improve accuracy of AI decision Input image 0 Benign 1 Malignant AI 0.53 Result: Malignant Decision AI decision without considering uncertainty Input images 0 Benign 1 Malignant Ave.: 0.45, S.D.: 0.2 Result: Benign but large ambiguity on decision Decision AI decision considering uncertainty AI Perturbation on image （Aleatoric uncertainty） Perturbation on AI model （Epistemic uncertainty）

Two Components of Uncertainty • Aleatoric uncertainty (data uncertainty) ‒ Uncertainty in AI decision caused by noise or variance in data • Epistemic uncertainty (model uncertainty) ‒ Uncertainty in AI decision caused by variance in model parameter determination (model training) Uncertainty maps in organ segmentation from CT image

Uncertainty Map • Uncertainty map visualizes ‒ Which area in image AI is high/low confident for its decision ‒ Which area in image AI knows or does not know (“does not know” means AI is not well trained with the pattern) ‒ Which area in image radiologists should check carefully High uncertainty means low confident Low uncertainty means high confident Uncertainty maps in organ segmentation from CT image

Uncertainty-based Multi Organ Segmentation[7] • Purpose 1: Segmentation with uncertainty estimation – Obtain uncertainty estimation for multi-organ (liver, spleen and stomach) segmentation • Purpose 2: Uncertainty-guided Interactive refinement – Make use of uncertainty estimation to guide interactive segmentation Segmentation result Input CT volume Uncertainty estimation Initial segmentation uncertainty estimation Interactive segmentation Refined segmentation [7] Zheng Z., et al., Taking full advantage of uncertainty estimation: an uncertainty-assisted two-stage pipeline for multi-organ segmentation, Proc. SPIE 12033, Medical Imaging 2022

Uncertainty-based Multi Organ Segmentation • Two-stage segmentation pipeline 1st: Segmentation with uncertainty estimation 2nd: Uncertainty-guided Interactive refinement

Segmentation with Uncertainty Estimation • Epistemic uncertainty with test-time dropout (TTD) Trained network with weight 𝑤 Monte Carlo simulation: Dropout sampling Probability map Sampled network with weight 𝑤1 Sampled network with weight 𝑤2 Sampled network with weight 𝑤3 Sampled network with weight 𝑤4 𝑦2 𝑦3 𝑦4 𝑦1 Average Posterior probability 𝑌 Input volume 𝑋 Argmax Entropy Prediction Epistemic Uncertainty Similar to model ensemble

Segmentation with Uncertainty Estimation • Aleatoric uncertainty with test-time augmentation (TTA) Monte Carlo simulation: Augmentation sampling (Gaussian noise, spatial flip and rotation) Probability map 𝑦2 𝑦3 𝑦4 𝑦1 Average Posterior probability 𝑌 Argmax Entropy Prediction Aleatoric uncertainty Trained network Trained network Trained network Trained network Input volume 𝑋 Augmented sample 𝑋1 Augmented sample 𝑋2 Augmented sample 𝑋3 Augmented sample 𝑋4

Uncertainty Guided Interactive Refinement • Proposed UI-DRLSE follows the framework of Distance Regularization Level Set Evolution (DRLSE)[8] Region information edge information Interaction constraint UI-DRLSE DRLSE [8] Li, C., et al. Distance regularized level set evolution and its application to image segmentation. IEEE Trans Image Processing 19(12), 3243–3254, 2010

Results of Uncertainty-based Segmentation 3D visual comparison of different methods. From (a) to (h) are respectively results of (a) ground truth, (b)V-Net (baseline), (c) MSPF, (d) UAMT, (e) EMMA, (f) V-Net + TTD, (g) V-Net + TTA, (h) V-Net + TTD + TTA. Moreover, Dice (%) and ASD (voxel) are indicated for each result in the top right corner.

Results of Uncertainty-based Segmentation Uncertainty maps in organ segmentation from CT volume Structure-level uncertainty estimation

Results of Uncertainty Guided Interactive Refinement • Uncertainty-guided interactive refinement Qualitative comparison of different interactive methods for refinement, where all methods are given the same interactions (note that our UI-DRLSE is only given the foreground interactions). Qualitative comparison of different interactive methods for refinement. We show the entire scribbles required for each method to get the final acceptable results.

Conclusions • Overview what is trustworthy AI – Clarify reason of decision by AI and what AI doesn’t know – Key research fields: Explainable AI, Uncertainty in AI • Explainable AI – Approaches: outcome explanation (LIME, CAM,…), model explanation, model inspection, transparent box design • Uncertainty in AI – Two components: aleatoric uncertainty, epistemic uncertainty – Detail of uncertainty-based multi-organ segmentation method was explained