Masahiro Oda1, Zhou Zheng1, Jie Qiu1, Yuichiro Hayashi1, Kensaku Mori1,2, Hirotsugu Takabatake3, Masaki Mori4, Hiroshi Natori5 1Nagoya University, Nagoya, Japan 2National Institute of Informatics, Tokyo, Japan 3Sapporo-Minami-Sanjo Hospital, Sapporo, Japan 4Sapporo-Kosei General Hospital, Sapporo, Japan 5Keiwakai Nishioka Hospital, Sapporo, Japan
as diagnosis assistance AI ‒ For research ‒ For commercial system • Interpretation of decision process in AI ‒ Interpretable AIs: • Rule-based decision, decision tree ‒ Non-interpretable AI (black box AI): • Deep learning-based AI COVID-19 AI : Normal region : Infection region EndoBRAIN-EYE Ali-M3
Provide reason of its decision that can interpretable by radiologists and patients – Provide confidence rate of its decision • Radiologists will decide he/she follow or ignore the decision by AI based on confidence rate • Clarification including the reason of decision by AI and what AI doesn’t know is necessary • Important research fields in trustworthy AI – Explainable AI, Uncertainty in AI
to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function GradCAM visualization of chest X-ray image classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020
classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 • Explainable AI ‒ (Mainly) Try to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function
make decision process of non-interpretable AI (black box AI) interpretable for human • Approaches in explainable AI[2] – Outcome explanation – Model explanation – Model inspection – Transparent box design [2] Guidotti R., et al. A Survey of Methods for Explaining Black Box Models. ACM Computing Surveys, 51(5), 1-42, 2018
for AI outcomes ‒ Methods: LIME, CAM, Grad-CAM • Model explanation ‒ Provide interpretable model that approximates black box model ‒ Method: Born again trees[4] • Model inspection ‒ Understand model from its inputs and outcomes • Transparent box design ‒ Make parts of black box model interpretable [3] https://jp.mathworks.com/help/deeplearning/ug/understand-network-predictions-using-lime.html [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 [4] Leo B. Nong S. Born Again Trees. University of California, Berkeley, Berkeley, CA, Technical Report, 1996 GradCAM visualization of chest X-ray image classification model[1] Visualization by LIME[3]
– Visualizes regions on image that contributed to making decision of AI – Applicable to any image processing AI • How LIME works – Separate an image into small components – Make images for testing by randomly removing some of the small components – Feed the test images to AI and check its responses – If response from AI largely changed, removed small component has highly contributed to AI decision [5] Ribeiro M.T., et al. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. KDD ‘16, 1135-1144, 2016 Images from https://theblue.ai/blog/lime-models-explanation/ Separation of image into small components Identification of small component that contributes to AI decision
Visualizes regions on image that contributed to making decision of AI – Applicable to CNN-based image processing AIs • How CAM works – Input an image to CNN-based AI – AI makes a decision by data propagation in its network – Get feature map and weight from network – Calculate heatmap (highlights regions on image that contributed to decision) from feature map and weight [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 [6] Zhou B., et al. Learning Deep Features for Discriminative Localization. Proceedings of CVPR, 2921-2929, 2016 Input image Conv Conv Pooling Feature extraction part Classification part Classification result G A P w1 w2 Class 1 Class 2 Class 3 w3 w4 * w1 + * w2 + * w3 + * w4 = CAM Heatmap Weighted sum of feature maps Contrib. High Low Feature map Weight Grad-CAM visualization (heatmap)[1]
classification model[1] Uncertainty maps from abdominal organ segmentation model [1] Rajaraman S., et al. Iteratively Pruned Deep Learning Ensembles for COVID-19 Detection in Chest X-rays. IEEE Access, 8, 115041-115050, 2020 • Explainable AI ‒ (Mainly) Try to clarify reason of decision ‒ GradCAM and LIME are popular in image processing • Uncertainty in AI ‒ Clarify confidence of decision by AI ‒ Results can be utilized in: • Request radiologist for further consideration • Clarify what image patterns should be provided for training to improve AI model • Uncertainty-based loss function
volatility of AI judgments resulting from various changes • Uncertainty can be utilized to improve accuracy of AI decision Input image 0 Benign 1 Malignant AI 0.53 Result: Malignant Decision AI decision without considering uncertainty Input images 0 Benign 1 Malignant Ave.: 0.45, S.D.: 0.2 Result: Benign but large ambiguity on decision Decision AI decision considering uncertainty AI Perturbation on image (Aleatoric uncertainty) Perturbation on AI model (Epistemic uncertainty)
Uncertainty in AI decision caused by noise or variance in data • Epistemic uncertainty (model uncertainty) ‒ Uncertainty in AI decision caused by variance in model parameter determination (model training) Uncertainty maps in organ segmentation from CT image
image AI is high/low confident for its decision ‒ Which area in image AI knows or does not know (“does not know” means AI is not well trained with the pattern) ‒ Which area in image radiologists should check carefully High uncertainty means low confident Low uncertainty means high confident Uncertainty maps in organ segmentation from CT image
estimation – Obtain uncertainty estimation for multi-organ (liver, spleen and stomach) segmentation • Purpose 2: Uncertainty-guided Interactive refinement – Make use of uncertainty estimation to guide interactive segmentation Segmentation result Input CT volume Uncertainty estimation Initial segmentation uncertainty estimation Interactive segmentation Refined segmentation [7] Zheng Z., et al., Taking full advantage of uncertainty estimation: an uncertainty-assisted two-stage pipeline for multi-organ segmentation, Proc. SPIE 12033, Medical Imaging 2022
(TTD) Trained network with weight 𝑤 Monte Carlo simulation: Dropout sampling Probability map Sampled network with weight 𝑤1 Sampled network with weight 𝑤2 Sampled network with weight 𝑤3 Sampled network with weight 𝑤4 𝑦2 𝑦3 𝑦4 𝑦1 Average Posterior probability 𝑌 Input volume 𝑋 Argmax Entropy Prediction Epistemic Uncertainty Similar to model ensemble
of Distance Regularization Level Set Evolution (DRLSE)[8] Region information edge information Interaction constraint UI-DRLSE DRLSE [8] Li, C., et al. Distance regularized level set evolution and its application to image segmentation. IEEE Trans Image Processing 19(12), 3243–3254, 2010
From (a) to (h) are respectively results of (a) ground truth, (b)V-Net (baseline), (c) MSPF, (d) UAMT, (e) EMMA, (f) V-Net + TTD, (g) V-Net + TTA, (h) V-Net + TTD + TTA. Moreover, Dice (%) and ASD (voxel) are indicated for each result in the top right corner.
Qualitative comparison of different interactive methods for refinement, where all methods are given the same interactions (note that our UI-DRLSE is only given the foreground interactions). Qualitative comparison of different interactive methods for refinement. We show the entire scribbles required for each method to get the final acceptable results.
of decision by AI and what AI doesn’t know – Key research fields: Explainable AI, Uncertainty in AI • Explainable AI – Approaches: outcome explanation (LIME, CAM,…), model explanation, model inspection, transparent box design • Uncertainty in AI – Two components: aleatoric uncertainty, epistemic uncertainty – Detail of uncertainty-based multi-organ segmentation method was explained