1 Explainable AI (XAI) - Introduction & Dissection - 

2 2 Profile Slide skydome20 2011 – 2015 : NCKU CSIE 2015 – 2018 : NCKU IMIS 2018 – Now : CTL Data Science; R and Python Learner Author of R 系列筆記 SHAO-YEN HUNG (洪紹嚴) 

3 3 Agenda (XAI) Motivation Approaches Cases Key Takeaways Backup 1 2 3 4 2. 5 

4 4 Motivation What is Explainable AI (XAI) ? Why we need Explainable AI (XAI) ? 1 

5 5 What is Explainable AI (XAI) ? 探討如何解釋模型中的重要特徵？ 資料科學 • Interpretable machine learning • Feature importance interpretation 2016 -2018 探討如何建立人類對 AI 的信任？如何解釋黑盒子模型？ 資料科學 法律 企業 使用者 國家政府 • Explainable AI (XAI) 2019/02/18 (Gartner report) 

6 6 What is Explainable AI (XAI) ? AI models are increasingly deployed to augment and replace human decision making. However, in some scenarios, businesses must justify how these models arrive at their decisions. To build trust with users and stakeholders, application leaders must make these models more interpretable and explainable. Unfortunately, most of these advanced AI models are complex black boxes that are not able to explain why they reached a specific recommendation or a decision. Explainable AI in data science and ML platforms, for example, auto-generates an explanation of models in terms of accuracy, attributes, model statistics and features in natural language. Trend No. 4: Explainable AI Reference 2019/02/18 Gartner Top 10 Data and Analytics Technology Trends for 2019 

7 7 What is Explainable AI (XAI) ? Reference 2019/08/29 Gartner Hype Cycle highlights 

8 8 The black-box AI will create business risks for enterprises Bias & Discrimination Black-box AI Why we need Explainable AI (XAI) ? Safety & Privacy Medicine & Health Insurence …… 圖片來源 

9 9 AI risk to enterprises (Bias & Discrimination) https://www.technologyreview.com/s/613274/facebook-algorithm-discriminates-ai-bias/ https://www.insidehook.com/daily_brief/news-opinion/guilty-ai-found-perpetuate-biases-jailing http://news.mit.edu/2018/study-finds-gender-skin-type-bias-artificial-intelligence-systems-0212 https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting- tool-that-showed-bias-against-women-idUSKCN1MK08G 

10 10 AI risk to enterprises (Safety) https://phys.org/news/2018-04-uber-tesla-incidents-artificial-intelligence.html https://www.bbc.com/news/technology-49594260 

11 11 AI risk to enterprises (Medicine & Health insurance ) http://theconversation.com/artificial-intelligence-in-medicine-raises- legal-and-ethical-concerns-122504 https://www.forbes.com/sites/alexknapp/2019/09/02/this-startup- used-ai-to-design-a-drug-in-21-days/ https://www.facebook.com/TMAL119/photos/a.399888690142545/1673 108509487217/ https://health.ettoday.net/news/1523823 

12 12 Governments Laws & Regulations  歐盟：GDPR (資料保護規範) (2018/05) • 針對個人化自動決策，賦予用戶請求解釋、拒絕適用的權利  美國：Algorithmic Accountability Act of 2019 (演算法問責法案) • 賦權用戶更正或拒絕個人化自動決策；科技公司應主動消除演算法偏見  新加坡：人工智慧監管模式框架 (2019/01) • AI 的決策過程，必須盡可能的達到可被解釋、透明且公正的目的 因為這些法律的規範 讓「可解釋性」成為現在所有的AI產品應用都必須解決的課題之一 Reference GDPR TechOrange 專欄 新加坡台北代表處 

13 13 Three waves of AI • Rule-based algorithms. • No learning mechanism and poor handling in uncertainty 1. Symbolic AI 2. Statistic AI • Systems generate explanatory models. • Insights from learning data. • Make people to trust and be comfortable 3.Explainable AI • Statistical / Machine learning models for big data. • Minimal explainability. • No contextual induction capability. True & Strong AI ? Reference 2019 KDD Tutorial LinkedIn Explainable AI in Industry 

14 14 The proposed concept of XAI from LinkedIn 目前取得的 AI 成就 未來是「可靠、可信的 AI」趨勢 Reference 2019 KDD Tutorial LinkedIn Explainable AI in Industry 

15 15 Why we need Explainable AI (XAI) ? 圖片來源 For enterprises and engineers Compliance to legislation • Enterprises have responsibility for explaining how the applied AI influence the decision-making • Prevent black-box models from discrimination and bias Verify the AI system that works as expected Understand the weakness and improve the AI system Learning new insights from machine learning For human beings & society Make people comfortable & relieved 

16 16 Approaches 2 1. Basic interpretable models 2. Post-hoc explain methods XAI 

17 17 1. Basic interpretable models (Flow graph) Feature Importance Regression Based Tree Based Deep Learning Linear / Logistic regression Lasso / Ridge GLM / GAM Decision Tree RF / XGBoost / LightGBM RF DNN / CNN / RNN …… β (weights) p-values (build-in) Data Feature Engineering ML Models Prediction Result (build-in) X Basic interpretable models 

18 18 2. Post-hoc Explain Methods (Flow graph) Regression Based Tree Based Deep Learning Linear / Logistic regression Lasso / Ridge GLM / GAM Decision Tree RF / XGBoost / LightGBM RF DNN / CNN / RNN …… β (weights) p-values (build-in) Basic interpretable models Data Feature Engineering ML Models Prediction Result (build-in) X Post-hoc Methods 解釋模型如何 「理解資料」 Post-hoc explain (plug-in) Feature Importance 

19 19 XAI Approaches Basic interpretable models Post-hoc Explain Methods • Regression (GLMs) • Tree based (Decision Tree, Xgb, Lgb…) • Generalized Additive Models (GAMs) • Attribution Methods  Integrated Gradient (IG)  Shapley Value (SHAP) • LIME 

20 20 Introduction of paper and author Integrated Gradient (IG) Axiomatic Attribution for Deep Networks (2017-ICM) Google Brain Team LinkedIn profile GitHub(IG tutorial) Ankur Taly 

21 21 Integrated Gradient (IG) Integrate the gradients along a straight-line path from baseline to input Gradients • 中文上：「梯度」 • 數學上：「偏微分/一次導數」 • 經濟學：「Margin contribution (邊際效益/每變動一單位之貢獻)」 Integrate the gradients 將特徵們的邊際貢獻加總起來 = 

22 22 Integrated Gradient (IG) Baselines (or Norms) are essential to explanation [Kahneman & Miller, 1986] “A man suffers from indigestion” Doctor blames it to a stomach ulcer. Wife blames it on eating turnips. Both are correct relative to their baselines 消化不良 胃潰瘍 蕪菁 Integrate the gradients along a straight-line path from baseline to input Daniel Kahneman (2002 Nobel Prize in Economic Sciences) 

23 23 Integrated Gradient (IG) Baseline (informationless input) • Black color in image • Nan / 0 input • Empty text or zero embedding vector for text Integrate the gradients along a straight-line path from baseline to input 

24 24 Integrated Gradient (IG) Integrate the gradients along a straight-line path from baseline to input Model label: “fireboat” Original 解釋模型是「怎麼看」這張圖! Integrated Gradient(IG) 

25 25 Integrated Gradient (IG) Integrate the gradients along a straight-line path from baseline to input Clog (木靴) AI 解釋模型是「怎麼看」這張圖! 

26 26 Introduction of paper and framework SHapley Additive exPlanations (SHAP) A Unified Approach to Interpreting Model Predictions (2017-NIPS) GitHub 

27 27 Shapley Value SHapley Additive exPlanations (SHAP) Lloyd S. Shapley (2012 Nobel Prize in Economic Sciences) • A Value for n-person Games (1953) • In game theory, for cooperative game • Q: How to distribute gain for n-person in a group? • A: By marginal contribution (Shapley Value) 

28 28 SHapley Additive exPlanations (SHAP) Output = 0 1 2 3 4 5 6 7 8 9 Input 

29 29 SHapley Additive exPlanations (SHAP) 黑腹濱鷸 長嘴半蹼鷸 獴科 狐獴 解釋 DNN 模型 是「怎麼看」這張圖! 

30 30 SHapley Additive exPlanations (SHAP) Data Profiling If the LSTAT value is higher It’s SHAP value will be lower An negative impact on predicting output 

31 31 SHapley Additive exPlanations (SHAP) SHAP package GitHub Reference 

32 32 Attribution Methods Pros Cons • Good visualization • Humans are easy to interpret attributions and get insights • Clear connection with the ground truth / human intelligence • Computationally expensive • Difficult to explain feature interactions • Output-driven explain 

33 33 Attribution Methods (additional materials) • A Survey Of Methods For Explaining Black Box Models(Guidotti et al. [arxiv 2018]) • Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation (Ancona et al. [ICML 2019]) 

34 34 Introduction of paper and framework Local Interpretable Model-agnostic Explanations (LIME) “Why Should I Trust You?” Explaining the Predictions of Any Classifier (2016) 

35 35 Local Interpretable Model-agnostic Explanations (LIME) 將資料分割成小區域 抽樣不同的小區域 丟進模型預測結果 進行解釋 各次抽樣結果 再丟進一個迴歸 LIME: An Introduction Reference 

36 36 XAI Approaches(Flow graph) Regression Based Tree Based Deep Learning Linear / Logistic regression Lasso / Ridge GLM / GAM Decision Tree RF / XGBoost / LightGBM RF DNN / CNN / RNN …… β (weights) p-values (build-in) Basic interpretable models Data Feature Engineering ML Models Prediction Result (build-in) X Post-hoc Methods 解釋模型如何 「理解資料」 Post-hoc explain (plug-in) Feature Importance 

37 37 Cases • IBM • LinkedIn • Google • Fiddler (startup) • H2O.ai 3 

38 38 Introduction of Watson Openscale IBM (Watson Openscale) 亞太區技術長Dev Mookerjee 【XAI 挑戰主要發生在兩大 AI 模型】 1. 深度學習模型 2. 非結構化資料訓練的模型 【深度學習結合知識圖譜】 建構出資料之間的關聯性，一種特徵提取技 術(句子與句子之間的語意關係)，可用來優 化聊天機器人的模型。 要理解聊天機器人如何決策，需其他技術。 【IBM 使用 XAI 技術：Watson OpenScale】 1. LIME 2. CEM AI不可解釋難題真的無解？ 靠LIME框架能略窺關鍵特徵 Reference 

39 39 IBM (Watson Openscale) 【LIME】某客戶申請保險理賠，AI 模型判斷 90% 不適合，故拒絕 用 LIME 去檢測，這 90% 不合適的原因組成：  「未達年齡限制」45%  「索賠頻率過高」32%  「女性」8% <= 模型可能有性別歧視，需進行調整修正 【CEM】Contrastive Explanations Method：對比解釋方法 (IBM, 2018) 透過比較「缺少的特徵」，更完整地去詮釋模型  病人有症狀：「發燒、咳嗽、感冒」  為什麼 AI 模型判斷是流行感冒，而非肺炎？  答案是：病人「沒有」多痰及畏寒的特徵 Github: https://github.com/IBM/Contrastive-Explanation-Method 

40 40 IBM (Watson Openscale) Watson Openscale 

41 41 XAI application Cases LinkedIn • Recruiter Talent search Service • Interpretation for B2B sales predictive models Reference 2019 KDD Tutorial LinkedIn Explainable AI in Industry 

42 42 LinkedIn (Recruiter Talent search Service) Current case: inMail Accepts  inMail sent by recruiter, and positively responded by candidate  Mutual interest between the recruiter and the candidate 使用模型 運用 XAI 技巧 XGBoost Feature Importance Graph GLMix Inherently explainable DNN IG (Integrated Gradient) 

43 43 LinkedIn (Recruiter Talent search Service) 1. XGBoost ([XAI] Feature Importance Graph) Understanding feature digressions 1. Which a feature that was impactful no longer is? 2. Should we debug feature generation? Introducing new features in bulk and identifying effective ones 1. An activity feature for last 3 hours, 6 hours, 12 hours, 24 hours introduced (costly to compute) 2. Should we keep all such features? Separating the factors for that caused an improvement 1. Did an improvement come from a new feature, or a new labeling strategy, data source? 2. Did the ordering between features change? 查看在資料整備 上是否出現 Bug？ 觀察不同的資料整備區間 是否所有特徵都需納入建模？ 若改變策略(納入新特徵、 重新貼標)會造成什麼影響？ 

44 44 LinkedIn (Recruiter Talent search Service) 2. GLMix ([XAI] Inherently explainable) Contribution of a feature is “the weight of x feature” “IG works well for us”, LinkedIn says 3. DNN ([XAI] IG Integrated Gradient) Which features are critically influencing the model output? 解釋 「DNN 是如何理解資料？注重在哪些重要特徵上？」 找出特徵的影響力(根據迴歸模型的權重) 

45 45 LinkedIn (Recruiter Talent search Service) Local Views Global Views • Better for making modeling decisions • Gives an overview of data • Global explanations is the next step for Deep Models • Better for debugging DNN models • Be more useful for the non-technical user • “IG works well for us”, LinkedIn says XGBoost, GLMix DNN(IG) 

46 46 LinkedIn (Interpretation for B2B sales predictive models) Current situation & Problem Definition • RF, GBM, DNN • High accuracy, low interpretation • Feature contributor : what are the top features for a certain company to have high/low probability to upsell? • Feature influencer : Which top features can be perturbed if we want to increase/decrease the upsell probability for a certain company? 影響 upsell 成功率的重要特徵有哪些？ (Feature contributor) 如果想增加 upsell 的成功率，該調整哪些特徵？ (Feature influencer) 

47 47 LinkedIn (Interpretation for B2B sales predictive models) They want to found • Feature contributor : what are the top features for a certain company to have high/low probability to upsell? • Feature influencer : Which top features can be perturbed if we want to increase/decrease the upsell probability for a certain company? 

48 48 LinkedIn (Interpretation for B2B sales predictive models) Models & XAI approach RF, GBM, DNN 黑盒子 xLIME (improved LIME) 加入兩個技巧，改善LIME  Piecewise linear regression  Localized stratified sampling 參考投影片中有簡單介紹此二方法，不過目 前無更進一步的 Paper 跟 Source Code 釋出 They want to found • Feature contributor : what are the top features for a certain company to have high/low probability to upsell? • Feature influencer : Which top features can be perturbed if we want to increase/decrease the upsell probability for a certain company? 

49 49 LinkedIn (Interpretation for B2B sales predictive models) LinkedIn Career Page Upsell Data • Instances(row): 19k • Features(column): 117 

50 50 LinkedIn (Interpretation for B2B sales predictive models) How to compare xLIME with LIME? • Pick top-n features for modeling => AUC • [前40個重要特徵]：xLIME 所找到的， 比LIME所找到的，重要性更強一點 xLIME LIME 117 

51 51 LinkedIn (Interpretation for B2B sales predictive models) Summary & insights • Feature contributor vs influencer • Contributor: Which features end-up in the current outcome case-by-case • Influencer: What needs to be done to improve likelihood, case-by-case • Better captures the important features from models (DNNs / GBMs / RF) • xLIME aims to improve LIME via: • Piecewise linear regression: more accurately describes local points • Localized stratified sampling: more realistic set of local points 

52 52 IG for Question-Answer model Google Google Brain Team LinkedIn profile GitHub(IG tutorial) Ankur Taly Did the Model Understand the Question ? (Google Brain Team, 2018) 論文中文講解：https://kknews.cc/zh-tw/tech/vk34k4l.html 

53 53 Google(IG for Question-Answer model) Q: How many medals did India win? A: 197 論文連結 33.5% (accuracy) Tabular QA Three types of Question-Answer model Q: How symmetrical are the white bricks on either side of the building? A: Very 61.1% (accuracy) Visual QA 論文連結 Q: Name of the quarterback who was 38 in Super Bowl XXXIII? A: John Elway 84.6 (F1 scores) Reading Comprehension 論文連結 

54 54 Google(IG for Question-Answer model) • Baseline: Empty question, but full context (image, text, paragraph) • Attribute the answer (or answer selection logic) to question words Integrated Gradient(IG) • red: high attribution • blue: negative attribution • gray: near-zero attribution [symmetrical] [white] [bricks] are not important？ 模型並未理解這些重要字詞！ 這些模型其實忽略了問題中重要的詞，導致錯誤歸因（把無關詞 當成決定答案的重要線索），這也是Q-A 系統不穩定的原因之一 IG for explaining Q-A model 

55 55 Google(IG for Question-Answer model) Q: How many medals did India win? A: 197 33.5% (accuracy) Tabular QA Adversarial attack for Q-A models Q: How symmetrical are the white bricks on either side of the building? A: Very 61.1% (accuracy) Visual QA Q: Name of the quarterback who was 38 in Super Bowl XXXIII? A: John Elway 84.6 (F1 scores) Reading Comprehension 11.4% (accuracy) 19% (accuracy) X 對抗樣本攻擊 對抗樣本攻擊 

56 56 Introduction of this startup Fiddler Google Brain Team LinkedIn profile GitHub(IG tutorial) Ankur Taly • A startup provides explainable AI engine for the enterprise • 2019-05: Hiring Ankur Taly as the head of DS Fiddler welcome letter 

57 57 Fiddler Fiddler welcome letter 

58 58 The proposed XAI engine Fiddler XAI approaches they used • Integrated Gradient (IG) • SHAP • LIME • GAM (GA2M) • Logistic Regression • Randomized controlled trial (A/B test) • …… Fiddler.ai Fiddler blogs Reference 

59 59 Fiddler Fiddler.ai Fiddler blogs Reference 

60 60 Explainable AI tools H2O.ai Collect basic interpretable models • Regression (GLMs) • Tree based (Decision Tree, XGBoost…) • Generalized Additive Models (GAMs) • SHAP 

61 61 Key Takeaways 4 Summary & Reference 

62 62 Key Takeaways Summary Why we need explainable AI? • For people: comfortable & relieved • For enterprises: avoid risks & compliance to legislation • Benefits for different groups XAI approaches • Basic interpretable models (regression, XGB, LGB, trees, GAMs) • Attribution methods (IG, SHAP…) • LIME • …… Industrial practices/cases • IBM, LinkedIn, Google, Fiddler, H2O.ai 

63 63 Key Takeaways Reference & Materials Book：Interpretable machine learning Video(李宏毅)： Explainable ML Slides • Explainable AI in Industry (2019-KDD) • Tutorial on Interpretable Machine Learning (2018-MICCAI) • Explaining Deep Neural Networks with a Polynomial Time Algorithm for Shapley Values Approximation All Papers mentioned in this slides 

64 END