'Why Should I Trust You?' Explaining the Predictions of Any Classifier

Transcript

1. "Why Should I Trust You?" Explaining the Predictions of Any Classifier FATML 2016 November 18th 2016 Sameer Singh Marco Tulio Ribeiro Carlos Guestrin
2. Machine Learning is everywhere 2
3. Only 1 mistake! Predict wolf vs husky 3
4. Visual Question Answering Is there a moustache in the picture? > Yes What is the moustache made of? > Banana 4
5. Is it doing what we want? Or a neural network with more than a thousand layers? 5
6. They are essentially black-boxes How can we trust the predictions are correct? Trust How can we understand and predict the behavior? Predict How do we improve it to prevent potential mistakes? Improve 6
7. Fairness in ML I would like to apply for a loan.. Your loan has been denied Machine Learning What? Why? Cannot explain.. [0.25,-4.5,3.5,-10.4,…] Currently 7
8. Only 1 mistake! Predict wolf vs husky We’ve built a snow detector… 8
10. Visual Question Answering What is the moustache made of? > Banana What are the eyes made of? > Bananas What? > Banana What is? > Banana 10
11. What an explanation looks like Why did this happen? From: Keith Richards Subject: Christianity is the answer NTTP-Posting-Host: x.x.com I think Christianity is the one true religion. If you’d like to know more, send me a note 11
12. Three must-haves for a good explanation Definitely not interpretable Potentially interpretable Interpretability Humans can easily interpret reasoning 12
13. Three must-haves for a good explanation x y Learned model Not faithful to model Interpretability Humans can easily interpret reasoning Faithful Describe how the model actually works 13
14. Three must-haves for a good explanation Interpretability Humans can easily interpret reasoning Faithful Describe how the model actually works Model Agnostic Can explain any classifier 14
15. Local, Interpretable, Model-Agnostic Explanations (LIME) 1. Pick a model class interpretable by humans - May not be globally faithful… 2. Locally approximate global (blackbox) model - Simple model is globally bad, but locally good Line, shallow decision tree, sparse features, … Locally-faithful simple decision boundary ➔ Good explanation for prediction 15
16. LIME: General framework Instance Explanation Family Universe of possible explanations to search over Faithfulness of Explanation Is the explanation faithful to the model in context of x Interpretability Is the explanation simple enough to read? 16
17. 1. Sample points around xi 2. Use complex model to predict labels for each sample 3. Weigh samples according to distance to xi 4. Learn new simple model on weighted samples 5. Use simple model to explain Sparse Linear Explanations 17
18. Perturbed Instances P(Labrador ) Sampling example - images Original Image 0.92 0.001 0.34 P(labrador) = 0.21 Locally weighted regression Explanation 18
19. Explaining Google’s Inception NN P( ) = 0.21 P( ) = 0.24 P( ) = 0.32 19
20. Only 1 mistake! Predict wolf vs husky 20
21. Explanations for neural network prediction We’ve built a great snow detector… ☹ 21
22. Did ML experts notice it? 0. 25. 50. 75. 100. Didn't trust the model "Snow insight" % of subjects (out of 27) Before explanations After explanations 22
23. Explanations should match our intuitions Original Image “Bad” Classifier “Good” Classifier 23
24. Explanation for a bad classifier From: Keith Richards Subject: Christianity is the answer NTTP-Posting-Host: x.x.com I think Christianity is the one true religion. If you’d like to know more, send me a note After looking at the explanation, we shouldn’t trust the model! 24
25. Explanation for a good classifier It seems to be picking up on more reasonable things.. good! 25
26. Comparing models with explanations 26
27. Comparing models If we picked based on accuracy, we would get it wrong. 40 53 65 78 90 103 Guessing w/ Explanations % picked better model 89% of users identify the more trustworthy model 27
28. Problem with linear explanations Gives a “general idea”, but not precise. What can I change, and not change, to keep the same prediction? 28
29. Prediction Invariance Instance Set of Constraints on x Conjunction of constraints on individual features Quality of Constraints Whether all instances that satisfy the constraints e have the same prediction Interpretability Are the set of constraints small? Find a set of constraints so that any other change to the input does not change the prediction 29
30. Tabular Data 30
31. Visual QA 31
32. Image Classification 32
33. “Programs” as Explanations Instance Set of Programs Universe of all possible programs/functions on x Quality of Program Does the function e on instances similar to x perform similar to the model Interpretability Is the program short? There are so many choices for interpretable models: Rules, decision trees, decision sets, falling lists, sparse linear models, etc. Why decide between them, search over ALL possible “programs”! 33
34. Salary Prediction 34
35. Model-Agnostic Explanations Model-agnostic explanations are critical for understanding ML! Can be applied on future models as well LIME Local, Interpretable Model-Agnostic Explanations Linear explanations: useful for complex text and image classifiers Prediction Invariant explanations: a more precise definition “Programs” as explanations: generalizes over any representations 35
36. Thank you! LIME Local, Interpretable Model-Agnostic Explanations Linear explanations: useful for complex text and image classifiers Prediction Invariant explanations: a more precise definition “Programs” as explanations: generalizes over any representations www.sameersingh.org github.com/marcotcr/lime