Just-So Stories for AI: Explaining Black Box Predictions

As machine learning techniques become more powerful, humans and companies are offloading more and more ethical decisions to ML models. Which person should get a loan? Where should I direct my time and attention? Algorithms often outperform humans, so we cede our control happily and love the extra time and leverage this gives us.

There's lurking danger here. Many of the most successful machine learning algorithms are black boxes - they give us predictions without the "why" that accompanies human decision-making. Trust without understanding is a scary thing. Why did the self-driving car swerve into traffic, killing its driver? How did the robotic doctor choose that dose? The ability to demand a plausible explanation for each decision is humanity's key to maintaining control over our ethical development.

In this talk we'll explore several state-of-the-art strategies for explaining the decisions of our black box models. We'll talk about why some algorithms are so difficult to interpret and discuss the insights that our explanation-generating models can give us into free will and how humans invent explanations for our own decisions.

Transcript

1. Just-So Stories for AI
   Explaining Black Box Predictions
   Sam Ritchie (@sritchie)
   Strange Loop 2017
   St Louis, Missouri
   

   View Slide

2. Hi!
   

   View Slide

3. Outline
   • ML Techniques for detecting Fraud
   • Interpretable Models vs “Black Box” Models
   • Explanations (?) for Black Box Models
   • Why this is so important!
   

   View Slide

4. View Slide

5. View Slide

6. -> =
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. Problem?
    

    View Slide

14. View Slide

15. MACHINE LEARNING!
    Thank you XKCD!
    

    View Slide

16. View Slide

17. class BinaryClassifier {
    def predict(features: Map[String, FeatureValue])
    : Double // % chance of fraud - [0, 1]
    }
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. Decision Trees :)
    

    View Slide

22. Decision Trees :)
    • Interpretable
    

    View Slide

23. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!
    

    View Slide

24. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!
    • Easy to Trust
    

    View Slide

25. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!
    • Easy to Trust
    • NOT a Black Box.
    

    View Slide

26. Decision Trees :(
    

    View Slide

27. Decision Trees :(
    • Shallow trees are not very accurate
    

    View Slide

28. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable
    

    View Slide

29. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable
    • Deep trees can overfit to a training set
    

    View Slide

30. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable
    • Deep trees can overfit to a training set
    • Limitations on Public Data
    

    View Slide

31. View Slide

32. Random Forest
    

    View Slide

33. Random Forest
    

    View Slide

34. Random Forest
    

    View Slide

35. Random Forest
    

    View Slide

36. Random Forest
    

    View Slide

37. View Slide

38. 200 Trees
    12k Splits / Tree
    2.4M Possible “Explanations”
    

    View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. Post-Hoc Explanations
    

    View Slide

44. View Slide

45. Paths are Rules
    

    View Slide

46. case class Predicate(
    name: String,
    op: Operation,
    constant: FeatureValue
    )
    val predicate = Predicate(
    “unique_card_ip_address_24_hrs”,
    Op.Gt,
    10)
    

    View Slide

47. case class Explanation(preds: List[Predicate])
    val explanations = Explanation(
    List(
    Predicate(
    “unique_card_ip_address_24_hrs”, Op.Gt, 10),
    Predicate(
    “billing_country_matches_card_country”,
    Op.NotEq, true),
    Predicate(“card_type”, Op.Eq, “mastercard”)
    ))
    

    View Slide

48. type Model = Vector[Explanation]
    type ExplanationModel =
    Vector[(Model, Label, ScoreRange)]
    Explanation Model Structure
    

    View Slide

49. Training Explanation Models
    1. Generate a ton of possible predicates
    2. weight predicates by precision given SOME recall (5% or higher)
    3. Sample top-N predicates (1000 or so)
    4. Set those aside
    5. Combine all predicates to make N x N "explanations"
    6. GO back to step 3!
    7. Iterate...
    8. Set aside the highest precision “explanation” of ALL.
    9. Toggle the positives that this catches, go back to 2.
    10. COVERAGE! (~80 explanations)
    

    View Slide

50. {
    "prediction": 0.48,
    "threshold": 0.3,
    "explanation": [
    {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    },
    {
    "feature": "billing_country_matches_card_country",
    "op": "!=",
    "constant": true
    },
    {
    "feature":"card_type",
    "op": "==",
    "constant": "mastercard"
    }]}
    

    View Slide

51. {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    }
    

    View Slide

52. {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    }
    

    View Slide

53. View Slide

54. View Slide

55. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/
    

    View Slide

56. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/
    

    View Slide

57. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/
    

    View Slide

58. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/
    

    View Slide

59. View Slide

60. View Slide

61. View Slide

62. “Humans decisions might admit
    post-hoc interpretability despite the
    black box nature of human brains,
    revealing a contradiction between
    two popular notions of
    interpretability.”
    

    View Slide

63. “You can ask a human, but… what
    cognitive psychologists have
    discovered is that when you ask a
    human you’re not really getting at the
    decision process. They make a decision
    first, and then you ask, and then they
    generate an explanation and that may
    not be the true explanation.”
    - Peter Norvig
    

    View Slide

64. View Slide

65. P. 65, “Waking Up” by Sam Harris
    

    View Slide

66. View Slide

67. View Slide

68. Dataism
    

    View Slide

69. View Slide

70. "In the rush to gain acceptance for
    machine learning and to emulate
    human intelligence, we should be
    careful not to reproduce
    pathological behavior at scale."
    

    View Slide

71. View Slide

72. View Slide

73. “If I apply for a loan and I get turned
    down, whether it’s by a human or by a
    machine, and I say what’s the
    explanation, and it says well you didn’t
    have enough collateral. That might be
    the right explanation or it might be it
    didn’t like my skin colour. And I can’t
    tell from that explanation.”
    

    View Slide

74. View Slide

75. In Conclusion…
    

    View Slide

76. Sam Ritchie (@sritchie)
    Strange Loop 2017 :: St Louis, Missouri
    Thanks!
    

    View Slide