$30 off During Our Annual Pro Sale. View Details »

Just-So Stories for AI: Explaining Black Box Predictions

Sam Ritchie
September 29, 2017

Just-So Stories for AI: Explaining Black Box Predictions

As machine learning techniques become more powerful, humans and companies are offloading more and more ethical decisions to ML models. Which person should get a loan? Where should I direct my time and attention? Algorithms often outperform humans, so we cede our control happily and love the extra time and leverage this gives us.

There's lurking danger here. Many of the most successful machine learning algorithms are black boxes - they give us predictions without the "why" that accompanies human decision-making. Trust without understanding is a scary thing. Why did the self-driving car swerve into traffic, killing its driver? How did the robotic doctor choose that dose? The ability to demand a plausible explanation for each decision is humanity's key to maintaining control over our ethical development.

In this talk we'll explore several state-of-the-art strategies for explaining the decisions of our black box models. We'll talk about why some algorithms are so difficult to interpret and discuss the insights that our explanation-generating models can give us into free will and how humans invent explanations for our own decisions.

Sam Ritchie

September 29, 2017
Tweet

More Decks by Sam Ritchie

Other Decks in Programming

Transcript

  1. Just-So Stories for AI
    Explaining Black Box Predictions
    Sam Ritchie (@sritchie)
    Strange Loop 2017
    St Louis, Missouri

    View Slide

  2. Hi!

    View Slide

  3. Outline
    • ML Techniques for detecting Fraud
    • Interpretable Models vs “Black Box” Models
    • Explanations (?) for Black Box Models
    • Why this is so important!

    View Slide

  4. View Slide

  5. View Slide

  6. -> =

    View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. Problem?

    View Slide

  14. View Slide

  15. MACHINE LEARNING!
    Thank you XKCD!

    View Slide

  16. View Slide

  17. class BinaryClassifier {
    def predict(features: Map[String, FeatureValue])
    : Double // % chance of fraud - [0, 1]
    }

    View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. Decision Trees :)

    View Slide

  22. Decision Trees :)
    • Interpretable

    View Slide

  23. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!

    View Slide

  24. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!
    • Easy to Trust

    View Slide

  25. Decision Trees :)
    • Interpretable
    • Every decision has a built-in explanation!
    • Easy to Trust
    • NOT a Black Box.

    View Slide

  26. Decision Trees :(

    View Slide

  27. Decision Trees :(
    • Shallow trees are not very accurate

    View Slide

  28. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable

    View Slide

  29. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable
    • Deep trees can overfit to a training set

    View Slide

  30. Decision Trees :(
    • Shallow trees are not very accurate
    • Deep trees are not interpretable
    • Deep trees can overfit to a training set
    • Limitations on Public Data

    View Slide

  31. View Slide

  32. Random Forest

    View Slide

  33. Random Forest

    View Slide

  34. Random Forest

    View Slide

  35. Random Forest

    View Slide

  36. Random Forest

    View Slide

  37. View Slide

  38. 200 Trees
    12k Splits / Tree
    2.4M Possible “Explanations”

    View Slide

  39. View Slide

  40. View Slide

  41. View Slide

  42. View Slide

  43. Post-Hoc Explanations

    View Slide

  44. View Slide

  45. Paths are Rules

    View Slide

  46. case class Predicate(
    name: String,
    op: Operation,
    constant: FeatureValue
    )
    val predicate = Predicate(
    “unique_card_ip_address_24_hrs”,
    Op.Gt,
    10)

    View Slide

  47. case class Explanation(preds: List[Predicate])
    val explanations = Explanation(
    List(
    Predicate(
    “unique_card_ip_address_24_hrs”, Op.Gt, 10),
    Predicate(
    “billing_country_matches_card_country”,
    Op.NotEq, true),
    Predicate(“card_type”, Op.Eq, “mastercard”)
    ))

    View Slide

  48. type Model = Vector[Explanation]
    type ExplanationModel =
    Vector[(Model, Label, ScoreRange)]
    Explanation Model Structure

    View Slide

  49. Training Explanation Models
    1. Generate a ton of possible predicates
    2. weight predicates by precision given SOME recall (5% or higher)
    3. Sample top-N predicates (1000 or so)
    4. Set those aside
    5. Combine all predicates to make N x N "explanations"
    6. GO back to step 3!
    7. Iterate...
    8. Set aside the highest precision “explanation” of ALL.
    9. Toggle the positives that this catches, go back to 2.
    10. COVERAGE! (~80 explanations)

    View Slide

  50. {
    "prediction": 0.48,
    "threshold": 0.3,
    "explanation": [
    {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    },
    {
    "feature": "billing_country_matches_card_country",
    "op": "!=",
    "constant": true
    },
    {
    "feature":"card_type",
    "op": "==",
    "constant": "mastercard"
    }]}

    View Slide

  51. {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    }

    View Slide

  52. {
    "feature": "unique_card_ip_address_24_hrs",
    "op": ">",
    "constant": 10
    }

    View Slide

  53. View Slide

  54. View Slide

  55. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

    View Slide

  56. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

    View Slide

  57. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

    View Slide

  58. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

    View Slide

  59. View Slide

  60. View Slide

  61. View Slide

  62. “Humans decisions might admit
    post-hoc interpretability despite the
    black box nature of human brains,
    revealing a contradiction between
    two popular notions of
    interpretability.”

    View Slide

  63. “You can ask a human, but… what
    cognitive psychologists have
    discovered is that when you ask a
    human you’re not really getting at the
    decision process. They make a decision
    first, and then you ask, and then they
    generate an explanation and that may
    not be the true explanation.”
    - Peter Norvig

    View Slide

  64. View Slide

  65. P. 65, “Waking Up” by Sam Harris

    View Slide

  66. View Slide

  67. View Slide

  68. Dataism

    View Slide

  69. View Slide

  70. "In the rush to gain acceptance for
    machine learning and to emulate
    human intelligence, we should be
    careful not to reproduce
    pathological behavior at scale."

    View Slide

  71. View Slide

  72. View Slide

  73. “If I apply for a loan and I get turned
    down, whether it’s by a human or by a
    machine, and I say what’s the
    explanation, and it says well you didn’t
    have enough collateral. That might be
    the right explanation or it might be it
    didn’t like my skin colour. And I can’t
    tell from that explanation.”

    View Slide

  74. View Slide

  75. In Conclusion…

    View Slide

  76. Sam Ritchie (@sritchie)
    Strange Loop 2017 :: St Louis, Missouri
    Thanks!

    View Slide