Just-So Stories for AI: Explaining Black Box Predictions

Cd378611a91eb7852ae19cd582de718a?s=47 Sam Ritchie
September 29, 2017

Just-So Stories for AI: Explaining Black Box Predictions

As machine learning techniques become more powerful, humans and companies are offloading more and more ethical decisions to ML models. Which person should get a loan? Where should I direct my time and attention? Algorithms often outperform humans, so we cede our control happily and love the extra time and leverage this gives us.

There's lurking danger here. Many of the most successful machine learning algorithms are black boxes - they give us predictions without the "why" that accompanies human decision-making. Trust without understanding is a scary thing. Why did the self-driving car swerve into traffic, killing its driver? How did the robotic doctor choose that dose? The ability to demand a plausible explanation for each decision is humanity's key to maintaining control over our ethical development.

In this talk we'll explore several state-of-the-art strategies for explaining the decisions of our black box models. We'll talk about why some algorithms are so difficult to interpret and discuss the insights that our explanation-generating models can give us into free will and how humans invent explanations for our own decisions.

Cd378611a91eb7852ae19cd582de718a?s=128

Sam Ritchie

September 29, 2017
Tweet

Transcript

  1. Just-So Stories for AI Explaining Black Box Predictions Sam Ritchie

    (@sritchie) Strange Loop 2017 St Louis, Missouri
  2. Hi!

  3. Outline • ML Techniques for detecting Fraud • Interpretable Models

    vs “Black Box” Models • Explanations (?) for Black Box Models • Why this is so important!
  4. None
  5. None
  6. -> =

  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. Problem?

  14. None
  15. MACHINE LEARNING! Thank you XKCD!

  16. None
  17. class BinaryClassifier { def predict(features: Map[String, FeatureValue]) : Double //

    % chance of fraud - [0, 1] }
  18. None
  19. None
  20. None
  21. Decision Trees :)

  22. Decision Trees :) • Interpretable

  23. Decision Trees :) • Interpretable • Every decision has a

    built-in explanation!
  24. Decision Trees :) • Interpretable • Every decision has a

    built-in explanation! • Easy to Trust
  25. Decision Trees :) • Interpretable • Every decision has a

    built-in explanation! • Easy to Trust • NOT a Black Box.
  26. Decision Trees :(

  27. Decision Trees :( • Shallow trees are not very accurate

  28. Decision Trees :( • Shallow trees are not very accurate

    • Deep trees are not interpretable
  29. Decision Trees :( • Shallow trees are not very accurate

    • Deep trees are not interpretable • Deep trees can overfit to a training set
  30. Decision Trees :( • Shallow trees are not very accurate

    • Deep trees are not interpretable • Deep trees can overfit to a training set • Limitations on Public Data
  31. None
  32. Random Forest

  33. Random Forest

  34. Random Forest

  35. Random Forest

  36. Random Forest

  37. None
  38. 200 Trees 12k Splits / Tree 2.4M Possible “Explanations”

  39. None
  40. None
  41. None
  42. None
  43. Post-Hoc Explanations

  44. None
  45. Paths are Rules

  46. case class Predicate( name: String, op: Operation, constant: FeatureValue )

    val predicate = Predicate( “unique_card_ip_address_24_hrs”, Op.Gt, 10)
  47. case class Explanation(preds: List[Predicate]) val explanations = Explanation( List( Predicate(

    “unique_card_ip_address_24_hrs”, Op.Gt, 10), Predicate( “billing_country_matches_card_country”, Op.NotEq, true), Predicate(“card_type”, Op.Eq, “mastercard”) ))
  48. type Model = Vector[Explanation] type ExplanationModel = Vector[(Model, Label, ScoreRange)]

    Explanation Model Structure
  49. Training Explanation Models 1. Generate a ton of possible predicates

    2. weight predicates by precision given SOME recall (5% or higher) 3. Sample top-N predicates (1000 or so) 4. Set those aside 5. Combine all predicates to make N x N "explanations" 6. GO back to step 3! 7. Iterate... 8. Set aside the highest precision “explanation” of ALL. 9. Toggle the positives that this catches, go back to 2. 10. COVERAGE! (~80 explanations)
  50. { "prediction": 0.48, "threshold": 0.3, "explanation": [ { "feature": "unique_card_ip_address_24_hrs",

    "op": ">", "constant": 10 }, { "feature": "billing_country_matches_card_country", "op": "!=", "constant": true }, { "feature":"card_type", "op": "==", "constant": "mastercard" }]}
  51. { "feature": "unique_card_ip_address_24_hrs", "op": ">", "constant": 10 }

  52. { "feature": "unique_card_ip_address_24_hrs", "op": ">", "constant": 10 }

  53. None
  54. None
  55. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

  56. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

  57. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

  58. Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

  59. None
  60. None
  61. None
  62. “Humans decisions might admit post-hoc interpretability despite the black box

    nature of human brains, revealing a contradiction between two popular notions of interpretability.”
  63. “You can ask a human, but… what cognitive psychologists have

    discovered is that when you ask a human you’re not really getting at the decision process. They make a decision first, and then you ask, and then they generate an explanation and that may not be the true explanation.” - Peter Norvig
  64. None
  65. P. 65, “Waking Up” by Sam Harris

  66. None
  67. None
  68. Dataism

  69. None
  70. "In the rush to gain acceptance for machine learning and

    to emulate human intelligence, we should be careful not to reproduce pathological behavior at scale."
  71. None
  72. None
  73. “If I apply for a loan and I get turned

    down, whether it’s by a human or by a machine, and I say what’s the explanation, and it says well you didn’t have enough collateral. That might be the right explanation or it might be it didn’t like my skin colour. And I can’t tell from that explanation.”
  74. None
  75. In Conclusion…

  76. Sam Ritchie (@sritchie) Strange Loop 2017 :: St Louis, Missouri

    Thanks!