Just-So Stories for AI: Explaining Black Box Predictions

Slide 1

Slide 1 text

Just-So Stories for AI Explaining Black Box Predictions Sam Ritchie (@sritchie) Strange Loop 2017 St Louis, Missouri

Slide 2

Slide 2 text

Hi!

Slide 3

Slide 3 text

Outline • ML Techniques for detecting Fraud • Interpretable Models vs “Black Box” Models • Explanations (?) for Black Box Models • Why this is so important!

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

-> =

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Problem?

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

MACHINE LEARNING! Thank you XKCD!

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

class BinaryClassifier { def predict(features: Map[String, FeatureValue]) : Double // % chance of fraud - [0, 1] }

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Decision Trees :)

Slide 22

Slide 22 text

Decision Trees :) • Interpretable

Slide 23

Slide 23 text

Decision Trees :) • Interpretable • Every decision has a built-in explanation!

Slide 24

Slide 24 text

Decision Trees :) • Interpretable • Every decision has a built-in explanation! • Easy to Trust

Slide 25

Slide 25 text

Decision Trees :) • Interpretable • Every decision has a built-in explanation! • Easy to Trust • NOT a Black Box.

Slide 26

Slide 26 text

Decision Trees :(

Slide 27

Slide 27 text

Decision Trees :( • Shallow trees are not very accurate

Slide 28

Slide 28 text

Decision Trees :( • Shallow trees are not very accurate • Deep trees are not interpretable

Slide 29

Slide 29 text

Decision Trees :( • Shallow trees are not very accurate • Deep trees are not interpretable • Deep trees can overﬁt to a training set

Slide 30

Slide 30 text

Decision Trees :( • Shallow trees are not very accurate • Deep trees are not interpretable • Deep trees can overﬁt to a training set • Limitations on Public Data

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Random Forest

Slide 33

Slide 33 text

Random Forest

Slide 34

Slide 34 text

Random Forest

Slide 35

Slide 35 text

Random Forest

Slide 36

Slide 36 text

Random Forest

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

200 Trees 12k Splits / Tree 2.4M Possible “Explanations”

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Post-Hoc Explanations

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Paths are Rules

Slide 46

Slide 46 text

case class Predicate( name: String, op: Operation, constant: FeatureValue ) val predicate = Predicate( “unique_card_ip_address_24_hrs”, Op.Gt, 10)

Slide 47

Slide 47 text

case class Explanation(preds: List[Predicate]) val explanations = Explanation( List( Predicate( “unique_card_ip_address_24_hrs”, Op.Gt, 10), Predicate( “billing_country_matches_card_country”, Op.NotEq, true), Predicate(“card_type”, Op.Eq, “mastercard”) ))

Slide 48

Slide 48 text

type Model = Vector[Explanation] type ExplanationModel = Vector[(Model, Label, ScoreRange)] Explanation Model Structure

Slide 49

Slide 49 text

Training Explanation Models 1. Generate a ton of possible predicates 2. weight predicates by precision given SOME recall (5% or higher) 3. Sample top-N predicates (1000 or so) 4. Set those aside 5. Combine all predicates to make N x N "explanations" 6. GO back to step 3! 7. Iterate... 8. Set aside the highest precision “explanation” of ALL. 9. Toggle the positives that this catches, go back to 2. 10. COVERAGE! (~80 explanations)

Slide 50

Slide 50 text

{ "prediction": 0.48, "threshold": 0.3, "explanation": [ { "feature": "unique_card_ip_address_24_hrs", "op": ">", "constant": 10 }, { "feature": "billing_country_matches_card_country", "op": "!=", "constant": true }, { "feature":"card_type", "op": "==", "constant": "mastercard" }]}

Slide 51

Slide 51 text

{ "feature": "unique_card_ip_address_24_hrs", "op": ">", "constant": 10 }

Slide 52

Slide 52 text

{ "feature": "unique_card_ip_address_24_hrs", "op": ">", "constant": 10 }

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

Slide 56

Slide 56 text

Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

Slide 57

Slide 57 text

Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

Slide 58

Slide 58 text

Credit: https://homes.cs.washington.edu/~marcotcr/blog/lime/

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

“Humans decisions might admit post-hoc interpretability despite the black box nature of human brains, revealing a contradiction between two popular notions of interpretability.”

Slide 63

Slide 63 text

“You can ask a human, but… what cognitive psychologists have discovered is that when you ask a human you’re not really getting at the decision process. They make a decision ﬁrst, and then you ask, and then they generate an explanation and that may not be the true explanation.” - Peter Norvig

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

P. 65, “Waking Up” by Sam Harris

Slide 66

Slide 66 text

No content

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

Dataism

Slide 69

Slide 69 text

No content

Slide 70

Slide 70 text

"In the rush to gain acceptance for machine learning and to emulate human intelligence, we should be careful not to reproduce pathological behavior at scale."

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

“If I apply for a loan and I get turned down, whether it’s by a human or by a machine, and I say what’s the explanation, and it says well you didn’t have enough collateral. That might be the right explanation or it might be it didn’t like my skin colour. And I can’t tell from that explanation.”