Demystifying complex models: Learnings from using SHAP explainers in the real world at GoCardless (London Python meetup)

Complex algorithms (GBDTs, deep neural nets, etc.) have the ability to perform far better than linear models because they can capture non-linear behaviour and interaction effects. However, interpreting these models was typically more difficult or in some cases (e.g. neural nets) even impossible. For ML applications where explainability on the local (individual) level was key, this typically meant we were limited to simpler, more explainable models like Logistic Regression.

Recently we have seen advances in using simpler, locally interpretable models that are trained on top of the outputs of complex models. SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model.

In this talk, we will share our experience of using SHAP in a real-world ML application, the changes we made to both our training and prediction phases and considerations to take into account when using SHAP.

Transcript

1. Demystifying complex models What we learned using SHAP explainers in the field Yasir Ekinci
2. Start with why
3. Start with why Why is this score high? How do we know if the model is right? If this is not fraud, why did the model make a mistake?
4. The dilemma: simple vs complex models Simple model Complex model Explainable Accurate
5. WTF is SHAP? SHapley Additive exPlanations a unified approach to explain the output of any machine learning model [kind of]
6. Diagram: how SHAP fits in the ML process Input data Prediction Explanation SHAP model Complex model https://github.com/slundberg/shap/blob/master/README.md
7. _ How we use SHAP
8. Estimator: wrapping a classifier + explainer Estimator Untrained complex model Placeholder for Explainer https://gist.github.com/yoziru-desu/8093eada2b612ca144b233d538488340
9. Training phase: adding explainer to fit function Fit model with training data Trained complex model SHAP TreeExplainer Fit explainer with trained model
10. Prediction phase: using a new explain method Explanation SHAP TreeExplainer Input data
11. What do SHAP explanations look like? Feature Input value Explanation (log odds) feature_2 7.81804 + 5.74192 feature_7 TRUE +0.717551 feature_18 FALSE -0.33685 feature_28 0.208333 +0.554768 feature_17 0.87623 -0.27466
12. _ What we learned
13. 1 SHAP adds additional computation and doesn’t scale as well
14. SHAP explanations don’t come for free 6.8x explanation time 2.6x prediction time 10x data
15. 2 SHAP values work well for explaining local predictions
16. Core properties of explainability on the local level Positive vs negative impact Relative importance SHAP values are additive Feature Explanation (log odds) feature_2 + 5.74192 feature_7 +0.717551 feature_28 +0.554768 feature_17 -0.27466 feature_18 -0.33685
17. Diagnosing score changes by using SHAP values Why did the score suddenly change?
18. 3 Adding SHAP to your complex model doesn’t solve global explainability completely
19. Global SHAP summary: what does it tell us? Positive vs negative impact Relative importance But what about categorical features? Missing or non-numeric
20. Visualising a single feature can uncover relationships Is the relationship between the output and this feature linear / exponential / ... Density of the impact: how likely will a certain feature value occur?
21. Interaction effects: still hard to explain Doable for 2 features (2D ) Human brain is limited to 3 dimensions What’s the interaction with the other 20 features? (20D) feature_13 = TRUE feature_13 = FALSE
22. Summary 1 SHAP is relatively easy to integrate 2 It comes with a computational cost 3 Works well for explaining local predictions 4 Makes the model more explainable globally 5 But interaction effects are still hard to understand
23. Questions?
24. We’re hiring! Want to know more about the job opportunities within our data and engineering teams? Speak with Haron or email haron@gocardless.com