Creating Correct Classifiers PyDataAmsterdam 2018 Ian Ozsvald @IanOzsvald ModelInsight.io

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Introductions ● I’m an engineering data scientist ● Consulting in AI + Data Science for 15+ years Blog->IanOzsvald.com

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 NumFOCUS ● Have you thanked a speaker, a volunteer and a NumFOCUS organiser yet? Lots of volunteered time – please say thanks ● Thank contributors too!

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Goals today ● Get a baseline model ● Visualise errors & diagnose problem areas ● Explain decisions ● Github for examples:

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 pandas_profiling

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 pandas_profiling

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 DummyClassifier

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 DummyClassifier

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Eyeball imputed results

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 RandomForest

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 RandomForest

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 ConfusionMatrix (YellowBrick)

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Confusion’s Probabilities

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick)

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick) LogisticRegression classifier to show a contrast with lower AUC

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Worst Errors by Row

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Worst Errors by Row

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Errors by Major Feature

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features Oddly similar cluster? Conflicted?

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features Features for this cluster - lots of imputed ages! We’ve filtered by x, y region on the TSNE plot

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Examine conflicted area Oddly similar cluster?

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley explanations

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley – model-wide behaviour

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley – model-wide behaviour

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 SHAP summary_plot

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Closing... ● Diagnose your ML just like you debug your code – explain its working to colleagues ● Do you want training on topics like this? ● Write-up + more: http://ianozsvald.com/ ● Questions in exchange for beer :-) ● Learnt something? Please send me a postcard! ● See my longer diagnosis Notebook on github:

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 Appendix ● Ian’s “Machine Learning Libraries You’d Wish You’d Knew” @ PyConUK 2017 ● Ian’s “Using Machine Learning to solve a classification problem with scikit-learn” @ PyConUK 2016 ● Gael Varoquaux’s tutorial “Understanding and diagnosing your machine-learning models” @ PyDataLondon 2018 http://gael-varoquaux.info/interpreting_ml_tuto/ ● Also see Kat Jarmul’s keynote @ PyDataWarsaw 2017: https://blog.kjamistan.com/towards-interpretable-reliable-model s ● Michał Łopuszyński @ PyDataWarsaw https://www.slideshare.net/lopusz/debugging-machinelearning

[email protected] @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick) LogisticRegression classifier to show a contrast with lower AUC