Creating Correct Classifiers PyDataLondon 2018 Ian Ozsvald @IanOzsvald ModelInsight.io 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Introductions ● I’m an engineering data scientist ● Consulting in AI + Data Science for 15+ years Blog->IanOzsvald.com 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 NumFOCUS ● Have you thanked a speaker, a volunteer and a NumFOCUS organiser yet? Lots of volunteered time – please say thanks ● Leah can’t make it due to illness – please Tweet “@numfocus Leah get well soon from London!” ● Book signing (High Performance Python) at lunch 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Goals today ● Get a baseline model ● Visualise errors & diagnose problem areas ● Explain decisions ● Github for examples: 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 pandas_profiling 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 pandas_profiling 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 DummyClassifier 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 DummyClassifier 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Eyeball imputed results 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 RandomForest 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 RandomForest 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 ConfusionMatrix (YellowBrick) 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Confusion’s Probabilities 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 PermutationImportance ELI5 https://github.com/TeamHG-Memex/eli5/issues/256 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Worst Errors by Row 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Worst Errors by Row 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Errors by Major Feature 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 TSNE by features 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 TSNE by features Oddly similar cluster? Conflicted? 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 TSNE by features Features for this cluster - lots of imputed ages! We’ve filtered by x, y region on the TSNE plot 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Examine conflicted area Oddly similar cluster? 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 ELI5 show_prediction 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 ELI5 show_prediction 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Last mentions ● skopt’s BayesSearchCV perhaps “beats” RandomizedSearchCV & GridSearchCV ● New iteration of this talk for PyDataAmsterdam 2018 in 1 month (with SHAP replacing ELI5 + other tools) ● If you can’t reliably explain why a prediction happens – do you really understand your model? 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Closing... ● Diagnose your ML just like you debug your code – explain its working to colleagues ● Do you want training on topics like this? ● Write-up + more: http://ianozsvald.com/ ● Questions in exchange for beer :-) ● Learnt something? Please send me a postcard! ● See my longer diagnosis Notebook on github: 

[email protected] @IanOzsvald[.com] PyDataLondon 2018 Appendix ● Ian’s “Machine Learning Libraries You’d Wish You’d Knew” @ PyConUK 2017 ● Ian’s “Using Machine Learning to solve a classification problem with scikit-learn” @ PyConUK 2016 ● Gael Varoquaux’s tutorial “Understanding and diagnosing your machine-learning models” @ PyDataLondon 2018 http://gael-varoquaux.info/interpreting_ml_tuto/ ● Also see Kat Jarmul’s keynote @ PyDataWarsaw 2017: https://blog.kjamistan.com/towards-interpretable-reliable-model s ● Michał Łopuszyński @ PyDataWarsaw https://www.slideshare.net/lopusz/debugging-machinelearning