"Creating correct and capable classifiers" at PyDataLondon 2018

Iteratively building a classifier requires a mix of skill, diagnostic ability and guesswork. I'll lay out a framework that helps you build reliable classifiers with greater confidence and less random guesswork. Tools demonstrated will include sklearn, YellowBrick, ELI5, pandas_profiling and skopt.

Blog: http://ianozsvald.com/2018/04/30/pydatalondon-2018-and-creating-correct-and-capable-classifiers/

Transcript

1. Creating Correct Classifiers PyDataLondon 2018 Ian Ozsvald @IanOzsvald ModelInsight.io
2. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Introductions • I’m an engineering data scientist • Consulting in AI + Data Science for 15+ years Blog->IanOzsvald.com
3. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 NumFOCUS • Have you thanked a speaker, a volunteer and a NumFOCUS organiser yet? Lots of volunteered time – please say thanks • Leah can’t make it due to illness – please Tweet “@numfocus Leah get well soon from London!” • Book signing (High Performance Python) at lunch
4. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Goals today • Get a baseline model • Visualise errors & diagnose problem areas • Explain decisions • Github for examples:
5. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 pandas_profiling
6. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 pandas_profiling
7. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 DummyClassifier
8. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 DummyClassifier
9. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Eyeball imputed results
10. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 RandomForest
11. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 RandomForest
12. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 ConfusionMatrix (YellowBrick)
13. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Confusion’s Probabilities
14. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 PermutationImportance ELI5 https://github.com/TeamHG-Memex/eli5/issues/256
15. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Worst Errors by Row
16. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Worst Errors by Row
17. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Errors by Major Feature
18. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 TSNE by features
19. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 TSNE by features Oddly similar cluster? Conflicted?
20. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 TSNE by features Features for this cluster - lots of imputed ages! We’ve filtered by x, y region on the TSNE plot
21. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Examine conflicted area Oddly similar cluster?
22. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 ELI5 show_prediction
23. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 ELI5 show_prediction
24. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Last mentions • skopt’s BayesSearchCV perhaps “beats” RandomizedSearchCV & GridSearchCV • New iteration of this talk for PyDataAmsterdam 2018 in 1 month (with SHAP replacing ELI5 + other tools) • If you can’t reliably explain why a prediction happens – do you really understand your model?
25. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Closing... • Diagnose your ML just like you debug your code – explain its working to colleagues • Do you want training on topics like this? • Write-up + more: http://ianozsvald.com/ • Questions in exchange for beer :-) • Learnt something? Please send me a postcard! • See my longer diagnosis Notebook on github:
26. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataLondon 2018 Appendix • Ian’s “Machine Learning Libraries You’d Wish You’d Knew” @ PyConUK 2017 • Ian’s “Using Machine Learning to solve a classification problem with scikit-learn” @ PyConUK 2016 • Gael Varoquaux’s tutorial “Understanding and diagnosing your machine-learning models” @ PyDataLondon 2018 http://gael-varoquaux.info/interpreting_ml_tuto/ • Also see Kat Jarmul’s keynote @ PyDataWarsaw 2017: https://blog.kjamistan.com/towards-interpretable-reliable-model s • Michał Łopuszyński @ PyDataWarsaw https://www.slideshare.net/lopusz/debugging-machinelearning