"Creating correct and capable classifiers" at PyDataAmsterdam 2018

My thoughts on starting with baseline estimators, visualising their capabilities, diagnosing where they might not work so well and digging to see if we trust the underlying data.

Blog: http://ianozsvald.com/2018/05/26/creating-correct-and-capable-classifiers-at-pydataamsterdam-2018/

Transcript

1. Creating Correct Classifiers PyDataAmsterdam 2018 Ian Ozsvald @IanOzsvald ModelInsight.io
2. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Introductions • I’m an engineering data scientist • Consulting in AI + Data Science for 15+ years Blog->IanOzsvald.com
3. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 NumFOCUS • Have you thanked a speaker, a volunteer and a NumFOCUS organiser yet? Lots of volunteered time – please say thanks • Thank contributors too!
4. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Goals today • Get a baseline model • Visualise errors & diagnose problem areas • Explain decisions • Github for examples:
5. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 pandas_profiling
6. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 pandas_profiling
7. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 DummyClassifier
8. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 DummyClassifier
9. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Eyeball imputed results
10. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 RandomForest
11. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 RandomForest
12. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 ConfusionMatrix (YellowBrick)
13. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Confusion’s Probabilities
14. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick)
15. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick) LogisticRegression classifier to show a contrast with lower AUC
16. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Worst Errors by Row
17. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Worst Errors by Row
18. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Errors by Major Feature
19. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features
20. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features Oddly similar cluster? Conflicted?
21. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 TSNE by features Features for this cluster - lots of imputed ages! We’ve filtered by x, y region on the TSNE plot
22. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Examine conflicted area Oddly similar cluster?
23. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley explanations
24. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley – model-wide behaviour
25. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 SHAPley – model-wide behaviour
26. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 SHAP summary_plot
27. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Closing... • Diagnose your ML just like you debug your code – explain its working to colleagues • Do you want training on topics like this? • Write-up + more: http://ianozsvald.com/ • Questions in exchange for beer :-) • Learnt something? Please send me a postcard! • See my longer diagnosis Notebook on github:
28. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 Appendix • Ian’s “Machine Learning Libraries You’d Wish You’d Knew” @ PyConUK 2017 • Ian’s “Using Machine Learning to solve a classification problem with scikit-learn” @ PyConUK 2016 • Gael Varoquaux’s tutorial “Understanding and diagnosing your machine-learning models” @ PyDataLondon 2018 http://gael-varoquaux.info/interpreting_ml_tuto/ • Also see Kat Jarmul’s keynote @ PyDataWarsaw 2017: https://blog.kjamistan.com/towards-interpretable-reliable-model s • Michał Łopuszyński @ PyDataWarsaw https://www.slideshare.net/lopusz/debugging-machinelearning
29. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] PyDataAmsterdam 2018 ROC Curve (YellowBrick) LogisticRegression classifier to show a contrast with lower AUC