Debugging Classifiers

Transcript

1. Debugging Classifiers Olamilekan Wahab Inspired by Michał Łopuszyński talks on Debugging ML
2. Classifier Failure Modes 1. Classifier does not work at all 2. Classifier works but it can be better 3. Classifier works on test data but fails in production 4. Classifier deteriorates over time.
3. Classifier does not work at all • Check everything • Check your code • Read the docs • Check your data • Check your data quality • Data providers tend to overestimate the quality of their data • Understand your data • Do an exploration • Visualize • Talk to domain experts • Check for outliers • Make sure your data is • Correct • Complete • Coherent, • Stationary • De-duplicated • Up to date • Representative* • Ask Questions [Stackoverflow, Twitter, Forums]
4. Classifier works well but it can be better • The Featurists • The Speculators • The Localizers • The “Convolutors” • The “Trainalyzers”
5. Featurists • Find features important to your model • Do some feature selections • Tools : • eli5 • lime • skopt • yellowbrick • pandas-profiling
6. Spectulators • Check how model corresponds to change in a variable
7. Speculators • Find DDP [Misclassified cases in cross validation] • Examine DDPs • Remove DDPs
8. Localizers
9. LIME
10. Convolutors [Convnets]
11. Trainalyzers
13. Classifier works on test data but fails in production • Check for overfitting • Check for data leakage • Check for covariate shift
14. • https://github.com/ianozsvald/data_science_delivered/blob/master/ ml_creating_correct_capable_classifiers.ipynb