On the Diagramatic Diagnosis of Data (BuapestBI 2018)

The wrong way to start your machine learning project is to “chuck everything into a model to see what happens”. The better way is to visualise your data to expose the relationships that you see, to confirm that your data looks good and to identify problems that are likely to make your life difficult. You’ll save time, you’ll understand “why” your data works and you’ll uncover problems sooner.
We’ll review ways to quickly and visually diagnose your data, to check it meets your assumptions and to prepare it for discussion with your colleagues. We’ll look at tools including Pandas, Seaborn and Pandas Profiling. At the end you’ll have new tools to help you confidently investigate new data with your associates.
This talk introduces Ian’s new discover_feature_relationships tool which will save you time during your Exploratory Data Analysis phase.
http://budapestbiforum.hu/2018/hu/eloadasok/on-the-diagramatic-diagnosis-of-data-ian-ozsvald-mor-consulting-ltd/

Transcript

1. Diagramatic Diagnosis of Data BudapestBI 2018 Ian Ozsvald @IanOzsvald ModelInsight.io
2. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Introductions • I’m an engineering data scientist • Consulting in AI + Data Science for 15+ years Blog->IanOzsvald.com
3. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Community Announcement! • Have you thanked a speaker, an organiser or Bence yet? Lots of volunteered time – please say thanks • Thank contributors too! • Did I take a photo?
4. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Goals today • How long since you had brand new data? • Univariate investigations • Show relationships with seaborn • discover_feature_relationships – my new tool (feedback please!) • Data stories
5. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 pandas_profiling (Titanic)
6. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 pandas_profiling
7. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Describing Titanic relationships
8. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Seaborn (Titanic data) Non-formatted default pivot result
9. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Seaborn (Titanic data)
10. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Seaborn (Titanic data)
11. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Discovering relationships • Project is on GitHub • Shows correlations and machine learned relationships for all feature pairs • RandomForest + cross validation + some assumptions • Categories encoded->Labels • Feedback please!
12. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Spearman feature correlations
13. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Discovering relationships
14. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Pandas scatter LSTAT vs MEDV
15. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Seaborn JointGrid with alpha
16. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Seaborn hex jointplot
17. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Diff the upper and lower triangles CRIM vs RAD might be interesting as both sides had "some" predictive power
18. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 A non-symmetric relationship CRIM can predict RAD but RAD poorly predicts CRIM So maybe we need better data?
19. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 NetworkX to show relationships Who predicts MEDV directly and indirectly? What new data might we try to get, given these relationships?
20. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Data Stories (@bertil_hatt) • A short report describing the data and proposing things we could do with it • Stuff we trust (or don’t) • Interesting or unexpected relationships • Missing data (e.g. missingno) • Propose experiments that we might run on this data which generate a benefit • Document, don’t forget! https://medium.com/@bertil_hatt/what-does-bad-data-look-like-91dc2a7bcb7a
21. Ian.Ozsvald@ModelInsight.io @IanOzsvald[.com] BudapestBI 2018 Conclusions • Visualise and communicate all of your data relationships • Visit PyDataLondon 2019 :-) • I’d love a postcard if you learned something? • See more examples: https://github.com/ianozsvald