HOWTO Make Your Future Data Science Team Love You

My talk from PyData NYC 2014. For more info on the talk, go to http://blog.sashalaundy.com/talks/data-audit/

There's a lot of the talk that is not in these slides—the video will make much more sense.

Transcript

1. HOWTO make your future data science team LOVE YOU
   
2. Let me tell you a data story.
3. Let me tell you a data horror story.
4. BAD DATA
5. This is for those who START
6. Hi. I’m Sasha.
9. complete?
   Is your data set correct?
   connectable?
10. Why bother?
12. ! Algorithms are useless if you have no data.
13. What is CRITICAL?
14. What are your business needs?
15. What will move the needle?
18. What is CRITICAL?
19. complete?
    Is your data set correct?
    connectable?
20. Is your data set COMPLETE?
21. Are business-critical fields OK?
22. An example: Do we refer better traffic?
23. http://example.com/123? referrer=ourclient.com
24. http://example.com/123? referrer=ourclient.com
25. Are business-critical fields OK?
26. Check VOLUME
27. Google Analytics: 500,000 Logs: 5,000 Missing: 99%
28. Check VOLUME
29. Not just ERRORS
30. Articles Newsletters Audio Video Donate
31. Articles Newsletters Audio Video Donate
32. Not just ERRORS
33. Storage is cheap. SO CHEAP
34. Is your data set COMPLETE?
35. complete?
    Is your data set correct?
    connectable?
36. Is your data set CORRECT?
37. Check against EXISTING KNOWLEDGE
38. about 60% male
41. CSVKIT
42. CSVKIT It’s Python!
44. in2csv sql2csv csvjson csvlook csvgrep csvcut
45. $ cat logs.csv uid, name, gender, join_date 1, John Smith, M, 01/01/2001 2, Sarah Stein, F, 02/02/2002 3, Lee Jones, '', 03/03/2003
46. $ cat logs.csv | csvcut -n 1: uid 2: name 3: gender 4: join_date
47. $ cat logs.csv | csvcut -c 3 gender M F ''
48. $ cat logs.csv | csvcut -c 3 | sort | uniq -c 598 F 6254 M 1 gender 321 ''
49. $ cat logs.csv | csvcut -c 3 | sort | uniq -c 598 F 6254 M 1 gender 321 '' 90% male :(
50. DIY CLI
51. Is your data set CORRECT?
52. complete?
    Is your data set correct?
    connectable?
53. Is your data set CONNECTABLE?
54. Can you join?
55. 11201 $95,000 10038 $35,000 11456 $65,000 Census Income Data
56. Noise 10014 Rats 11218 No heat 11201 3-1-1 Calls
57. 11201 $95,000 10038 $35,000 11456 $65,000 Noise 10014 Rats 11218 No heat 11201 Census Income Data 3-1-1 Calls
58. 11201 $95,000 10038 $35,000 11456 $65,000 Noise 10014 Rats 11218 No heat 11201 Census Income Data 3-1-1 Calls
59. 11201 $95,000 No heat 11219 $75,000 Bagel ! theft 10095 $35,000 Pigeon attack Joined table
61. ACTUALLY DO THE JOIN
62. Check against EXISTING KNOWLEDGE
63. Histograms
64. github.com/bitly/data_hacks
65. $ cat /tmp/data | histogram.py # NumSamples = 29; Max = 10.00; Min = 1.00 # Mean = 4.379310; Variance = 5.131986; SD = 2.265389 # each * represents a count of 1 1.0000 - 1.9000 [ 1]: * 1.9000 - 2.8000 [ 5]: ***** 2.8000 - 3.7000 [ 8]: ******** 3.7000 - 4.6000 [ 3]: *** 4.6000 - 5.5000 [ 4]: **** 5.5000 - 6.4000 [ 2]: ** 6.4000 - 7.3000 [ 3]: *** 7.3000 - 8.2000 [ 1]: * 8.2000 - 9.1000 [ 1]: * 9.1000 - 10.0000 [ 1]: *
66. github.com/bitly/data_hacks
67. Can you join?
68. OOPS: no joinable IDs
69. OOPS: no unique IDs at all
70. Is your data set CONNECTABLE?
71. complete?
    Is your data set correct?
    connectable?
72. What next?
73. THANKS sashalaundy.com/talks/data-audit