HOWTO Make Your Future Data Science Team Love You

My talk from Strata San Jose 2015.

You'll have a better time if you watch the video, which will be on my site sasha.io.

Transcript

1. HOWTO make your future data science team LOVE YOU
   
2. Let me tell you a data story.
3. Let me tell you a data horror story.
4. BAD DATA
5. This is for those who START
8. complete?
   Is your data set correct?
   connectable?
9. Why bother?
11. ! Algorithms are useless if you have bad data.
12. "It’s worth noting the obvious: without a reliable and complete data flow, a Hadoop cluster is little more than a very expensive and difficult-to-assemble space heater.” ! Jay Kreps, I Heart Logs
13. I Heart Logs
15. What is CRITICAL?
16. What are your business needs?
17. What will move the needle?
19. What is CRITICAL?
20. complete?
    Is your data set correct?
    connectable?
21. Is your data set COMPLETE?
22. Are business-critical fields OK?
23. An example: Do we refer better traffic?
24. http://example.com/123? referrer=ourclient.com
25. http://example.com/123? referrer=ourclient.com
26. Are business-critical fields OK?
27. Check VOLUME
28. Google Analytics: 500,000 Logs: 5,000 Missing: 99%
29. Check VOLUME
30. Capture EVENTS, not just errors
31. Articles Newsletters Audio Video Donate
32. Articles Newsletters Audio Video Donate
33. Not just ERRORS
34. Storage is cheap. SO CHEAP
35. Is your data set COMPLETE?
36. complete?
    Is your data set correct?
    connectable?
37. Is your data set CORRECT?
38. Check against EXISTING KNOWLEDGE
39. about 60% male
40. $_
42. CSVKIT
43. $ cat logs.csv uid, name, gender, join_date 1, John Smith, M, 01/01/2001 2, Sarah Stein, F, 02/02/2002 3, Lee Jones, '', 03/03/2003
44. $ cat logs.csv | csvcut -c 3 gender M F ''
45. $ cat logs.csv | csvcut -c 3 | sort | uniq -c 598 F 6254 M 1 gender 321 ''
46. $ cat logs.csv | csvcut -c 3 | sort | uniq -c 598 F 6254 M 1 gender 321 '' 90% male :(
47. OTHER TOOLS bitly’s data_hacks bubbles datagristle
48. $ cat /tmp/data | histogram.py # NumSamples = 29; Max = 10.00; Min = 1.00 # Mean = 4.379310; Variance = 5.131986; SD = 2.265389 # each * represents a count of 1 1.0000 - 1.9000 [ 1]: * 1.9000 - 2.8000 [ 5]: ***** 2.8000 - 3.7000 [ 8]: ******** 3.7000 - 4.6000 [ 3]: *** 4.6000 - 5.5000 [ 4]: **** 5.5000 - 6.4000 [ 2]: ** 6.4000 - 7.3000 [ 3]: *** 7.3000 - 8.2000 [ 1]: * 8.2000 - 9.1000 [ 1]: * 9.1000 - 10.0000 [ 1]: *
49. Is your data set CORRECT?
50. complete?
    Is your data set correct?
    connectable?
51. Is your data set CONNECTABLE?
52. Can you join?
53. 11201 $95,000 10038 $35,000 11456 $65,000 Census Income Data
54. Noise 10014 Rats 11218 No heat 11201 3-1-1 Calls
55. 11201 $95,000 10038 $35,000 11456 $65,000 Noise 10014 Rats 11218 No heat 11201 Census Income Data 3-1-1 Calls
56. 11201 $95,000 10038 $35,000 11456 $65,000 Noise 10014 Rats 11218 No heat 11201 Census Income Data 3-1-1 Calls
57. 11201 $95,000 No heat 11219 $75,000 Bagel ! theft 10095 $35,000 Pigeon attack Joined table
59. ACTUALLY DO THE JOIN
60. Can you join?
61. UH OH: no joinable IDs
62. UH OH: no unique IDs at all
63. Is your data set CONNECTABLE?
64. complete?
    Is your data set correct?
    connectable?
65. What’s next?
67. THANKS more at sasha.io