Luck Driven Development: Building for Serendipity in Slack's Data Platform

1. Luck Driven Development Building for Serendipity in Slack's Data Platform Ronnie Chen @rondo!w Data Engineering Slack CrunchConf 2017 | Ronnie Chen @rondo!w 1
2. First, a confession CrunchConf 2017 | Ronnie Chen @rondo!w 2
3. Mar 2016 5 data engs / 350+ employees / 2M+ DAU Oct 2017 10 data engs / 800+ employees / 6M+ DAU 1 in 9 employees access the data warehouse / day 1 in 3 employees access the data warehouse / week CrunchConf 2017 | Ronnie Chen @rondo!w 3
4. CrunchConf 2017 | Ronnie Chen @rondo!w 4
5. Your Lucky Day — Recognizing and fostering luck — Taking advantage of luck — Building resiliency to bad luck — Saving your luck for when you need it — Slack data's lucky break CrunchConf 2017 | Ronnie Chen @rondo!w 5
6. ! Recognizing and Fostering Luck CrunchConf 2017 | Ronnie Chen @rondo!w 6
7. Luck is what happens when preparation meets opportunity — [Not] Seneca CrunchConf 2017 | Ronnie Chen @rondo!w 7
8. If one does not know to which port one is sailing, no wind is favorable. — Seneca, Letter LXXI CrunchConf 2017 | Ronnie Chen @rondo!w 8
9. Our plans miscarry because we have no aim CrunchConf 2017 | Ronnie Chen @rondo!w 9
10. CrunchConf 2017 | Ronnie Chen @rondo!w 10
11. Setting goals: separate the What from the How CrunchConf 2017 | Ronnie Chen @rondo!w 11
12. ❌ Build a new service for running large queries using [XYZ technology] CrunchConf 2017 | Ronnie Chen @rondo!w 12
13. ✅ Make querying the data warehouse reliable & performant even for large data sets ✅ Supports queries with billions of rows ✅ 95% of typical queries run in under 20 seconds ✅ Data shows up within 5 min of being written CrunchConf 2017 | Ronnie Chen @rondo!w 13
14. Maintain contex by focusing on what you need CrunchConf 2017 | Ronnie Chen @rondo!w 14
15. CrunchConf 2017 | Ronnie Chen @rondo!w 15
16. ! Taking advantage of luck when it appears CrunchConf 2017 | Ronnie Chen @rondo!w 16
17. Adaptability via incremental development CrunchConf 2017 | Ronnie Chen @rondo!w 17
18. If I had asked people what they wanted, they would have said faster horses — [Not] Henry Ford CrunchConf 2017 | Ronnie Chen @rondo!w 18
19. CrunchConf 2017 | Ronnie Chen @rondo!w 19
20. ! – earliest testable ! – earliest usable ! – earliest lovable CrunchConf 2017 | Ronnie Chen @rondo!w 20
21. Pay No Attention to the Service Behind the Curtain CrunchConf 2017 | Ronnie Chen @rondo!w 21
22. ! Building resiliency to bad luck CrunchConf 2017 | Ronnie Chen @rondo!w 22
23. You get love for it. You get hate for it. You get nothing if you wait for it, wait for it. — Alexander Hamilton (via Lin Manuel Miranda) CrunchConf 2017 | Ronnie Chen @rondo!w 23
24. CrunchConf 2017 | Ronnie Chen @rondo!w 24
25. CrunchConf 2017 | Ronnie Chen @rondo!w 25
26. ! Saving your luck for when you need it CrunchConf 2017 | Ronnie Chen @rondo!w 26
27. Testing on production CrunchConf 2017 | Ronnie Chen @rondo!w 27
29. Dark reads CrunchConf 2017 | Ronnie Chen @rondo!w 29
30. CrunchConf 2017 | Ronnie Chen @rondo!w 30
31. ! Slack data's lucky break CrunchConf 2017 | Ronnie Chen @rondo!w 31
32. php CrunchConf 2017 | Ronnie Chen @rondo!w 32
33. – make a query – create a graph – build a dashboard with selectors CrunchConf 2017 | Ronnie Chen @rondo!w 33
34. presto visualizations CrunchConf 2017 | Ronnie Chen @rondo!w 34
35. CrunchConf 2017 | Ronnie Chen @rondo!w 35
36. Me and My Shadow function presto_query($query, $opts=[]){ if (is_select($query) && !$opts['is_shadow']){ presto_query($query, $opts + ['is_shadow' => true]); } ... } CrunchConf 2017 | Ronnie Chen @rondo!w 36
37. Data for the Data God data engineers queries success rate is shadow 3623 0.99558 false 3083 0.96497 true CrunchConf 2017 | Ronnie Chen @rondo!w 37
38. CrunchConf 2017 | Ronnie Chen @rondo!w 38
39. But wait! There's More CrunchConf 2017 | Ronnie Chen @rondo!w 39
40. CrunchConf 2017 | Ronnie Chen @rondo!w 40
41. CrunchConf 2017 | Ronnie Chen @rondo!w 41
42. CrunchConf 2017 | Ronnie Chen @rondo!w 42
43. CrunchConf 2017 | Ronnie Chen @rondo!w 43
44. ! Luck is real. The effects of luck are undeniable. CrunchConf 2017 | Ronnie Chen @rondo!w 44
45. optimizing outcomes and managing uncertainty in stochastic systems CrunchConf 2017 | Ronnie Chen @rondo!w 45
46. Good luck is a terrible thing to waste CrunchConf 2017 | Ronnie Chen @rondo!w 46
47. CrunchConf 2017 | Ronnie Chen @rondo!w 47
48. Any Questions? CrunchConf 2017 | Ronnie Chen @rondo!w 48