Monitorama EU 2013

Transcript

1. Monitoring the Obama Campaign
2. me: Director of Engineering, OFA typical political operative...
3. what I’m gonna talk about: •campaign context •how we monitored •what I wish we had
4. Crate&Barrel
5. threadless
7. rebuild
10. while it’s in heavy use
11. 18 months
12. 7 days a week
13. 18 hours a day
14. ~40 engineers
15. 7 teams
16. ~300 repos
17. ~200 deployed products
18. ~3,000 servers
19. some million views a day
20. 1,000,000+ volunteers
21. ~8,000 staff
22. 1 metric that matters
24. campaigns in 3 minutes
25. a campaign: •plan •build •execute •GOTV
26. PLAN!
27. dreams
28. not fast
29. for tech: start building
30. BUILD!
31. staff/teams
32. volunteer base
33. $
34. technology
35. for tech: keep building, make money
36. EXECUTE!
37. talk to voters
38. get volunteers
39. do ALL the things!
40. for tech: keep building, get feedback!
41. GOTV!
42. get out the vote
43. this is it
44. for tech:
45. scale.
46. scale!
47. SCALE!
50. also...
52. so.
53. so many apps
54. so many eyes
55. hiccup in our apps => press, legal, election?
56. keep the hiccups to a minimum
57. 2 ways
58. more monitoring (to keep the hiccups small)
59. less things to fail
60. architectural simplicity
61. if you make things that look like this:
64. out of small, well understood pieces, you’re going to have an easier time
65. make the easy things easy
66. monitoring what and how?
67. monitor ALL the things
68. we used some tools:
69. loggly
70. graylog
71. nagios
72. statsd
73. pingdom
74. cloudwatch
75. seyren
76. graphite
77. newrelic
78. google analytics
79. chartbeat
80. cloudability
81. pagerduty
82. ...more
83. we were indiscriminate about the tools that we used to monitor.
84. why?
85. [story time]
86. monitoring is only worth it if people are... monitoring.
87. alerts & dashboards!
88. alerting effectively is hard
89. my inbox was a terrible place
91. alerting needs appropriate severity
92. YELLING!
93. phone calls
94. calls from pagerduty are nice
95. calls from your boss at 2am are less nice, but more powerful
96. <secret sauce>people</secret sauce>
97. your power users often know LONG before you do.
98. Craig
99. there are friendly people already monitoring your site
100. they find things that can be hard to monitor
101. they know how to make it your priority
102. they only monitor downstream
103. you need to know what that means upstream.
104. look at ALL your monitors when you get the downstream alert
105. drill down into base level monitors.
106. recognize problems by their side affects!
107. make custom dashboards
108. dreams
109. 1. monitor people
110. ಠ_ಠ [insert NSA joke]
111. my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
112. how can I monitor the health of my team like we monitored the health of the systems?
113. aggregate fitbit dashboard?
114. simple feedback input while tests are running/things are building?
115.          >  how  are  you?          >
116. sentiment analysis of commit messages?
117. 2. automate power users
118. what're they doing?
119. looking at the site and seeing things change.
120. it's not super sophisticated
121. what are people who are smarter than me doing?
122. Brett Slatkin (smarter than me)
123. dpxdt p-diffs for release testing
124. why not p-diffs for monitoring changes?
126. Summary:
127. monitor the things that matter any way you can
128. learn what downstream looks like
129. listen to your people, they know what's up
130. be careful about crying wolf with your alerts.
131. dreams:
132. why aren't we watching ourselves?
133. how do we enable that? [hint: I don't know.]
134. automate your power users
135. maybe p-diffs?
136. thanks!
137. Dylan Richard t: @dylanr e: me@dylanr.com