Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Keynote - Dylan Richard
Search
Monitorama
September 19, 2013
1
250
Berlin 2013 - Keynote - Dylan Richard
Monitorama
September 19, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
560
PDX 2017 - Pedro Andrade
monitorama
0
630
PDX 2017 - Roy Rapoport
monitorama
4
880
PDX 2017 - Julia Evans
monitorama
0
410
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
650
Berlin 2013 - Session - Alex Petrov
monitorama
6
630
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
570
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
490
Berlin 2013 - Session - David Goodlad
monitorama
0
380
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
503
140k
10 Git Anti Patterns You Should be Aware of
lemiorhan
653
59k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
158
15k
Six Lessons from altMBA
skipperchong
26
3.4k
Producing Creativity
orderedlist
PRO
340
39k
The Pragmatic Product Professional
lauravandoore
31
6.2k
Designing on Purpose - Digital PM Summit 2013
jponch
114
6.9k
Designing the Hi-DPI Web
ddemaree
279
34k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
Design by the Numbers
sachag
277
19k
What's new in Ruby 2.0
geeforr
340
31k
Optimising Largest Contentful Paint
csswizardry
31
2.8k
Transcript
Monitoring the Obama Campaign
me: Director of Engineering, OFA typical political operative...
what I’m gonna talk about: •campaign context •how we monitored
•what I wish we had
Crate&Barrel
threadless
None
rebuild
None
None
while it’s in heavy use
18 months
7 days a week
18 hours a day
~40 engineers
7 teams
~300 repos
~200 deployed products
~3,000 servers
some million views a day
1,000,000+ volunteers
~8,000 staff
1 metric that matters
270
campaigns in 3 minutes
a campaign: •plan •build •execute •GOTV
PLAN!
dreams
not fast
for tech: start building
BUILD!
staff/teams
volunteer base
$
technology
for tech: keep building, make money
EXECUTE!
talk to voters
get volunteers
do ALL the things!
for tech: keep building, get feedback!
GOTV!
get out the vote
this is it
for tech:
scale.
scale!
SCALE!
None
None
also...
None
so.
so many apps
so many eyes
hiccup in our apps => press, legal, election?
keep the hiccups to a minimum
2 ways
more monitoring (to keep the hiccups small)
less things to fail
architectural simplicity
if you make things that look like this:
None
None
out of small, well understood pieces, you’re going to have
an easier time
make the easy things easy
monitoring what and how?
monitor ALL the things
we used some tools:
loggly
graylog
nagios
statsd
pingdom
cloudwatch
seyren
graphite
newrelic
google analytics
chartbeat
cloudability
pagerduty
...more
we were indiscriminate about the tools that we used to
monitor.
why?
[story time]
monitoring is only worth it if people are... monitoring.
alerts & dashboards!
alerting effectively is hard
my inbox was a terrible place
None
alerting needs appropriate severity
YELLING!
phone calls
calls from pagerduty are nice
calls from your boss at 2am are less nice, but
more powerful
<secret sauce>people</secret sauce>
your power users often know LONG before you do.
Craig
there are friendly people already monitoring your site
they find things that can be hard to monitor
they know how to make it your priority
they only monitor downstream
you need to know what that means upstream.
look at ALL your monitors when you get the downstream
alert
drill down into base level monitors.
recognize problems by their side affects!
make custom dashboards
dreams
1. monitor people
ಠ_ಠ [insert NSA joke]
my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
how can I monitor the health of my team like
we monitored the health of the systems?
aggregate fitbit dashboard?
simple feedback input while tests are running/things are building?
> how are you?
>
sentiment analysis of commit messages?
2. automate power users
what're they doing?
looking at the site and seeing things change.
it's not super sophisticated
what are people who are smarter than me doing?
Brett Slatkin (smarter than me)
dpxdt p-diffs for release testing
why not p-diffs for monitoring changes?
None
Summary:
monitor the things that matter any way you can
learn what downstream looks like
listen to your people, they know what's up
be careful about crying wolf with your alerts.
dreams:
why aren't we watching ourselves?
how do we enable that? [hint: I don't know.]
automate your power users
maybe p-diffs?
thanks!
Dylan Richard t: @dylanr e:
[email protected]