Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Keynote - Dylan Richard
Search
Monitorama
September 19, 2013
1
260
Berlin 2013 - Keynote - Dylan Richard
Monitorama
September 19, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
560
PDX 2017 - Pedro Andrade
monitorama
0
670
PDX 2017 - Roy Rapoport
monitorama
4
900
PDX 2017 - Julia Evans
monitorama
0
420
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
660
Berlin 2013 - Session - Alex Petrov
monitorama
6
640
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
580
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
500
Berlin 2013 - Session - David Goodlad
monitorama
0
390
Featured
See All Featured
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Testing 201, or: Great Expectations
jmmastey
40
7.1k
Producing Creativity
orderedlist
PRO
341
39k
Into the Great Unknown - MozCon
thekraken
33
1.5k
Code Review Best Practice
trishagee
65
17k
Code Reviewing Like a Champion
maltzj
520
39k
Writing Fast Ruby
sferik
628
61k
How GitHub (no longer) Works
holman
311
140k
A better future with KSS
kneath
238
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Designing Experiences People Love
moore
138
23k
Transcript
Monitoring the Obama Campaign
me: Director of Engineering, OFA typical political operative...
what I’m gonna talk about: •campaign context •how we monitored
•what I wish we had
Crate&Barrel
threadless
None
rebuild
None
None
while it’s in heavy use
18 months
7 days a week
18 hours a day
~40 engineers
7 teams
~300 repos
~200 deployed products
~3,000 servers
some million views a day
1,000,000+ volunteers
~8,000 staff
1 metric that matters
270
campaigns in 3 minutes
a campaign: •plan •build •execute •GOTV
PLAN!
dreams
not fast
for tech: start building
BUILD!
staff/teams
volunteer base
$
technology
for tech: keep building, make money
EXECUTE!
talk to voters
get volunteers
do ALL the things!
for tech: keep building, get feedback!
GOTV!
get out the vote
this is it
for tech:
scale.
scale!
SCALE!
None
None
also...
None
so.
so many apps
so many eyes
hiccup in our apps => press, legal, election?
keep the hiccups to a minimum
2 ways
more monitoring (to keep the hiccups small)
less things to fail
architectural simplicity
if you make things that look like this:
None
None
out of small, well understood pieces, you’re going to have
an easier time
make the easy things easy
monitoring what and how?
monitor ALL the things
we used some tools:
loggly
graylog
nagios
statsd
pingdom
cloudwatch
seyren
graphite
newrelic
google analytics
chartbeat
cloudability
pagerduty
...more
we were indiscriminate about the tools that we used to
monitor.
why?
[story time]
monitoring is only worth it if people are... monitoring.
alerts & dashboards!
alerting effectively is hard
my inbox was a terrible place
None
alerting needs appropriate severity
YELLING!
phone calls
calls from pagerduty are nice
calls from your boss at 2am are less nice, but
more powerful
<secret sauce>people</secret sauce>
your power users often know LONG before you do.
Craig
there are friendly people already monitoring your site
they find things that can be hard to monitor
they know how to make it your priority
they only monitor downstream
you need to know what that means upstream.
look at ALL your monitors when you get the downstream
alert
drill down into base level monitors.
recognize problems by their side affects!
make custom dashboards
dreams
1. monitor people
ಠ_ಠ [insert NSA joke]
my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
how can I monitor the health of my team like
we monitored the health of the systems?
aggregate fitbit dashboard?
simple feedback input while tests are running/things are building?
> how are you?
>
sentiment analysis of commit messages?
2. automate power users
what're they doing?
looking at the site and seeing things change.
it's not super sophisticated
what are people who are smarter than me doing?
Brett Slatkin (smarter than me)
dpxdt p-diffs for release testing
why not p-diffs for monitoring changes?
None
Summary:
monitor the things that matter any way you can
learn what downstream looks like
listen to your people, they know what's up
be careful about crying wolf with your alerts.
dreams:
why aren't we watching ourselves?
how do we enable that? [hint: I don't know.]
automate your power users
maybe p-diffs?
thanks!
Dylan Richard t: @dylanr e:
[email protected]