Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Berlin 2013 - Keynote - Dylan Richard
Search
Monitorama
September 19, 2013
1
250
Berlin 2013 - Keynote - Dylan Richard
Monitorama
September 19, 2013
Tweet
Share
More Decks by Monitorama
See All by Monitorama
Monitorama PDX 2017 - Ian Bennett
monitorama
1
560
PDX 2017 - Pedro Andrade
monitorama
0
660
PDX 2017 - Roy Rapoport
monitorama
4
890
PDX 2017 - Julia Evans
monitorama
0
420
Berlin 2013 - Session - Brad Lhotsky
monitorama
5
660
Berlin 2013 - Session - Alex Petrov
monitorama
6
640
Berlin 2013 - Session - Jeff Weinstein
monitorama
2
570
Berlin 2013 - Session - Oliver Hankeln
monitorama
1
490
Berlin 2013 - Session - David Goodlad
monitorama
0
380
Featured
See All Featured
A designer walks into a library…
pauljervisheath
203
24k
How To Stay Up To Date on Web Technology
chriscoyier
788
250k
Keith and Marios Guide to Fast Websites
keithpitt
409
22k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Measuring & Analyzing Core Web Vitals
bluesmoon
4
120
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Statistics for Hackers
jakevdp
796
220k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
370
Side Projects
sachag
452
42k
Making Projects Easy
brettharned
115
5.9k
Transcript
Monitoring the Obama Campaign
me: Director of Engineering, OFA typical political operative...
what I’m gonna talk about: •campaign context •how we monitored
•what I wish we had
Crate&Barrel
threadless
None
rebuild
None
None
while it’s in heavy use
18 months
7 days a week
18 hours a day
~40 engineers
7 teams
~300 repos
~200 deployed products
~3,000 servers
some million views a day
1,000,000+ volunteers
~8,000 staff
1 metric that matters
270
campaigns in 3 minutes
a campaign: •plan •build •execute •GOTV
PLAN!
dreams
not fast
for tech: start building
BUILD!
staff/teams
volunteer base
$
technology
for tech: keep building, make money
EXECUTE!
talk to voters
get volunteers
do ALL the things!
for tech: keep building, get feedback!
GOTV!
get out the vote
this is it
for tech:
scale.
scale!
SCALE!
None
None
also...
None
so.
so many apps
so many eyes
hiccup in our apps => press, legal, election?
keep the hiccups to a minimum
2 ways
more monitoring (to keep the hiccups small)
less things to fail
architectural simplicity
if you make things that look like this:
None
None
out of small, well understood pieces, you’re going to have
an easier time
make the easy things easy
monitoring what and how?
monitor ALL the things
we used some tools:
loggly
graylog
nagios
statsd
pingdom
cloudwatch
seyren
graphite
newrelic
google analytics
chartbeat
cloudability
pagerduty
...more
we were indiscriminate about the tools that we used to
monitor.
why?
[story time]
monitoring is only worth it if people are... monitoring.
alerts & dashboards!
alerting effectively is hard
my inbox was a terrible place
None
alerting needs appropriate severity
YELLING!
phone calls
calls from pagerduty are nice
calls from your boss at 2am are less nice, but
more powerful
<secret sauce>people</secret sauce>
your power users often know LONG before you do.
Craig
there are friendly people already monitoring your site
they find things that can be hard to monitor
they know how to make it your priority
they only monitor downstream
you need to know what that means upstream.
look at ALL your monitors when you get the downstream
alert
drill down into base level monitors.
recognize problems by their side affects!
make custom dashboards
dreams
1. monitor people
ಠ_ಠ [insert NSA joke]
my team was overworked, frustrated, unhealthy, tired, cranky, underfed, etc.
how can I monitor the health of my team like
we monitored the health of the systems?
aggregate fitbit dashboard?
simple feedback input while tests are running/things are building?
> how are you?
>
sentiment analysis of commit messages?
2. automate power users
what're they doing?
looking at the site and seeing things change.
it's not super sophisticated
what are people who are smarter than me doing?
Brett Slatkin (smarter than me)
dpxdt p-diffs for release testing
why not p-diffs for monitoring changes?
None
Summary:
monitor the things that matter any way you can
learn what downstream looks like
listen to your people, they know what's up
be careful about crying wolf with your alerts.
dreams:
why aren't we watching ourselves?
how do we enable that? [hint: I don't know.]
automate your power users
maybe p-diffs?
thanks!
Dylan Richard t: @dylanr e:
[email protected]