Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Martyrs On Film: learning to hate the #oncallse...
Search
Alice Goldfuss
May 22, 2017
Technology
5
1.1k
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
1.1k
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
1.1k
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.7k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
1k
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
1.1k
Docker in a Flash
alicegoldfuss
2
510
Other Decks in Technology
See All in Technology
MUITにおける開発プロセスモダナイズの取り組みと開発生産性可視化の取り組みについて / Modernize the Development Process and Visualize Development Productivity at MUIT
muit
1
17k
Model Mondays S2E04: AI Developer Experiences
nitya
0
140
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
340
ビズリーチにおけるリアーキテクティング実践事例 / JJUG CCC 2025 Spring
visional_engineering_and_design
1
130
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
2
7.8k
Glacierだからってコストあきらめてない? / JAWS Meet Glacier Cost
taishin
1
160
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
7
5.3k
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
310
IPA&AWSダブル全冠が明かす、人生を変えた勉強法のすべて
iwamot
PRO
2
150
第4回Snowflake 金融ユーザー会 Snowflake summit recap
tamaoki
1
290
Yahoo!しごとカタログ 新しい境地を創るエンジニア募集!
lycorptech_jp
PRO
0
110
DBのスキルで生き残る技術 - AI時代におけるテーブル設計の勘所
soudai
PRO
50
20k
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
For a Future-Friendly Web
brad_frost
179
9.8k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
BBQ
matthewcrist
89
9.7k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
The Pragmatic Product Professional
lauravandoore
35
6.7k
GraphQLとの向き合い方2022年版
quramy
49
14k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
820
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Writing Fast Ruby
sferik
628
62k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None