Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Martyrs On Film: learning to hate the #oncallselfie
Search
Alice Goldfuss
May 22, 2017
Technology
5
970
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
930
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
970
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.6k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
850
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
940
Docker in a Flash
alicegoldfuss
2
440
Other Decks in Technology
See All in Technology
BPStudyの200回を中心にIT業界を振り返る。そしてこれから
haru860
3
430
M5stackで使用できるpHセンサの開発
shinrinakamura
1
290
How to Lead? Testimonial of a Lead Android Engineer
oleur
1
120
Cypress or Playwright?
rainerhahnekamp
0
180
CockroachDB はどのくらい「しぶとい」のか? / How tough is CockroachDB?
kota2and3kan
6
3.9k
止まらないLinuxシステムを構築する_高信頼性クラスタ入門
koedoyoshida
3
2.4k
本当のガバクラ基礎
toru_kubota
0
210
リテール金融(キャッシュレス・ネット銀行・ネット証券)の競争環境と経済圏
8maki
0
1.8k
個人的、Kubernetes の最新注目機能! (2024年5月版) / TechFeed Experts Night#28 〜 コンテナ技術最前線
pfn
PRO
1
120
令和最新版 Ruby プロファイラ "Pf2" のご紹介
osyoyu
0
170
実例で紹介するRAG導入時の知見と精度向上の勘所
yamahiro
7
2.3k
「知的単純作業」を自動化する、地に足の着いた大規模言語モデル (LLM) の活用
nrryuya
8
6.3k
Featured
See All Featured
Designing the Hi-DPI Web
ddemaree
276
33k
How GitHub (no longer) Works
holman
305
140k
Atom: Resistance is Futile
akmur
260
25k
How STYLIGHT went responsive
nonsquared
92
4.8k
How to Ace a Technical Interview
jacobian
273
22k
Being A Developer After 40
akosma
67
580k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
123
39k
Building Applications with DynamoDB
mza
88
5.6k
Faster Mobile Websites
deanohume
300
30k
Fireside Chat
paigeccino
22
2.6k
Principles of Awesome APIs and How to Build Them.
keavy
121
16k
Creatively Recalculating Your Daily Design Routine
revolveconf
211
11k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None