Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Technology
5
800
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
790
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
820
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.5k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
780
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
890
Docker in a Flash
alicegoldfuss
2
410
Other Decks in Technology
See All in Technology
複数のスクラムチームをサポートするエンジニアリングマネジメントの話
okeicalm
0
1.1k
今どきのLinux事情
tokida
40
33k
JUnit5.7, 5.8の新機能紹介 #jjug_ccc #jjug_ccc_b / junit 5.7, 5.8 new features
kyonmm
PRO
2
420
MRTK3 - DataBinding and Theming 入門
futo23
0
190
【SAP知らない人向け】SAP on AWS 個人学習メモ/sap-on-aws-study
emiki
3
1.5k
Swift Regex Builder
kumamotone
1
100
DOM Invader - prototype pollution対応の衝撃 - / DOM Invader - prototype pollution
okuken
0
150
データをモデリングしていたら、組織をモデリングし始めた話 / engineers-in-carta-vol3-data-engineer
pei0804
4
3.3k
FoodTechにおける商流・金流・物流の進化/Evolution of Commercial, Financial, and Logistics in FoodTech
dskst
0
400
oakのミドルウェアを書くときの技のらしきもの
toranoana
0
130
RDRA + JavaによるレジャーSaaSプロダクトの要件定義と実装のシームレスな接続
jjebejj
PRO
3
700
SlackBotで あらゆる業務を自動化。問い合わせ〜DevOpsまで #CODT2022
kogatakanori
0
880
Featured
See All Featured
Done Done
chrislema
174
14k
How To Stay Up To Date on Web Technology
chriscoyier
780
250k
Agile that works and the tools we love
rasmusluckow
319
19k
How to Ace a Technical Interview
jacobian
265
21k
How GitHub (no longer) Works
holman
296
140k
Facilitating Awesome Meetings
lara
29
4k
In The Pink: A Labor of Love
frogandcode
131
21k
GraphQLとの向き合い方2022年版
quramy
16
8.3k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
29
4.3k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
4
510
The World Runs on Bad Software
bkeepers
PRO
57
5.3k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
100
5.9k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None