Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Martyrs On Film: learning to hate the #oncallselfie
Search
Alice Goldfuss
May 22, 2017
Technology
5
1k
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Tweet
Share
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
960
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
1k
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.6k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
880
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
980
Docker in a Flash
alicegoldfuss
2
450
Other Decks in Technology
See All in Technology
楽しくGoを学び合う、LayerXの勉強会文化 / LayerX's study culture of having fun and learning Go together
ar_tama
2
350
LLMアプリケーションの評価の実践と課題 ~PharmaXにおける今後の展望~
pharma_x_tech
2
160
簡単に始めるSnowflakeの機械学習
nayuts
1
190
目標設定は好きですか? アジャイルとともに目標と向き合い続ける方法 / Do you like target Management?
kakehashi
10
3k
Scaling Technical Excellence at 104: Evolution in AWS and Developer Empowerment
scotthsieh825
1
150
[NIKKEI Tech Talk] KDDI/KAG Scrum & Community for Engineering Training
curanosuke
2
220
Docker互換のセキュアなコンテナ実行環境「Podman」超入門
devops_vtj
6
3.2k
AWSで”最小権限の原則”を実現するための考え方 /20240722-ssmjp-aws-least-privilege
opelab
10
4.3k
AIアシスタントの活用で品質の向上と開発ワークフローのスピードアップ
nagix
1
200
サービスの持続的な成長と技術負債について
siva_official
PRO
10
4.4k
年間一億円削減した時系列データベースのアーキテクチャ改善~不確実性の高いプロジェクトへの挑戦~
lycorptech_jp
PRO
3
2.9k
コンテナ・K8s研修 - 前半 コンテナ基礎・ハンズオン【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
170
Featured
See All Featured
Visualization
eitanlees
139
14k
Gamification - CAS2011
davidbonilla
78
4.9k
5 minutes of I Can Smell Your CMS
philhawksworth
200
19k
A designer walks into a library…
pauljervisheath
201
24k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
245
1.2M
We Have a Design System, Now What?
morganepeng
46
7k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.4k
It's Worth the Effort
3n
181
27k
Designing with Data
zakiwarfel
96
5k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
34
1.9k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Transcript
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None