Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Martyrs On Film: learning to hate the #oncallse...
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
Alice Goldfuss
May 22, 2017
Technology
1.2k
5
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Martyrs On Film: learning to hate the #oncallselfie
Alice Goldfuss
May 22, 2017
Transcript
101 slides
Martyrs on Film
Hi! I’m Alice I like systems and Twitter and tea.
Hi! I’m Alice And not getting paged.
None
None
Benefits of On-Call • Hones troubleshooting skills • Forces you
to identify the weak points in your systems • Teaches you what is and isn’t production-ready
Team bonding
None
None
None
None
None
None
None
PagerDuty
None
None
PagerDuty
PagerDuty New Year’s Eve
None
None
None
PagerDuty New Year’s Eve
PagerDuty New Year’s Eve S3 Outage
None
None
None
None
Me
None
None
None
None
A totally normal on-call routine • Don’t leave house except
to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
None
None
None
None
Be heroes
Prepare for battle
Naomi Orwin Writer
“Action scenes stop the plot.” - Naomi Orwin
Pages stop the plot of your career
None
Miserable on-call professionals • Have terrible work/life balance • Are
supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
None
Red flag: too few owning too much
Centralia infrastructure
None
None
None
Red flag: bandaids
None
• Bump thresholds • Snooze pages • Delays
Red flag: no visibility
Systems visibility
Team visibility
Too many pages
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
Average # of weekly pages during WORST on-call
None
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
Average # of weekly pages during BEST on-call
How do we get there?
Notification cleanup
Actionable alerts
Actionable Alerts • Something breaks • Customers notice • I
am the best person to fix it • I need to fix it immediately
Cluster alerts
None
Devs on-call
None
“If a developer is good, being ‘on call’ just means
having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
“Fixed this for him! ‘Put your developers on call. You’ll
be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
“If your org has change control working properly, if code
breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
The right tool
Work together
None
Start small
None
Your people will burn out before your company does
None
None
None
Where does that leave #oncallselfie?
None
None
None
None
None
None
None
None
None
None
None
“Why are you getting paged so much?”
Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you
None
More Decks by Alice Goldfuss
See All by Alice Goldfuss
The Container Operator’s Manual
alicegoldfuss
6
1.4k
Passing the Console: Fostering the Next Generation of Ops Professionals
alicegoldfuss
0
1.2k
Rockstars, Builders, and Janitors: You're doing it wrong
alicegoldfuss
11
1.9k
nrrd 911 ic me: The Incident Commander Role
alicegoldfuss
3
1.1k
Scalable Meatfrastructure: Building Stable DevOps Teams
alicegoldfuss
2
1.3k
Docker in a Flash
alicegoldfuss
2
590
Other Decks in Technology
See All in Technology
From vibe coding to reliable (agentic) development
jcabot
0
240
認知負荷の減らし方 ~開発にも日常業務にも活かせるノウハウ~
minorun365
PRO
17
10k
みん強Lightning! データ基盤ひとりでできるかな
endonanana
0
190
サーバーラックは作る時代! 3Dプリンターで作る10インチラックのススメ
jacopen
2
520
Amazon Bedrock AgentCore の基礎とセキュリティ・ガバナンス
ren8k
0
270
ポケモンバトルのAIエージェント
koji_kawamura
0
220
AWS Well-Arhitected Frameworkセキュリティの柱 について
miu_crescent
PRO
2
250
OCI Artifact による自動運転タクシーに向けた OTA 基盤の構築
biwashi
1
300
セキュリティチームは どのコンテナイメージの脆弱性対応を優先すべきか 〜CISA新指針とSCCで作る脆弱性トリアージ〜
swdtkuy
0
330
AI-Ready なデータ基盤ってなに? & 新卒未経験が、いま考えていること
imuta
0
1.1k
OpenTelemetryで変わった 新時代のCloudWatchメトリクス監視
tsukuboshi
0
500
MCPでネットワーク障害調査を自動化する前に直すべき3つのこと
yama3133
1
660
Featured
See All Featured
Marketing Yourself as an Engineer | Alaka | Gurzu
gurzu
0
390
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
56
3.5k
The Mindset for Success: Future Career Progression
greggifford
PRO
0
570
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
rkaga
63
48k
Optimising Largest Contentful Paint
csswizardry
37
4k
A Soul's Torment
seathinner
9
3.9k
Into the Great Unknown - MozCon
thekraken
41
2.8k
Git: the NoSQL Database
bkeepers
PRO
433
67k
The Anti-SEO Checklist Checklist. Pubcon Cyber Week
ryanjones
0
310
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
davidcarrasco
3
310
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
kimpetersen
PRO
0
550
DevOps and Value Stream Thinking: Enabling flow, efficiency and business value
helenjbeal
2
450