Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Martyrs On Film: learning to hate the #oncallse...

Avatar for Alice Goldfuss Alice Goldfuss
May 22, 2017
Technology
5
1.1k

Martyrs On Film: learning to hate the #oncallselfie

Avatar for Alice Goldfuss

Alice Goldfuss

May 22, 2017
Tweet

More Decks by Alice Goldfuss

See All by Alice Goldfuss
The Container Operator’s Manual
Avatar for Alice Goldfuss alicegoldfuss
6
1.2k
Passing the Console: Fostering the Next Generation of Ops Professionals
Avatar for Alice Goldfuss alicegoldfuss
0
1.1k
Rockstars, Builders, and Janitors: You're doing it wrong
Avatar for Alice Goldfuss alicegoldfuss
11
1.7k
nrrd 911 ic me: The Incident Commander Role
Avatar for Alice Goldfuss alicegoldfuss
3
1k
Scalable Meatfrastructure: Building Stable DevOps Teams
Avatar for Alice Goldfuss alicegoldfuss
2
1.2k
Docker in a Flash
Avatar for Alice Goldfuss alicegoldfuss
2
530

Other Decks in Technology

See All in Technology
ソースを読む時の思考プロセスの例-MkDocs
Avatar for Satoru Takeuchi sat
PRO
1
320
AIエージェントによる業務効率化への飽くなき挑戦 -AWS上の実開発事例から学んだ効果、現実そしてギャップ-
Avatar for Kiminori Yokoi nasuvitz
5
1.4k
serverless team topology
Avatar for kensh _kensh
3
240
Oracle Database@Google Cloud：サービス概要のご紹介
Avatar for oracle4engineer oracle4engineer
PRO
0
390
知覚とデザイン
Avatar for Rinchoku rinchoku
1
630
プロダクト開発と社内データ活用での、BI×AIの現在地 / Data_Findy
Avatar for Sansan R&D sansan_randd
1
630
CREが作る自己解決サイクルSlackワークフローに組み込んだAIによる社内ヘルプデスク改革 #cre_meetup
Avatar for 弁護士ドットコム bengo4com
0
370
abema-trace-sampling-observability-cost-optimization
Avatar for tetsuya28 tetsuya28
0
360
個人でデジタル庁の デザインシステムをVue.jsで 作っている話
Avatar for にしはら nishiharatsubasa
3
5.2k
実践マルチモーダル検索！
Avatar for shibuiwilliam shibuiwilliam
1
380
スタートアップの現場で実践しているテストマネジメント #jasst_kyushu
Avatar for Makky makky_tyuyan
0
140
仕様駆動開発を実現する上流工程におけるAIエージェント活用
Avatar for sergicalsix sergicalsix
8
4k

Featured

See All Featured
Done Done
Avatar for chrislema chrislema
185
16k
The Pragmatic Product Professional
Avatar for Laura Van Doore lauravandoore
36
7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
34
2.3k
Put a Button on it: Removing Barriers to Going Fast.
Avatar for kastner kastner
60
4k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
Avatar for panda_program panda_program
116
20k
ピンチをチャンスに：未来をつくるプロダクトロードマップ #pmconf2020
Avatar for Aki / @LoveIdahoBurger aki_iinuma
127
54k
How STYLIGHT went responsive
Avatar for nonsquared nonsquared
100
5.9k
Speed Design
Avatar for Sergey Chernyshev sergeychernyshev
32
1.2k
How to Ace a Technical Interview
Avatar for Jacob Kaplan-Moss jacobian
280
24k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
Avatar for Addy Osmani addyosmani
9
930
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
46
7.7k
GitHub's CSS Performance
Avatar for Jon Rohan jonrohan
1032
470k

Transcript

  1. Martyrs on Film

  2. Hi! I’m Alice I like systems and Twitter and tea.

  3. Hi! I’m Alice And not getting paged.

  4. None
  5. None

  6. Benefits of On-Call • Hones troubleshooting skills • Forces you

    to identify the weak points in your systems • Teaches you what is and isn’t production-ready

  7. Team bonding

  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None

  15. PagerDuty

  16. None
  17. None

  18. PagerDuty

  19. PagerDuty New Year’s Eve

  20. None
  21. None
  22. None

  23. PagerDuty New Year’s Eve

  24. PagerDuty New Year’s Eve S3 Outage

  25. None
  26. None
  27. None
  28. None

  29. Me

  30. None
  31. None
  32. None
  33. None

  34. A totally normal on-call routine • Don’t leave house except

    to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
  35. None
  36. None
  37. None
  38. None

  39. Be heroes

  40. Prepare for battle

  41. Naomi Orwin Writer

  42. “Action scenes stop the plot.” - Naomi Orwin

  43. Pages stop the plot of your career

  44. None

  45. Miserable on-call professionals • Have terrible work/life balance • Are

    supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
  46. None

  47. Red flag: too few owning too much

  48. Centralia infrastructure

  49. None
  50. None
  51. None

  52. Red flag: bandaids

  53. None

  54. • Bump thresholds • Snooze pages • Delays

  55. Red flag: no visibility

  56. Systems visibility

  57. Team visibility

  58. Too many pages

  59. Average # of weekly pages during WORST on-call

  60. Average # of weekly pages during WORST on-call

  61. Average # of weekly pages during WORST on-call

  62. Average # of weekly pages during WORST on-call

  63. None

  64. Average # of weekly pages during BEST on-call

  65. Average # of weekly pages during BEST on-call

  66. Average # of weekly pages during BEST on-call

  67. How do we get there?

  68. Notification cleanup

  69. Actionable alerts

  70. Actionable Alerts • Something breaks • Customers notice • I

    am the best person to fix it • I need to fix it immediately

  71. Cluster alerts

  72. None

  73. Devs on-call

  74. None

  75. “If a developer is good, being ‘on call’ just means

    having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter

  76. “Fixed this for him! ‘Put your developers on call. You’ll

    be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter

  77. “If your org has change control working properly, if code

    breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter

  78. The right tool

  79. Work together

  80. None

  81. Start small

  82. None

  83. Your people will burn out before your company does

  84. None
  85. None
  86. None

  87. Where does that leave #oncallselfie?

  88. None
  89. None
  90. None
  91. None
  92. None
  93. None
  94. None
  95. None
  96. None
  97. None
  98. None

  99. “Why are you getting paged so much?”

  100. Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you

  101. None