Martyrs On Film: learning to hate the #oncallselfie

Transcript

1. 1.

   Martyrs on Film

2. 2.

   Hi! I’m Alice I like systems and Twitter and tea.

3. 3.

   Hi! I’m Alice And not getting paged.

4. 4.

   None
5. 5.

   None
6. 6.

   Benefits of On-Call • Hones troubleshooting skills • Forces you

   to identify the weak points in your systems • Teaches you what is and isn’t production-ready
7. 7.

   Team bonding

8. 8.

   None
9. 9.

   None
10. 10.

    None
11. 11.

    None
12. 12.

    None
13. 13.

    None
14. 14.

    None
15. 15.

    PagerDuty

16. 16.

    None
17. 17.

    None
18. 18.

    PagerDuty

19. 19.

    PagerDuty New Year’s Eve

20. 20.

    None
21. 21.

    None
22. 22.

    None
23. 23.

    PagerDuty New Year’s Eve

24. 24.

    PagerDuty New Year’s Eve S3 Outage

25. 25.

    None
26. 26.

    None
27. 27.

    None
28. 28.

    None
29. 29.

    Me

30. 30.

    None
31. 31.

    None
32. 32.

    None
33. 33.

    None
34. 34.

    A totally normal on-call routine • Don’t leave house except

    to commute to work • Clear all non-work appointments • Cook all meals beforehand • Have soup on hand • Don’t sleep
35. 35.

    None
36. 36.

    None
37. 37.

    None
38. 38.

    None
39. 39.

    Be heroes

40. 40.

    Prepare for battle

41. 41.

    Naomi Orwin Writer

42. 42.

    “Action scenes stop the plot.” - Naomi Orwin

43. 43.

    Pages stop the plot of your career

44. 44.

    None
45. 45.

    Miserable on-call professionals • Have terrible work/life balance • Are

    supporting poorly-designed systems • Feel powerless to solve problems • Generally hate the role
46. 46.

    None
47. 47.

    Red flag: too few owning too much

48. 48.

    Centralia infrastructure

49. 49.

    None
50. 50.

    None
51. 51.

    None
52. 52.

    Red flag: bandaids

53. 53.

    None
54. 54.

    • Bump thresholds • Snooze pages • Delays

55. 55.

    Red flag: no visibility

56. 56.

    Systems visibility

57. 57.

    Team visibility

58. 58.

    Too many pages

59. 59.

    Average # of weekly pages during WORST on-call

60. 60.

    Average # of weekly pages during WORST on-call

61. 61.

    Average # of weekly pages during WORST on-call

62. 62.

    Average # of weekly pages during WORST on-call

63. 63.

    None
64. 64.

    Average # of weekly pages during BEST on-call

65. 65.

    Average # of weekly pages during BEST on-call

66. 66.

    Average # of weekly pages during BEST on-call

67. 67.

    How do we get there?

68. 68.

    Notification cleanup

69. 69.

    Actionable alerts

70. 70.

    Actionable Alerts • Something breaks • Customers notice • I

    am the best person to fix it • I need to fix it immediately
71. 71.

    Cluster alerts

72. 72.

    None
73. 73.

    Devs on-call

74. 74.

    None
75. 75.

    “If a developer is good, being ‘on call’ just means

    having to fix other people’s problems and inconsequential stuff on Sat.” - dev on Twitter
76. 76.

    “Fixed this for him! ‘Put your developers on call. You’ll

    be surprised by how quickly they go work for someone that isn’t an #$%.’” - dev on Twitter
77. 77.

    “If your org has change control working properly, if code

    breaks, the jr sysadmin should simply roll back the update as documented.” - dev on Twitter
78. 78.

    The right tool

79. 79.

    Work together

80. 80.

    None
81. 81.

    Start small

82. 82.

    None
83. 83.

    Your people will burn out before your company does

84. 84.

    None
85. 85.

    None
86. 86.

    None
87. 87.

    Where does that leave #oncallselfie?

88. 88.

    None
89. 89.

    None
90. 90.

    None
91. 91.

    None
92. 92.

    None
93. 93.

    None
94. 94.

    None
95. 95.

    None
96. 96.

    None
97. 97.

    None
98. 98.

    None
99. 99.

    “Why are you getting paged so much?”

100. 100.

     Thanks! @alicegoldfuss Special Thanks: PagerDuty VictorOps oncallselfies.com All of you

101. 101.

     None