On-Calliday: Un-sucking Your On-Call Experience

Being on call sucks, but it doesn’t have to. Many organizations struggle to maintain effective on-call rotations that minimize the number of people woken up in the middle of the night -- so much so, that “burnout syndrome” is an actual disorder that’s commonly experienced by professionals in on-call positions. In this talk, we’ll talk about how you can set up your on-call rotations to optimize the uptime of your infrastructure and your engineering team.

Transcript

1. On-Calliday: Un-sucking Your On-Call Experience Amanda Folson Developer Advocate @ GitLab
2. Amanda Folson Developer Advocate @ GitLab amanda@gitlab.com
3. Who’s on call?
4. Why does being on call suck? • No sleep • Too many distractions • Too little time • Tethered to your phone/laptop • Exhaustion
5. It Can Be Better!
6. Burnout Caution Needing to prove yourself, feeling you don’t provide value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
7. Burnout Caution Needing to prove yourself, feeling you don’t provide value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
8. Burnout Caution Needing to prove yourself, feeling you don’t provide value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
9. Burnout Caution Needing to prove yourself, feeling you don’t provide value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
10. Burnout Caution Needing to prove yourself, feeling you don’t provide value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
11. Burnout Happens in Tech • Caused by high-stress environments • Caused by not having basic needs met ◦ NEVER underestimate the value of sleep ◦ Water is also pretty important • Caused by emphasis on performance ◦ Fear of underperforming ◦ Not taking vacation • Sedentary lifestyle (often found in tech!) can exacerbate the problem
13. On Call Burnout
14. Preparation is Key • Keep your laptop charged • Keep your MiFi charged • Keep a spare battery pack in your backpack • Toss an energy drink and a small snack in your bag • Bring cables you may need
15. Reduce Stress • Pay important bills • Run errands • Set yourself up for an easier work week if possible ◦ Do the hard stuff now instead of later ◦ Don’t put off the boring/hard tasks until the last minute
16. Forgetting You’re On Shift • It happens more than you think. • RELAX! • If your bag is packed and you have cell reception, you’re probably okay. • If reception = false, get to a place that has it
17. How do we make life easier in-house?
18. Step One: Set better schedules!
19. Starting/Ending Rotations • Start and end rotations during the day • Midweek • NEVER start/end on weekends • DO NOT START ON A HOLIDAY ◦ Avoid scheduling people on holidays at all if possible ◦ Share the load
20. VACATION
21. Step Two: Select the right shift length
22. Common Shift Lengths • 8 hours • 12 hours • 24 hours • 168 hours/1 week
23. 8-hour Shifts
24. 12-hour Shifts
26. 24-hour Shifts
27. 168-hour Shifts/1 Week Shifts
28. What rotation should I use?
29. Week+Weekends
30. Scheduling Overrides “No battle plan survives first contact with the enemy.” Roughly translated from the thesis of Helmuth von Moltke the Elder
31. Step Three: Don’t Make the Pain in Vain
32. Make Your Data Count!
33. Tooling Matters!
34. Step Four: Keep Staff Sharp
35. Incident Response http://www.blackrock3.com/
36. You write it, you wear it.
37. Thank You! Amanda Folson amanda@gitlab.com @AmbassadorAwsum