On-Calliday: Un-sucking Your On-Call Experience

Being on call sucks, but it doesn’t have to. Many organizations struggle to maintain effective on-call rotations that minimize the number of people woken up in the middle of the night -- so much so, that “burnout syndrome” is an actual disorder that’s commonly experienced by professionals in on-call positions. In this talk, we’ll talk about how you can set up your on-call rotations to optimize the uptime of your infrastructure and your engineering team.

Transcript

1. On-Calliday: Un-sucking
   Your On-Call Experience
   Amanda Folson
   Developer Advocate @ GitLab
   

   View Slide

2. Amanda Folson
   Developer Advocate @ GitLab
   amanda@gitlab.com
   

   View Slide

3. Who’s on
   call?
   

   View Slide

4. Why does being on call suck?
   ● No sleep
   ● Too many distractions
   ● Too little time
   ● Tethered to your phone/laptop
   ● Exhaustion
   

   View Slide

5. It Can Be
   Better!
   

   View Slide

6. Burnout
   Caution Needing to prove yourself, feeling you don’t
   provide value, working harder, difficult to
   switch off
   Warning Start to ignore needs like sleeping/eating,
   reduced social interactions, panic, loss of
   interest in family/hobbies
   Danger Behavioral changes, cynicism,
   drugs/alcohol, promiscuousness,
   aggression, worthlessness/hopelessness,
   on autopilot
   Emergency Inner emptiness, more risk-taking,
   depression, suicide
   

   View Slide

7. Burnout
   Caution Needing to prove yourself, feeling you don’t
   provide value, working harder, difficult to
   switch off
   Warning Start to ignore needs like sleeping/eating,
   reduced social interactions, panic, loss of
   interest in family/hobbies
   Danger Behavioral changes, cynicism,
   drugs/alcohol, promiscuousness,
   aggression, worthlessness/hopelessness,
   on autopilot
   Emergency Inner emptiness, more risk-taking,
   depression, suicide
   

   View Slide

8. Burnout
   Caution Needing to prove yourself, feeling you don’t
   provide value, working harder, difficult to
   switch off
   Warning Start to ignore needs like sleeping/eating,
   reduced social interactions, panic, loss of
   interest in family/hobbies
   Danger Behavioral changes, cynicism,
   drugs/alcohol, promiscuousness,
   aggression, worthlessness/hopelessness,
   on autopilot
   Emergency Inner emptiness, more risk-taking,
   depression, suicide
   

   View Slide

9. Burnout
   Caution Needing to prove yourself, feeling you don’t
   provide value, working harder, difficult to
   switch off
   Warning Start to ignore needs like sleeping/eating,
   reduced social interactions, panic, loss of
   interest in family/hobbies
   Danger Behavioral changes, cynicism,
   drugs/alcohol, promiscuousness,
   aggression, worthlessness/hopelessness,
   on autopilot
   Emergency Inner emptiness, more risk-taking,
   depression, suicide
   

   View Slide

10. Burnout
    Caution Needing to prove yourself, feeling you don’t
    provide value, working harder, difficult to
    switch off
    Warning Start to ignore needs like sleeping/eating,
    reduced social interactions, panic, loss of
    interest in family/hobbies
    Danger Behavioral changes, cynicism,
    drugs/alcohol, promiscuousness,
    aggression, worthlessness/hopelessness,
    on autopilot
    Emergency Inner emptiness, more risk-taking,
    depression, suicide
    

    View Slide

11. Burnout Happens in Tech
    ● Caused by high-stress environments
    ● Caused by not having basic needs met
    ○ NEVER underestimate the value of sleep
    ○ Water is also pretty important
    ● Caused by emphasis on performance
    ○ Fear of underperforming
    ○ Not taking vacation
    ● Sedentary lifestyle (often found in tech!) can exacerbate the problem
    

    View Slide

12. View Slide

13. On Call
    Burnout
    

    View Slide

14. Preparation is Key
    ● Keep your laptop charged
    ● Keep your MiFi charged
    ● Keep a spare battery pack in your backpack
    ● Toss an energy drink and a small snack in your bag
    ● Bring cables you may need
    

    View Slide

15. Reduce Stress
    ● Pay important bills
    ● Run errands
    ● Set yourself up for an easier work week if possible
    ○ Do the hard stuff now instead of later
    ○ Don’t put off the boring/hard tasks until the last minute
    

    View Slide

16. Forgetting You’re On Shift
    ● It happens more than you think.
    ● RELAX!
    ● If your bag is packed and you have cell reception, you’re probably okay.
    ● If reception = false, get to a place that has it
    

    View Slide

17. How do we make life easier
    in-house?
    

    View Slide

18. Step One:
    Set better schedules!
    

    View Slide

19. Starting/Ending Rotations
    ● Start and end rotations during the day
    ● Midweek
    ● NEVER start/end on weekends
    ● DO NOT START ON A HOLIDAY
    ○ Avoid scheduling people on holidays at
    all if possible
    ○ Share the load
    

    View Slide

20. VACATION
    

    View Slide

21. Step Two:
    Select the right shift length
    

    View Slide

22. Common Shift Lengths
    ● 8 hours
    ● 12 hours
    ● 24 hours
    ● 168 hours/1 week
    

    View Slide

23. 8-hour Shifts
    

    View Slide

24. 12-hour Shifts
    

    View Slide

25. View Slide

26. 24-hour Shifts
    

    View Slide

27. 168-hour Shifts/1 Week Shifts
    

    View Slide

28. What rotation
    should I use?
    

    View Slide

29. Week+Weekends
    

    View Slide

30. Scheduling
    Overrides
    “No battle plan
    survives first contact
    with the enemy.”
    Roughly translated from the
    thesis of Helmuth von Moltke the
    Elder
    

    View Slide

31. Step Three:
    Don’t Make the Pain in Vain
    

    View Slide

32. Make Your Data
    Count!
    

    View Slide

33. Tooling
    Matters!
    

    View Slide

34. Step Four:
    Keep Staff Sharp
    

    View Slide

35. Incident Response
    http://www.blackrock3.com/
    

    View Slide

36. You write it,
    you wear it.
    

    View Slide

37. Thank You!
    Amanda Folson
    amanda@gitlab.com
    @AmbassadorAwsum
    

    View Slide