Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On-Calliday: Un-sucking Your On-Call Experience

On-Calliday: Un-sucking Your On-Call Experience

Being on call sucks, but it doesn’t have to. Many organizations struggle to maintain effective on-call rotations that minimize the number of people woken up in the middle of the night -- so much so, that “burnout syndrome” is an actual disorder that’s commonly experienced by professionals in on-call positions. In this talk, we’ll talk about how you can set up your on-call rotations to optimize the uptime of your infrastructure and your engineering team.

F822f1d39cf52a0b934b9481472d7996?s=128

Amanda Folson

July 09, 2016
Tweet

Transcript

  1. On-Calliday: Un-sucking Your On-Call Experience Amanda Folson Developer Advocate @

    GitLab
  2. Amanda Folson Developer Advocate @ GitLab amanda@gitlab.com

  3. Who’s on call?

  4. Why does being on call suck? • No sleep •

    Too many distractions • Too little time • Tethered to your phone/laptop • Exhaustion
  5. It Can Be Better!

  6. Burnout Caution Needing to prove yourself, feeling you don’t provide

    value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
  7. Burnout Caution Needing to prove yourself, feeling you don’t provide

    value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
  8. Burnout Caution Needing to prove yourself, feeling you don’t provide

    value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
  9. Burnout Caution Needing to prove yourself, feeling you don’t provide

    value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
  10. Burnout Caution Needing to prove yourself, feeling you don’t provide

    value, working harder, difficult to switch off Warning Start to ignore needs like sleeping/eating, reduced social interactions, panic, loss of interest in family/hobbies Danger Behavioral changes, cynicism, drugs/alcohol, promiscuousness, aggression, worthlessness/hopelessness, on autopilot Emergency Inner emptiness, more risk-taking, depression, suicide
  11. Burnout Happens in Tech • Caused by high-stress environments •

    Caused by not having basic needs met ◦ NEVER underestimate the value of sleep ◦ Water is also pretty important • Caused by emphasis on performance ◦ Fear of underperforming ◦ Not taking vacation • Sedentary lifestyle (often found in tech!) can exacerbate the problem
  12. None
  13. On Call Burnout

  14. Preparation is Key • Keep your laptop charged • Keep

    your MiFi charged • Keep a spare battery pack in your backpack • Toss an energy drink and a small snack in your bag • Bring cables you may need
  15. Reduce Stress • Pay important bills • Run errands •

    Set yourself up for an easier work week if possible ◦ Do the hard stuff now instead of later ◦ Don’t put off the boring/hard tasks until the last minute
  16. Forgetting You’re On Shift • It happens more than you

    think. • RELAX! • If your bag is packed and you have cell reception, you’re probably okay. • If reception = false, get to a place that has it
  17. How do we make life easier in-house?

  18. Step One: Set better schedules!

  19. Starting/Ending Rotations • Start and end rotations during the day

    • Midweek • NEVER start/end on weekends • DO NOT START ON A HOLIDAY ◦ Avoid scheduling people on holidays at all if possible ◦ Share the load
  20. VACATION

  21. Step Two: Select the right shift length

  22. Common Shift Lengths • 8 hours • 12 hours •

    24 hours • 168 hours/1 week
  23. 8-hour Shifts

  24. 12-hour Shifts

  25. None
  26. 24-hour Shifts

  27. 168-hour Shifts/1 Week Shifts

  28. What rotation should I use?

  29. Week+Weekends

  30. Scheduling Overrides “No battle plan survives first contact with the

    enemy.” Roughly translated from the thesis of Helmuth von Moltke the Elder
  31. Step Three: Don’t Make the Pain in Vain

  32. Make Your Data Count!

  33. Tooling Matters!

  34. Step Four: Keep Staff Sharp

  35. Incident Response http://www.blackrock3.com/

  36. You write it, you wear it.

  37. Thank You! Amanda Folson amanda@gitlab.com @AmbassadorAwsum