Humane Treatment of On-Call Engineers

Exploring what sucks in the On Call process and how to fix it.

Transcript

1. Aaron Aldrich @CrayZeigh Cage Data, Inc. @CageData ON-CALL, WHY IT SUCKS AND HOW WE CAN MAKE IT A LITTLE BETTER. DEVOPS CT: BURNOUT
2. THANKS! AARON ALDRICH ▸ @CrayZeigh ▸ aaldrich@cagedata.com ▸ Cage Data, Inc. ▸ @CageData ▸ Part of On Call rotation since 2009 through 3 different Orgs ▸ Husband, Father of 4, Musician, Board Gamer, Team Mystic
3. Aaron Aldrich @CrayZeigh THE BURNOUT PROBLEM
4. Aaron Aldrich - @CrayZeigh
5. Aaron Aldrich - @CrayZeigh OVERALL CONTRIBUTORS TO BURNOUT ▸ Internal management Practice ▸ Change/Deployment Environment ▸ Organizational Culture ▸ Investment in Technology
6. Aaron Aldrich @CrayZeigh ON CALL ONE PIECE OF THE PUZZLE
7. Aaron Aldrich - @CrayZeigh
8. Aaron Aldrich - @CrayZeigh Humans can’t work all the time. We need food and sleep and free time.
9. Aaron Aldrich - @CrayZeigh
10. Aaron Aldrich - @CrayZeigh On-Call should be thought of as an Emergency Response Team for your technology assets
11. Aaron Aldrich - @CrayZeigh
12. Aaron Aldrich @CrayZeigh MAKING ON CALL MEANINGFUL ACTIONABLE ALERTS
13. Aaron Aldrich - @CrayZeigh
14. Aaron Aldrich - @CrayZeigh Being on call should mean something more than just being woken up in the middle of the night.
15. Aaron Aldrich - @CrayZeigh ACTIONABLE ALERTS ▸Something needs to be done ▸That something needs to be done now ▸The person receiving the alert is empowered to act upon it
16. Aaron Aldrich - @CrayZeigh Does it Matter?
17. Aaron Aldrich - @CrayZeigh Audible alarms are not only incredibly annoying, but almost entirely useless: they do not reduce crime or deter thieves, and they produce an overall net negative impact on society.
18. Aaron Aldrich - @CrayZeigh If there’s nothing to do, log it and move on.
19. Aaron Aldrich - @CrayZeigh Can it wait?
20. Aaron Aldrich - @CrayZeigh IT SOUNDS PLAUSIBLE ENOUGH TONIGHT, BUT WAIT UNTIL TOMORROW. WAIT FOR THE COMMON SENSE OF THE MORNING. HG Wells, The Time Machine
21. Aaron Aldrich - @CrayZeigh
22. Aaron Aldrich - @CrayZeigh Is the right person getting the call?
23. Aaron Aldrich - @CrayZeigh The person receiving an alert should be equipped to deal with it
24. Aaron Aldrich - @CrayZeigh
25. Aaron Aldrich - @CrayZeigh Does it Matter? Can it wait? Is the right person getting the call?
26. Aaron Aldrich @CrayZeigh BUILDING A SUPPORT STRUCTURE SHIFTING DEPARTMENTS
27. Aaron Aldrich - @CrayZeigh
28. Aaron Aldrich - @CrayZeigh
29. Aaron Aldrich - @CrayZeigh
30. Aaron Aldrich - @CrayZeigh Runbooks are the bread and butter of the late-night work process.
31. Aaron Aldrich - @CrayZeigh RUNBOOKS TELL US ▸ What’s impacted ▸ How to connect with it ▸ Clear Steps on what to do to fix a problem ▸ If it doesn’t fix the problem, where to begin next ▸ And/Or How to call in backup WHAT THE ALERT MEANS AND HOW TO ACT ON IT
32. Aaron Aldrich - @CrayZeigh
33. Aaron Aldrich - @CrayZeigh Good Runbooks make automation possible
34. Aaron Aldrich - @CrayZeigh Let the machines work for us
35. Aaron Aldrich - @CrayZeigh Any automated fix to a persistent problem should exist as a stop-gap only. Reducing unplanned work in order to plan on finding a solution.
36. Aaron Aldrich - @CrayZeigh It’s important that the on-call personnel are granted the authority to call in for backup
37. Aaron Aldrich @CrayZeigh CHANGING THE WAY WE TALK ABOUT ON CALL SHIFTING CULTURE
38. Aaron Aldrich - @CrayZeigh /r/notinteresting
39. Aaron Aldrich - @CrayZeigh We all tell stories, and it takes a level of struggle to make that story worth telling.
40. Aaron Aldrich - @CrayZeigh
41. Aaron Aldrich - @CrayZeigh Swapping stories about trying times helps deal with the stress, but an environment that does nothing to avoid them causes problems in the long run.
42. Aaron Aldrich - @CrayZeigh SOFTWARE DEVELOPMENT, IT TURNS OUT, IS A TEAM SPORT… AND WHAT’S WORSE, ENCOURAGING THE HERO MENTALITY LEADS TO CORROSIVE DYSFUNCTION IN SOFTWARE TEAMS. Rob Mee, Pivotal Labs
43. Aaron Aldrich - @CrayZeigh COMPENSATION FOR EXTRA UNPLANNED WORK ▸ Working after-hours should be the exception not the norm ▸ When it happens you deserve to get compensated for your time ▸ Large Organizations: ▸ Actual real monetary compensation ▸ Extra vacation time ▸ Company covers human fuel costs during long extra-workday hours ▸ Small Organizations & Startups: ▸ May be more baked into the culture ▸ Flexible hours, with expectations of taking time off ▸ Fostering a real culture of appreciating your employees
44. Aaron Aldrich - @CrayZeigh Postmortem?
45. Aaron Aldrich - @CrayZeigh WHAT DOES A POSTMORTEM DO? ▸ What happened? ▸ Create a timeline of events ▸ What went well? ▸ What should we keep doing? ▸ What went poorly? ▸ How can we improve?
46. Aaron Aldrich - @CrayZeigh WHAT IS A BLAMELESS POSTMORTEM? ▸ Team members are accountable but not responsible ▸ Complete Transparency ▸ Deeper look at circumstances ▸ What happened and how to improve it (specific details) ▸ Real conditions of failure in complex systems @jasonhand
47. Aaron Aldrich - @CrayZeigh WHY DO WE DO IT? ▸ Be aware of cognitive biases, especially hindsight ▸ Encourage & note divergent views ▸ The goal is to learn how to improve, not create a scapegoat ▸ Any tasks that effect quality of life for your employees should be prioritized first LEARNING RETROSPECTIVES
48. Aaron Aldrich - @CrayZeigh
49. Aaron Aldrich @CrayZeigh LET ME SUM UP. …THERE IS TOO MUCH
50. Aaron Aldrich - @CrayZeigh
51. Aaron Aldrich - @CrayZeigh
52. Aaron Aldrich - @CrayZeigh
53. THANKS! AARON ALDRICH ▸ @CrayZeigh ▸ aaldrich@cagedata.com