Human Centered On-Call Practices

Over the years we’ve been conditioned to “move fast and break things,” but sometimes the things we’re breaking are our people and our teams. On-call rotations can result in frustration and burnout, but we can recover from that by focusing on the people involved.

Transcript

1. Najla Elmachtoub Engineering Manager @ Etsy Human Centered On-Call Practices
2. 2 On call-rotations
3. 3 ⚠ Issue in production On call-rotations
4. 4 ⚠ Issue in production Engineer gets paged On call-rotations
5. 5 ⚠ Issue in production Engineer gets paged ‍ Issue is resolved On call-rotations
6. 6 My on-call rotation in 2019
7. 7 My on-call rotation in 2019 5 people on call
8. 8 My on-call rotation in 2019 5 people on call ➡ A lot of pages every week
9. 9 5 people on call ➡ A lot of pages every week ❗ ...especially outside of working hours My on-call rotation in 2019
10. 10 My on-call rotation in 2019
11. The human cost
12. The human cost
13. 13 Where do we go from here?
14. 14 Where do we go from here? Option #1 Do nothing
15. 15 Where do we go from here? Option #1 Do nothing Option #2 Take a data driven approach ❌
16. 16 Where do we go from here? Option #1 Do nothing Option #2 Take a data driven approach ❌ ❌ Option #3 Take a human- centered approach
17. Option #1 Do nothing Option #2 Take a data driven approach 17 ❌ ❌ ✅ Option #3 Take a human- centered approach Where do we go from here?
18. 18 Mitigate the biggest pain point
19. 19 Mitigate the biggest pain point ‍‍ Expanded from 5 → 15 Reduced frequency by 3x
20. 20 ‍‍ Expanded from 5 → 15 Reduced frequency by 3x New folks had little context Mitigate the biggest pain point
21. 21 Mitigate the biggest pain point New folks had little context
22. Team retro
23. Team retro “It seems like a good number of our alerts aren’t actionable.” “Some of our alerts could be mitigated with proactive eng work.” “How many of our alerts actually need to be fixed in the middle of the night?”
24. Team retro Root problem: Unactionable pages are stressful and feel like a waste of time and energy
25. 25 Digging into the data
26. Plan of action Tune top alerts
27. Plan of action Tune top alerts ✍ Write and share documentation
28. 28 Retro Data deep dive Alert tuning Creating the feedback loop
29. 29 Outcomes A happier team
30. 30 A happier team More robust systems Outcomes
31. 31 Off hours pages way down
32. 32 My takeaways Process debt is real
33. 33 My takeaways Process debt is real Focusing solely on number of pages misses the point
34. 34 Process debt is real Focusing solely on number of pages misses the point ‍♂ People love to help you My takeaways
35. Thank you LinkedIn: /najlaelmachtoub Twitter: @najtube Where to find me