Strategies For Being On Call & Keeping Your Sanity At The Same Time

Transcript

1. Strategies for being on-call & keeping your sanity at the

   same time. Eric Sigler, Engineering Manager, PagerDuty @esigler

2. First things first. @esigler

3. No silver bullets. @esigler

4. 3AM pages are a form of technical debt collection. @esigler

5. Invest time in the underlying issues. @esigler

6. Empathy takes time to build. @esigler

7. Before, During, After. @esigler

8. Before Going On Call @esigler

9. @esigler Before Going On Call > Get Help!

10. “I don’t want to get woken up in the middle

    of the night.” @esigler Before Going On Call > Get Help!

11. Question Everything. (Especially alerts.) @esigler Before Going On Call >

    Review your alerts

12. How is the alert triggered? @esigler Before Going On Call

    > Review your alerts

13. <Obligatory Nagios Joke Here> @esigler Before Going On Call >

    Review your alerts

14. What should someone do when they get the alert? @esigler

    Before Going On Call > Review your alerts

15. The “everything’s OK” alarm. @esigler Before Going On Call >

    Review your alerts

16. Why does it matter to the business? @esigler Before Going

    On Call > Review your alerts

17. Production is down at 3AM? I care! Staging is down

    at 3AM? Less so. @esigler Before Going On Call > Review your alerts

18. Practice makes perfect. @esigler Before Going On Call > Practice

    Beforehand

19. Game Days, Chaos Monkeys, Failure Fridays, pick what works for

    you. @esigler Before Going On Call > Practice Beforehand

20. During Your On Call shift. @esigler

21. Be flexible to those who are on call. @esigler During

    On-Call > Be Flexible.

22. Kick responders off as soon as possible. @esigler During On-Call

    > Scope down responders quickly

23. Scope down as soon as you know the business impact.

    @esigler During On-Call > Scope down based on impact quickly

24. After Going On Call. @esigler

25. Include the impact to the responder in your postmortems. @esigler

    After On-Call > Consider ALL Impacts In Postmortems

26. Do a periodic review of what alerts were triggered. @esigler

    After On-Call > Take A Look At What Actually Happened

27. https://github.com/etsy/ opsweekly https://opzzz.sh/ @esigler

28. Let’s recap: The business needs to invest in on call

    Get the right people involved, build empathy Take a close look at what actually can wake someone up Have everyone practice and exercise the systems during the day Kick people off the call quickly Kick the problem to daytime hours quickly Consider the impact to the responder in postmortems Do a periodic review of how painful the on call period was @esigler

29. On call can be reasonable, but it takes a lot

    of investment. @esigler

30. Thanks! @esigler