Paging, Alerting, and Chaos Overview

Slides from an internal talk at my company about adopting better SEV managment practices and how we might use chaos engineering to test our infrastructure regularly.

Transcript

1. Paging/Alerting Workshop #sre-office-hours | 04.04.2018
2. What qualifies as an incident?
3. What should happen in an incident?
4. Who should be alerted/paged?
5. What tools would we use to gather data?
6. How reliable/accurate would this information be?
7. What’s the acceptable time to be alerted/respond?
8. An Initial Framework for Discussing SEVs
9. What is a SEV? SEV is a term used to refer to an incident, it is derived from the word severity.
10. Common types of SEV - Availability Drop - Product Issue / Feature Broken - Data Loss - Security Risk - etc.
11. SEV Levels
12. Any SEV which involves a loss of customer data should be classified as SEV0.
13. Calculate Critical Uptime in 9’s https://uptime.is/
14. What uptime do you think we could safely publish? Would we be okay telling our customers/vendors that number?
15. SEV Terminology
16. Lifecycle of a SEV event
17. How to measure a SEV? % loss * outage duration
18. Visualizing/tracking SEV’s
19. How do we maintain combat effectiveness during a SEV?
20. Incident Manager On-Call (IMOC) - Should be a small rotation of Engineering Leaders - Only one person is on-call in this role at any point in time - These people should possess a wide knowledge of services and engineering teams - Will be our version of Air-Traffic-Control for the SEV, ensuring different people working on the SEV are organized and working coherently as a unit!
21. Tech Lead On-Call (TLOC) - This would be the engineer driving resolution of the SEV - Should have deep knowledge of specific domain of knowledge; be a SME (Subject Matter Expert) - Should have a deep knowledge of upstream and downstream dependencies
22. What we need to define to have these roles: - IMOC runbook/guide - Designate a Primary and Secondary IMOC at all times - Escalation should be automatic - Monthly sync for all IMOC and TLOC - Way to quickly triage what systems are effected/find root cause - How would we do this? - How do we record / document SEV’s? - Google Form? Git repo? Suggestions?? - SEV naming convention
23. What happens when we don’t meet our uptime requirements?!?
24. What causes SEV’s?
25. Technical Issues • Dependency Failure • Cloud Provider Region/Zone Failure • Provider Failure • Connectivity Issues • Power issues (our local office power affects AWS RDS!) • DNS outage/latency • Misconfiguration of machines/docker images • Software Bugs • Corrupt/unavailable backups
26. Cultural Issues • Lack of knowledge sharing • Lack of knowledge handover • Lack of on-call training • Lack of chaos engineering • Lack of a high severity incident management program • Lack of documentation and playbooks • Lack of alerts and pages • Lack of effective alerting thresholds • Lack of backup strategy
27. How do we prevent SEVs from repeating? • Combination of: ◦ Record outages ◦ Correlate failures ◦ Track SEVs
28. Chaos Engineering!
29. What if we could break things safely!? What lessons/data could we gather?
30. Chaos Engineering...yes, it is a real thing! • 2010 - Netflix created the Chaos Monkey which can wreak havoc in AWS at will deleting instances (but fully customizable/controllable) -- this is OSS as of 2012 • 2011 - Netflix creates the Simian Army--a host of chaos tools to test failure modes in your infrastructure and applications • 2014 - the Role of Chaos Engineer is created at Netflix
31. Principles of Chaos Engineering
32. Can we do this?
34. And we all get to “share the pain” with our new tool PagerDuty...
35. Credits - https://www.gremlin.com/community/tutorials/how-to-establish-a-high-severity-inciden t-management-program/ - https://www.gremlin.com/community/tutorials/chaos-engineering-the-history-principles -and-practice/ - https://www.gremlin.com/the-discipline-of-chaos-engineering/ - https://github.com/tammybutow/chaos_engineering_bootcamp - https://www.usenix.org/conference/srecon17americas/program/presentation/andrus They did a cool workshop about Chaos Engineering with hands-on labs at SREcon this year. If you like this notion of chaos, more of us should go next year! #notjustforSRE
36. Questions/thoughts? #sre-office-hours