The Perfect Storm: How We Talk About Disasters

A bit of a case-study on what to do when you have a perfect storm of errors that cause big prolific outages. I share tips on what PagerDuty has done internally to illustrate how DevOps principles have started to infect non-Engineering parts of our organization. This is a call to action to use business processes as a way to propagate DevOps adoption and remove organizational barriers.

Transcript

1. @gmiranda23 HOW WE TALK ABOUT DISASTERS THE PERFECT STORM
2. George Miranda Engineer, Traveler, First Responder Talky Person @PagerDuty gmiranda@pagerduty.com 2 @gmiranda23
3. @gmiranda23 I AM A SYSADMIN
4. @gmiranda23 RUNNING PRODUCTION SYSTEMS IS A MULTI-DISCIPLINARY PRACTICE
5. @gmiranda23 Broad Knowledge Deep Expertise
6. @gmiranda23 Broad Knowledge Deep Expertise Deep Expertise Deep Expertise
7. @gmiranda23 Broad Knowledge Deep Expertise Deep Expertise Deep Expertise Deep Expertise
8. @gmiranda23 DESTROYING SILOS
9. @gmiranda23
10. @gmiranda23 WE STILL NEED SILOS OF DEEP EXPERTISE
11. @gmiranda23 Running software in Production Software Development Running software in Production Infrastructure Running software in Production Release Engineering Running software in Production Capacity Planning Running software in Production Databases Running software in Production Observability
12. @gmiranda23 HIGH PERFORMING TEAMS ARE CROSS-FUNCTIONAL
13. @gmiranda23 10TH ANNIVERSARY OF DEVOPS
14. @gmiranda23 BROAD KNOWLEDGE OF RUNNING A SOFTWARE BUSINESS
15. @gmiranda23 Severe Technical Incident C-Suite Marketing Legal Engineering Finance Customer Facing PR Comms Team Sales
16. @gmiranda23 BUSINESS PROCESSES AS A BRIDGE
17. @gmiranda23 WHAT'S THE WORST OUTAGE YOU'VE EXPERIENCED IN YOUR CAREER?
18. @gmiranda23 WHAT'S THE WORST OUTAGE YOU'VE EXPERIENCED AS A USER?
19. @gmiranda23 YOUR CUSTOMERS FEEL THE SAME WAY
20. @gmiranda23 WHO HAS A PLAN FOR THAT SCENARIO?
21. @gmiranda23 A TALE ABOUT PAGERDUTY
22. @gmiranda23 CHAOS INSTEAD OF CALM
23. @gmiranda23 "LET'S TRY AND RESOLVE THIS IN THE NEXT 10 MINUTES, MMMKAY?"
24. @gmiranda23 EMERGENCY PREPAREDNESS
25. @gmiranda23 A TALE ABOUT PAGERDUTY
26. @gmiranda23 THE INCIDENT COMMAND SYSTEM
27. @gmiranda23 RESPONSE.PAGERDUTY.COM
28. OPERATIONS LIAISONS COMMAND Deputy Scribe Customer Liaison SME Incident Commander (IC) Internal Liaison SME SME SME TECHNICAL INCIDENT @gmiranda23
29. OPERATIONS LIAISONS COMMAND Deputy Scribe Customer Liaison SME Incident Commander (IC) Internal Liaison SME SME SME Technical SLT TECHNICAL INCIDENT @gmiranda23
30. OPERATIONS LIAISONS COMMAND Deputy Scribe SME Incident Commander (IC) Internal Liaison SME SME SME TECHNICAL INCIDENT BUSINESS INCIDENT APPROVAL Comms Legal C-Suite COMMAND Communications Lead Customer Liaison Technical SLT OPERATIONS Social Sales Marketing Customer Liaison (backup) @gmiranda23
31. @gmiranda23 SWITCH THE BUSINESS INTO EMERGENCY MODE
32. @gmiranda23
33. @gmiranda23 CHAOS ENGINEERING SUPPORT
34. @gmiranda23 BLAMELESS POSTMORTEM AWARENESS
35. @gmiranda23 ADOPTION OF RETROS
36. @gmiranda23 VISITING OTHER SILOS
37. @gmiranda23 "IT'S THE FEAR OF SOMETHING NEW (DEVOPS) THAT MAKES CONFLICT POSSIBLE"
38. @gmiranda23 BUSINESS-RESPONSE.PAGERDUTY.COM
39. @gmiranda23 ADDITIONAL RESOURCES • PAGERDUTY.COM/OPS-GUIDES • RESPONSE.PAGERDUTY.COM • POSTMORTEMS.PAGERDUTY.COM • BUSINESS-RESPONSE.PAGERDUTY.COM