Blameless Postmortems - Security by Inclusion

Transcript

1. Blameless Postmortems Security by Inclusion
2. VP Engineering - Pushpay @josh_robb Me
3. Pushpay 17 Members of Technical Staff (Engineering) Continuous Delivery Mobile Apps PCI DSS Level 1 Devops/#chatops (really Slackops) Heavy Code review culture METRICS (4,933 or so currently)
4. What Why How
5. Origins Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand. Retrospective Prime Directive (2001) Norman Kerth
6. Etsy Blameless Postmortems and a Just Culture - John Allspaw (2012) Human Factors Research (in Healthcare and Aviation)
8. SR71 Blackbird Then you'd debrief for an hour or more, [...] these [planes] were all hand-built, so you had to go through things with the other pilots and engineers like "I had this happen, and it's not in the checklist." Then another pilot would say, "I saw something like that before," and go back and try to correlate it.
9. SR71 Blackbird You were all working on it together, and there were no secrets. You'd say "I screwed this up" to everyone in order to grow the knowledge base.
10. Why? You want multiple and diverse perspectives. You get these by asking people for their own narratives. Effectively, you’re asking “how?“ Asking “why?” too easily gets you to an answer to the question “who?” The Infinite hows (not 5 whys) John Alspaw (again!) (2014)
11. Why -> Who is responsible How -> What is responsible
12. Why? Continuous improvement Increased quality Safe - people (more) willing to say if they’re under trained for the situations they find themselves in More secure
13. Not “How did this happen” BUT “What can we do to prevent this happening next time”
14. Not a whip
15. NOT A WHIP It’s tempting to “make” someone write a postmortem up. It’s tempting to use them as performance reviews or to pressure people. DONT
18. Lead by example Your behaviour sets the tone Be the first to write up postmortems Watch YOUR tone Coach others privately on tone “It’s not YOUR fault”
19. Etsy (again) An engineer who thinks they’re going to be reprimanded is disincentivized to give the details necessary to get an understanding of the mechanism, pathology, and operation of the failure. This lack of understanding of how the accident occurred all but guarantees that it will repeat. If not with the original engineer, another one in the future.
20. Look at what went well We try to mention/“celebrate” previous mitigations which reduced the blast radius this time around.
21. How Slack/Chat - not in person Alpha geek doesn’t shout the loudest in chat (sometimes)
22. Write it up We have a template
24. Write it up We have a template 1. Timeline (reconstructed from slack chatops #situation-room and #devops channels) 2. Discussion. What happened. Assumptions. Other factors. 3. Mitigation - What can we do to stop this next time?
25. Timeline What happened? When? • Assumptions • Workarounds • Solutions
26. Discussion Metrics Things which are being investigated Possible resolutions/mitigations Assumptions
27. Good mitigations • Anticipate • Monitor • Respond • Learn
28. Good mitigations cont’d • Owned by an individual • Tracked • Followed through • For us - this means a JIRA ticket
29. Mitigations Technical/automatic are STRONGLY preferred over “scar tissue” (i.e. human processes)
30. Postmortem Postmortem One thing we’re going to do next week is a postmortem on our postmortems. (Well - retrospective - but that sounds less recursive) What could we do better?
31. Tools Etsy Morgue - github.com/etsy/morgue Nonviolent Communication - Marshall Rosenberg The Human Side of Postmortems - Dave Zwieback