PyOhio 2019 - The Blameless Post Mortem: How Embracing Failure Makes Us Better

Transcript

1. crwilcox @chriswilcox47 The blameless post mortem How embracing failure makes us better.
2. crwilcox @chriswilcox47 Software is pretty amazing
3. crwilcox @chriswilcox47 crwilcox @chriswilcox47 And so are its bugs...
4. crwilcox @chriswilcox47 crwilcox @chriswilcox47 What if we could learn from the bugs?
5. crwilcox @chriswilcox47 crwilcox @chriswilcox47 https://crwilcox.com https://chriswilcox.racing https://speakerdeck.com/crwilcox About Me.
6. crwilcox @chriswilcox47 https://www.history.navy.mil
7. crwilcox @chriswilcox47 A look at other industries.
8. crwilcox @chriswilcox47 Sometimes decisions are actually, not in hyperbole, life and death.
9. crwilcox @chriswilcox47 Every “Mistake” is an opportunity to better the system.
10. crwilcox @chriswilcox47 The NTSB. National Transportation Safety Board
11. crwilcox @chriswilcox47 crwilcox @chriswilcox47 The NTSB has no formal, prosecutorial, authority.
12. crwilcox @chriswilcox47 crwilcox @chriswilcox47 And has provided 13,000+ safety recommendations.
13. crwilcox @chriswilcox47
14. crwilcox @chriswilcox47
15. crwilcox @chriswilcox47 https://www.ntsb.gov/investigations/AccidentReports/Reports/MAB1917.pdf
16. crwilcox @chriswilcox47 Industry Process: M&M Conferences https://www.marsfoodservices.com
17. crwilcox @chriswilcox47 Morbidity & Mortality conferences - Provide medical education. - Improve quality of care. - Highlight cases with diagnostic uncertainty, complex management, etc. https://www.imperial.ac.uk
18. crwilcox @chriswilcox47 What can we, as technologists, learn from these other industries?
19. crwilcox @chriswilcox47 crwilcox @chriswilcox47 How might we conduct a post mortem?
20. crwilcox @chriswilcox47 The structure of a post mortem. - Why did this happen? - Could it have been worse? Better? - How can we make sure it doesn’t happen again?
21. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation for the Google Cloud Python libraries was unavailable for users.
22. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. As part of repository cleanup, an engineer with write access to the development repository deleted the gh-pages branch.
23. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. During a two hour period, hosted reference documentation for our libraries was unavailable.
24. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. This was first detected via an external customer. Shortly after two internal teams also noticed and notified. The delay to the initial report was 30 minutes following the start of the outage.
25. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation was available after republishing to gh-pages.
26. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. 2 hours. 2019-12-05 (all times PDT) 09:45 Branch gh-pages deleted during clean up of repository branches. 10:21 GitHub issue filed stating that docs are not available. 10:45 Team responds to issue. 10:55 Branch has been republished 11:45 GitHub is serving docs again.
27. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. We were notified fairly quick and the team monitors GitHub issues well enough that we knew about it pretty fast.
28. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The branch should have been protected from deletion. GitHub page publishing is a bit opaque. While the branch was pushed within about 5 minutes of finding out it took an additional hour or so to get it actually serving the files. The lack of debugging information made this more difficult.
29. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. A member of the team had a local copy of the branch handy and was able to republish.
30. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger. - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. Protect gh-pages branch. Don’t allow deletion. Investigate if debugging gh-pages could be improved. Investigate if there are other technologies we ought to be using instead of gh-pages.
31. crwilcox @chriswilcox47 crwilcox @chriswilcox47 We fail so we can learn.
32. crwilcox @chriswilcox47 crwilcox @chriswilcox47 “If you've never missed a flight, you're probably spending too much time in airports“ -George Stigler
33. crwilcox @chriswilcox47 Blame isn’t a deterrent. It isn’t anything useful.
34. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Excusing failure as human error stops the conversation.
35. crwilcox @chriswilcox47 Why we don’t blame; an example.
36. crwilcox @chriswilcox47
37. crwilcox @chriswilcox47 Building a blameless post mortem culture.
38. crwilcox @chriswilcox47 Create psychological safety.
39. crwilcox @chriswilcox47 Find positive examples. https://landing.google.com/sre/books
40. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Outages happen and no one wanted them to.
41. crwilcox @chriswilcox47 A Test for Blamelessness 1. A script was executed that deleted the production database. 2. An engineer executed a script that deleted the production database without confirmation. 3. Chris executed a script that deleted the production database without confirmation.
42. crwilcox @chriswilcox47 crwilcox @chriswilcox47 “But Chris, we never have outages at my work, we are perfect, in every way” -Someone, Probably.
43. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Raise your own bar, Break stuff.
44. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Add some 9’s to that SLA!
45. crwilcox @chriswilcox47 Thanks! References. https://crwilcox.com https://www.ntsb.gov/investigations/AccidentReports/Reports/MAB1917.pdf https://acphospitalist.org/archives/2018/07/the-new-improved-m-and-m.htm https://insights.ovid.com/pubmed?pmid=26983075&_ga=2.103365704.971425802.15638 99461-1002013713.1563899459 https://www.ama-assn.org/residents-students/residency/presenting-your-first-mm -conference-5-things-you-need-know https://landing.google.com/sre/books https://cloud.google.com/blog/products/gcp/fearless-shared-postmortems-cre-lif e-lessons https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/