PyOhio 2019 - The Blameless Post Mortem: How Embracing Failure Makes Us Better

PyOhio 2019 - The Blameless Post Mortem: How Embracing Failure Makes Us Better

5d95f89b9892e12c0d7fa2f671edac62?s=128

Christopher Wilcox

July 27, 2019
Tweet

Transcript

  1. crwilcox @chriswilcox47 The blameless post mortem How embracing failure makes

    us better.
  2. crwilcox @chriswilcox47 Software is pretty amazing

  3. crwilcox @chriswilcox47 crwilcox @chriswilcox47 And so are its bugs...

  4. crwilcox @chriswilcox47 crwilcox @chriswilcox47 What if we could learn from

    the bugs?
  5. crwilcox @chriswilcox47 crwilcox @chriswilcox47 https://crwilcox.com https://chriswilcox.racing https://speakerdeck.com/crwilcox About Me.

  6. crwilcox @chriswilcox47 https://www.history.navy.mil

  7. crwilcox @chriswilcox47 A look at other industries.

  8. crwilcox @chriswilcox47 Sometimes decisions are actually, not in hyperbole, life

    and death.
  9. crwilcox @chriswilcox47 Every “Mistake” is an opportunity to better the

    system.
  10. crwilcox @chriswilcox47 The NTSB. National Transportation Safety Board

  11. crwilcox @chriswilcox47 crwilcox @chriswilcox47 The NTSB has no formal, prosecutorial,

    authority.
  12. crwilcox @chriswilcox47 crwilcox @chriswilcox47 And has provided 13,000+ safety recommendations.

  13. crwilcox @chriswilcox47

  14. crwilcox @chriswilcox47

  15. crwilcox @chriswilcox47 https://www.ntsb.gov/investigations/AccidentReports/Reports/MAB1917.pdf

  16. crwilcox @chriswilcox47 Industry Process: M&M Conferences https://www.marsfoodservices.com

  17. crwilcox @chriswilcox47 Morbidity & Mortality conferences - Provide medical education.

    - Improve quality of care. - Highlight cases with diagnostic uncertainty, complex management, etc. https://www.imperial.ac.uk
  18. crwilcox @chriswilcox47 What can we, as technologists, learn from these

    other industries?
  19. crwilcox @chriswilcox47 crwilcox @chriswilcox47 How might we conduct a post

    mortem?
  20. crwilcox @chriswilcox47 The structure of a post mortem. - Why

    did this happen? - Could it have been worse? Better? - How can we make sure it doesn’t happen again?
  21. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation for the Google Cloud Python libraries was unavailable for users.
  22. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. As part of repository cleanup, an engineer with write access to the development repository deleted the gh-pages branch.
  23. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. During a two hour period, hosted reference documentation for our libraries was unavailable.
  24. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. This was first detected via an external customer. Shortly after two internal teams also noticed and notified. The delay to the initial report was 30 minutes following the start of the outage.
  25. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation was available after republishing to gh-pages.
  26. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. 2 hours. 2019-12-05 (all times PDT) 09:45 Branch gh-pages deleted during clean up of repository branches. 10:21 GitHub issue filed stating that docs are not available. 10:45 Team responds to issue. 10:55 Branch has been republished 11:45 GitHub is serving docs again.
  27. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. We were notified fairly quick and the team monitors GitHub issues well enough that we knew about it pretty fast.
  28. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The branch should have been protected from deletion. GitHub page publishing is a bit opaque. While the branch was pushed within about 5 minutes of finding out it took an additional hour or so to get it actually serving the files. The lack of debugging information made this more difficult.
  29. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. A member of the team had a local copy of the branch handy and was able to republish.
  30. crwilcox @chriswilcox47 An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. Protect gh-pages branch. Don’t allow deletion. Investigate if debugging gh-pages could be improved. Investigate if there are other technologies we ought to be using instead of gh-pages.
  31. crwilcox @chriswilcox47 crwilcox @chriswilcox47 We fail so we can learn.

  32. crwilcox @chriswilcox47 crwilcox @chriswilcox47 “If you've never missed a flight,

    you're probably spending too much time in airports“ -George Stigler
  33. crwilcox @chriswilcox47 Blame isn’t a deterrent. It isn’t anything useful.

  34. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Excusing failure as human error stops

    the conversation.
  35. crwilcox @chriswilcox47 Why we don’t blame; an example.

  36. crwilcox @chriswilcox47

  37. crwilcox @chriswilcox47 Building a blameless post mortem culture.

  38. crwilcox @chriswilcox47 Create psychological safety.

  39. crwilcox @chriswilcox47 Find positive examples. https://landing.google.com/sre/books

  40. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Outages happen and no one wanted

    them to.
  41. crwilcox @chriswilcox47 A Test for Blamelessness 1. A script was

    executed that deleted the production database. 2. An engineer executed a script that deleted the production database without confirmation. 3. Chris executed a script that deleted the production database without confirmation.
  42. crwilcox @chriswilcox47 crwilcox @chriswilcox47 “But Chris, we never have outages

    at my work, we are perfect, in every way” -Someone, Probably.
  43. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Raise your own bar, Break stuff.

  44. crwilcox @chriswilcox47 crwilcox @chriswilcox47 Add some 9’s to that SLA!

  45. crwilcox @chriswilcox47 Thanks! References. https://crwilcox.com https://www.ntsb.gov/investigations/AccidentReports/Reports/MAB1917.pdf https://acphospitalist.org/archives/2018/07/the-new-improved-m-and-m.htm https://insights.ovid.com/pubmed?pmid=26983075&_ga=2.103365704.971425802.15638 99461-1002013713.1563899459 https://www.ama-assn.org/residents-students/residency/presenting-your-first-mm

    -conference-5-things-you-need-know https://landing.google.com/sre/books https://cloud.google.com/blog/products/gcp/fearless-shared-postmortems-cre-lif e-lessons https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/