DjangoCon 2019 - The Blameless Post Mortem: How Embracing Failure Makes Us Better

DjangoCon 2019 - The Blameless Post Mortem: How Embracing Failure Makes Us Better

5d95f89b9892e12c0d7fa2f671edac62?s=128

Christopher Wilcox

September 24, 2019
Tweet

Transcript

  1. 17.

    @chriswilcox47 https://chriswilcox.dev Morbidity & Mortality conferences - Provide medical education.

    - Improve quality of care. - Highlight cases with diagnostic uncertainty, complex management, etc. https://www.imperial.ac.uk
  2. 22.

    @chriswilcox47 https://chriswilcox.dev The structure of a post mortem. - Why

    did this happen? - Could it have been worse? Better? - How can we make sure it doesn’t happen again?
  3. 23.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation for the Google Cloud Python libraries was unavailable for users.
  4. 24.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. As part of repository cleanup, an engineer with write access to the development repository deleted the gh-pages branch.
  5. 25.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. During a two hour period, hosted reference documentation for our libraries was unavailable.
  6. 26.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. This was first detected via an external customer. Shortly after two internal teams also noticed and notified. The delay to the initial report was 30 minutes following the start of the outage.
  7. 27.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The documentation was available after republishing to gh-pages.
  8. 28.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. 2 hours. 2019-12-05 (all times PDT) 09:45 Branch gh-pages deleted during clean up of repository branches. 10:21 GitHub issue filed stating that docs are not available. 10:45 Team responds to issue. 10:55 Branch has been republished 11:45 GitHub is serving docs again.
  9. 29.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. We were notified fairly quick and the team monitors GitHub issues well enough that we knew about it pretty fast.
  10. 30.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. The branch should have been protected from deletion. GitHub page publishing is a bit opaque. While the branch was pushed within about 5 minutes of finding out it took an additional hour or so to get it actually serving the files. The lack of debugging information made this more difficult.
  11. 31.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. A member of the team had a local copy of the branch handy and was able to republish.
  12. 32.

    @chriswilcox47 https://chriswilcox.dev An example incident. - Summary. - Root Cause/Trigger.

    - User Impact. - Detection. - Resolution. - Duration/Timeline. - What Went Well? - What Went Poorly? - Where We Got Lucky? - Action Items. Protect gh-pages branch. Don’t allow deletion. Investigate if debugging gh-pages could be improved. Investigate if there are other technologies we ought to be using instead of gh-pages.
  13. 43.

    @chriswilcox47 https://chriswilcox.dev A Test for Blamelessness 1. A script was

    executed that deleted the production database. 2. An engineer executed a script that deleted the production database without confirmation. 3. Chris executed a script that deleted the production database without confirmation.
  14. 45.