Postmortems for App Outages and Incidents

Systems fail. Apps crash. When something goes wrong with your app it can be seen as a major problem. It can also be seen as an important learning opportunity — one that can be challenging to get right. This talk lays out steps to learn from incidents and crashes via a general introduction to postmortems, guidance on what makes a good postmortem, and some high-level app-specific learnings from past postmortems.

Transcript

1. Postmortems for App Outages and Incidents Kris Arnold — @wka 13 May, 2019
3. 1. What is a postmortem?
4. 1. What is a postmortem? 2. Holding a good postmortem
5. 1. What is a postmortem? 2. Holding a good postmortem 3. App-specific postmortems
7. New feature
8. Enhancement
9. API change
10. Bug fix
14. !
16. 1. What is a postmortem?
17. Terms • Postmortem • Learning review • Incident debrief • After-action review
18. Learning
20. Explain Produce an explanation of the incident
21. Prevent and Improve Create recommendations to prevent future incidents. Detect and respond to similar events better in the future
22. Styles
23. Styles • Meeting
24. Styles • Meeting • Document
25. Styles • Meeting • Document • Wide audience
26. Styles • Meeting • Document • Wide audience • Limited audience
28. Requestor
29. Requestor Facilitator
30. Requestor Facilitator Record
31. Meeting
32. Meeting Timeline
33. Meeting Timeline Surprises
34. Meeting Timeline Surprises Remediation items
36. 2. Holding a good postmortem
37. Reprimand?
38. Learning
39. No one comes to work to do a bad job
40. Best decisions
41. Best decisions given the information available
42. Reconstruct the world
43. Ask “How?” instead of “Why?”
44. Ask “What surprised you?”
45. Don't focus only on remediation items
46. 3. App-specific postmortems
47. 3. App-specific postmortems • Themes
48. 3. App-specific postmortems • Themes • Recommendations
49. Themes
50. Distant changes affect apps
51. Long time before detection
52. App version upgrades
53. Push notifications
54. It's not you, it’s…
55. It's not you, it’s… • the OS
56. It's not you, it’s… • the OS • a third-party SDK
57. It's not you, it’s… • the OS • a third-party SDK • external modifications
58. Recommendations
59. Recommendations Gaining confidence
60. Instrumentation
61. EtsyGraphite.metric(withName: "guided_search").increment()
62. EtsyGraphite.metric(withName: "guided_search").increment()
64. ELKLogger.log(inNamespace: "Buyer App", error: "Failed…")
65. Timers let timer = EtsyGraphiteTimeTracker.startNew() //… EtsyGraphite.metric(withName: "load_time").time(timer)
66. Timers let timer = EtsyGraphiteTimeTracker.startNew() //… EtsyGraphite.metric(withName: "load_time").time(timer)
67. Timers let timer = EtsyGraphiteTimeTracker.startNew() //… EtsyGraphite.metric(withName: "load_time").time(timer)
68. Timers let timer = EtsyGraphiteTimeTracker.startNew() //… EtsyGraphite.metric(withName: "load_time").time(timer)
70. Deploy dashboard
71. Deploy dashboard • Overall errors
72. Deploy dashboard • Overall errors • CDN errors
73. Deploy dashboard • Overall errors • CDN errors • 404 errors (web)
74. Deploy dashboard • Overall errors • CDN errors • 404 errors (web) • Other HTTP errors (web)
75. Deploy dashboard • Overall errors • CDN errors • 404 errors (web) • Other HTTP errors (web) • HTTP errors (native apps)
76. Deploy dashboard • Overall errors • CDN errors • 404 errors (web) • Other HTTP errors (web) • HTTP errors (native apps) • iOS crashes
77. Deploy dashboard • Overall errors • CDN errors • 404 errors (web) • Other HTTP errors (web) • HTTP errors (native apps) • iOS crashes • Android crashes
78. Instrumentation
79. Feature flags
80. 'ExpressCheckout' => [ ENABLED => 50 ]
81. 'ExpressCheckout' => [ ENABLED => 1 ]
82. 'ExpressCheckout' => [ ENABLED => 5 ]
83. 'ExpressCheckout' => [ ENABLED => 5, MINIMUM_VERSION => 4.2 ]
84. Kill switches 'RefactoredSocialSignInKillSwitch' => [ ENABLED => 0 ]
85. Kill switches 'RefactoredSocialSignInKillSwitch' => [ ENABLED => 0 ]
86. if analyticsContext.isFlagEnabled("RefactoredSocialSignInKillSwitch") { oldSocialSignIn() } else { refactoredSocialSignIn() }
87. if analyticsContext.isFlagEnabled("RefactoredSocialSignInKillSwitch") { oldSocialSignIn() } else { refactoredSocialSignIn() }
88. if analyticsContext.isFlagEnabled("RefactoredSocialSignInKillSwitch") { oldSocialSignIn() } else { refactoredSocialSignIn() }
90. Further reading Blameless Postmortems and a Just Culture John Allspaw https://codeascraft.com/2012/05/22/ blameless-postmortems Etsy’s Debriefing Facilitation Guide for Blameless Postmortems John Allspaw, Morgan Evans, Daniel Schauenberg https://github.com/etsy/ DebriefingFacilitationGuide PagerDuty Postmortem Best Practice Documentation Rachael Byrne https://postmortems.pagerduty.com The Field Guide to Understanding ‘Human Error’ Sidney Dekker Counterfactual Thinking, Rules, and The Knight Capital Accident John Allspaw https://kitchensoap.com/2013/10/29/ counterfactuals-knight-capital How Did Things Go Right? Learning More from Incidents Ryan Kitchens https://www.youtube.com/watch? v=5-2jMlsPqc8
91. Thanks Kris Arnold @wka