Data Driven Postmortems

Those who fail to learn from history are doomed to repeat it. When things go wrong, and our services are impacted, we need to tell the story of our failures so that we can grow and learn as team. Postmortems offer us an opportunity to share this knowledge, so that we can build on our successes and avoid repeating our mistakes in the future. In this session we will discuss how Datadog runs our internal postmortems from data collection to building timelines to the blameless review. Attendees will leave with a framework they can apply right away to make postmortems more impactful in their own organizations.

For more information about Datadog, visit: https://www.datadoghq.com/

Transcript

1. Data-driven Postmortems Matt Williams
 Evangelist @ Datadog 
 @technovangelist mattw@datadoghq.com
2. @datadoghq @technovangelist #nginx #nginxconf • SaaS based infrastructure and app monitoring • Open Source Agent • Time series data (metrics and events) • Processing about a trillion data points per day • Intelligent Dashboards & Alerting • We’re hiring! (www.datadoghq.com/careers/) 2
3. @datadoghq @technovangelist #nginx #nginxconf “The problems we work on at Datadog are hard and often don't have obvious, clean-cut solutions, so it's useful to cultivate your troubleshooting skills, no matter what role you work in.” 3 Internal Datadog Dev Guide
4. @datadoghq @technovangelist #nginx #nginxconf 4 “The only real mistake is the one from which we learn nothing.” Henry Ford
5. @datadoghq @technovangelist #nginx #nginxconf Oxford English Dictionary “An analysis or discussion of an event held soon after it has occurred, especially in order to determine why it was a failure.” 5 Postmortem
6. @datadoghq @technovangelist #nginx #nginxconf 6 What is DevOps? • Culture • Automation • Metrics • Sharing
7. @datadoghq @technovangelist #nginx #nginxconf 7 Title Text
8. @datadoghq @technovangelist #nginx #nginxconf 8 Title Text
9. @datadoghq @technovangelist #nginx #nginxconf 9 Our Focus Area • Culture • Sharing
10. #nginx #nginxconf 10 Blameless Postmortems Instead of naming, blaming, and shaming, our goal should always be to maximize opportunities for organizational learning - DevOps Handbook, Kim, Humble, Debois, Willis 2016
11. #nginx #nginxconf 11
12. @datadoghq @technovangelist #nginx #nginxconf • Blameless Postmortems by John Allspaw http://bit.ly/etsy-blameless • The Human Side of Postmortems by Dave Zwieback http://bit.ly/human-postmortem 12 Blameless Postmortem Resources
13. Metrics Culture and Sharing are great, but what about
14. @datadoghq @technovangelist #nginx #nginxconf 14
15. @datadoghq @technovangelist #nginx #nginxconf 15 Follow @honest_update on Twitter
16. Collecting data is cheap; not having it when you need it can be expensive So Instrument All The Things!
17. @datadoghq @technovangelist #nginx #nginxconf 17
18. @datadoghq @technovangelist #nginx #nginxconf 18
19. @datadoghq @technovangelist #nginx #nginxconf 19
20. @datadoghq @technovangelist #nginx #nginxconf 20
21. @datadoghq @technovangelist #nginx #nginxconf • Well-understood • Granular • Tagged by scope • Long-lived 21 4 Qualities of Good Metrics
22. @datadoghq @technovangelist #nginx #nginxconf 22 Recurse until you find the root cause
23. Who What When of postmortems
24. @datadoghq @technovangelist #nginx #nginxconf 24 If you’re still responding to the incident, it’s not time for a postmortem
25. @datadoghq @technovangelist #nginx #nginxconf • Everyone! • Responders • Identifiers • Affected Users 25 Data Collection: Who?
26. @datadoghq @technovangelist #nginx #nginxconf 26 • Their perspective • What they did • What they thought • Why they thought/did it Data Collection: What?
27. @datadoghq @technovangelist #nginx #nginxconf – John Allspaw Behind every seemingly technical problem is actually a human problem waiting to be found. 27
28. @datadoghq @technovangelist #nginx #nginxconf – Dr Sidney Dekker Human error is not our cause of troubles; instead human error is a consequence of the design of the tools that we gave them. 28
29. @datadoghq @technovangelist #nginx #nginxconf 29
30. @datadoghq @technovangelist #nginx #nginxconf • … we will be dramatically improving the tooling that humans (and systems) interact with such that input validation is much more strict and will not allow for all servers, and control plane servers to be rebooted simultaneously … 30 Joyent US-EAST-1 Postmortem 2014 http://bit.ly/joyent-post
31. @datadoghq @technovangelist #nginx #nginxconf – Richard Guindon “Writing is nature’s way of letting you know how sloppy your thinking is.” 31
32. @datadoghq @technovangelist #nginx #nginxconf – Chinese Proverb “One Picture is worth Ten thousand Words” 32
33. @datadoghq @technovangelist #nginx #nginxconf • As soon as possible. • Memory drops sharply within 20 minutes • Susceptibility to “false memory” increases 33 Data Collection: When?
34. @datadoghq @technovangelist #nginx #nginxconf • Stress • Sleep deprivation • Burnout 34 Data Skew/Corruption
35. @datadoghq @technovangelist #nginx #nginxconf • Blame/Fear of punitive action • Bias • Anchoring • Hindsight • Outcome • Availability • Recency 35 Data Skew/Corruption
36. @datadoghq @technovangelist #nginx #nginxconf 36 How we do Postmortems At Datadog
37. @datadoghq @technovangelist #nginx #nginxconf • Postmortems emailed to company wide • Scheduled recurring postmortem meetings 37 A Few Notes
38. @datadoghq @technovangelist #nginx #nginxconf • A real event - March 8, 2016 • Public-facing web of Datadog • Higher 5xx % on outbound requests (pull data) • No change on inbound requests or alerts (push data) • First navigations gets slow... • Then we get frequent “Down” pages • Alerts on 5xx % 38 About The Example
39. @datadoghq @technovangelist #nginx #nginxconf • Describe what happened here at a high-level -- think of it as an abstract in a scientific paper. • What was the impact on customers? • What was the severity of the outage? • What components were affected? • What ultimately resolved the outage? 39 Summary: What Happened?
40. @datadoghq @technovangelist #nginx #nginxconf 40
41. @datadoghq @technovangelist #nginx #nginxconf 41
42. @datadoghq @technovangelist #nginx #nginxconf • We want to make sure we detected the issue early and would catch the same issue if it were to repeat. • Did we have a metric that showed the outage? • Was there a monitor on that metric? • How long did it take for us to declare an outage? 42 How was the Outage Detected?
43. @datadoghq @technovangelist #nginx #nginxconf 43
44. @datadoghq @technovangelist #nginx #nginxconf 44
45. @datadoghq @technovangelist #nginx #nginxconf • Who was the incident owner & who else was involved? • Slack archive links and timeline of events! • What went well? • What didn’t go so well? 45 How Did We Respond?
46. @datadoghq @technovangelist #nginx #nginxconf 46 *Names changed
47. @datadoghq @technovangelist #nginx #nginxconf 47 ChatOps Archives FTW! *Names changed
48. @datadoghq @technovangelist #nginx #nginxconf 48 *Names changed Track Learnings As You Go
49. @datadoghq @technovangelist #nginx #nginxconf • Deep dive into the cause • Examples from this incident: • http://bit.ly/dd-statuspage • http://bit.ly/alq-postmortem 49 Why Did It Happen?
50. @datadoghq @technovangelist #nginx #nginxconf • Link to Github issues and Trello cards • Now? • Next? • Later? • Follow up notes 50 How Do We Prevent It In The Future?
51. @datadoghq @technovangelist #nginx #nginxconf 51
52. @datadoghq @technovangelist #nginx #nginxconf • What happened (summary)? • How did we detect it? • How did we respond? • Why did it happen (deep dive)? • Actionable next steps! 52 Recap:
53. @datadoghq @technovangelist #nginx #nginxconf • The Infinite Hows - John Allspaw
 http://bit.ly/infinite-hows • “Blameless” Postmortems don’t work - J Paul Reed
 http://bit.ly/blameless-dont-work • Monitoring 101 - Alexis Lê-Quôc
 http://dtdg.co/monitoring-101-data 53 More Resources
54. Thank You @datadoghq @technovangelist #nginx #nginxconf 54 linkedin.com/in/technovangelist @technovangelist