Postmortem Culture: Learning from Failure

Practicing Chaos Engineering and reproducing outages have taught us that the culture of postmortems must be open and blameless. That is difficult, in part, due to the social stigma associated with publicly acknowledging the contributions of persons to outages.

And although the scenarios simulated in a gameday are entirely realistic, it's hard to write-up postmortems that resume all events, hint human factors, recognize there is not a root cause and provide action items.

In Aval Digital Labs, we are implementing a toolbox that automates the steps involved in chaos game days and generates postmortems using available in the market.

Transcript

1. Postmortem Culture Learning from Failure
2. YURY NIÑO DevOps Engineer Chaos Engineering Advocate
3. Garagoa is a town located in the Boyacá Department in Colombia. Each December 16th people in Garagoa celebrate the end of the year with a postmortem ceremony called the dead of sadness.
4. 60% No 18 Yes 27 No Have you written a Postmortem? 40% Yes A survey of 45 Software Engineers, showed that Postmortems are not a common practice.
5. • About postmortems. • Why don’t we write postmortems? • Blameful culture. • Chaos Engineering. • Chaos Gamedays. • Automating Gamedays & Postmortems. • Gaveta by Digital Labs. Agenda
6. What went wrong, and how do we learn from it? A postmortem is an artifact with a detailed description of exactly what went wrong in an incident. A postmortem is a written record of an incident, its impact, the actions taken to mitigate it, the root cause, and the follow-up actions to prevent the incident.
8. If the Postmortems are a best practice, why don’t we do it?
9. Because
11. How to change a blameful culture?
12. Chaos Engineering It is the discipline of experimenting failures in production in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
13. Chaos GameDays They are events to conduct chaos experiments against a system to validate or invalidate hypothesis about a system’s resiliency. They are an ideal way to ease into Chaos Engineering. Brian Lee, Jason Doffing
14. Chaos Gamedays Master of Disaster declares start of incident and attack!!! First On-Call member sees, triages, and tries to mitigate the impact. Team understands, analyzes and solves. Postmortem
15. What does it mean in the practice?
16. Planning a Gameday • Create an agenda. • Define users & roles. • Send communications. • Design an experiment. • Provision: ◦ HW/SW ◦ Chaos attackers ◦ Observability
17. Planning a Gameday
18. She is Gaveta!
19. Gaveta uses a hexagonal architecture
20. Gaveta uses a hexagonal architecture
22. She is Gaveta!
23. The best way to promote a postmortem culture is adopting a new view, a view focused in the syntoms, no in the causes ...
24. Thanks for coming!!! @yurynino