Automating Chaos Engineering Gamedays with Terraform

Transcript

4. Automating Chaos Engineering Gamedays with Terraform Yury Niño Roa DevOps Engineer & Chaos Engineering Advocate ADL Digital Labs Copyright © 2019 HashiCorp
5. Don’t point at people, it is rude! Although our parents teach us not to do it, we finger point to others when something went wrong! Who is to blame?
6. Finger-pointing is self-preservation
7. How to avoid a blame culture in your teams?
8. Agenda • Chaos Engineering. • Chaos Principles. • Chaos Culture with Gamedays. • Planning a manual Gameday is hard! • Automate Chaos Gamedays with Terraform! • Terraform Modules.
9. Chaos Engineering Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org
10. Chaos Engineering Principles Injecting failure to achieve resilience! Hypothesize about Steady State Run Experiments Vary Real-World Events Automate Experiments
11. Chaos Culture Chaos Maturity Model Taken from Chaos Engineering Book
12. How to achieve Sophistication and Adoption?
13. Organization is sophisticated when ... Setup, analysis and termination are automated. • Experiments run in production. • Setup and analysis are automated. • Continuous experimentation. • Experimentation business metrics. • Results are tracked over time. • Tooling to support the chaos practice..
14. The organization is in adoption when ... Experimental verifications are performed in GameDays • A team is dedicated to Chaos Engineering. • Regression experiments. • Most services practice chaos experimentation. • Verifications are performed in GameDays.
15. Running Chaos GameDays!
16. Interactive, real-world and learning exercises. They are designed to give players a chance to put their skills in a technology to test. GameDays were created by Jesse Robbins inspired by his experience & training as a firefighter. GameDays are ...
17. Chaos GameDays Chaos Engineering promote Chaos Days! A Chaos GameDay is an event hosted to conduct chaos experiments against components of your system to validate or invalidate a hypothesis about a system’s resilience. Chaos
18. Chaos GameDays Chaos Engineering promote Chaos Days! First on Call member sees, triages, and tries to mitigate whatever failure the MoD has caused. Master of Disaster Decides the failure and declares start of incident and attack!!! Team will find and solve the issue in less than 75% of the allocated time. Finally they write up a Postmortem! Inspired in the James Burns’s work
19. How to plan a Chaos GameDay?
20. Planning and Running a GameDay 1. Pick a Hypothesis. 2. Pick a Style of GameDay. 3. Decide Who Participates. 4. Decide Who Observes. 5. Decide Where. 6. Decide When and For How Long. 7. Describe Your GameDay Experiment. 8. Get Approval. 9. Running the GameDay! 10. Consider a Safety Monitor.
21. Planning a GameDay is a lot of work!
22. Manual provisioning involve ...
23. Automate with Terraform ...
24. Terraform Infrastructure as Code Terraform is a tool to Build, Change, and Version Control your infrastructure. Automation Is the most powerful tool that the developers we have! Under the hood ... Terraform makes API calls on your behalf to one or more providers, such as AWs, Azure, Google Cloud, Datadog and Chaos Agents!
25. Terraform Infrastructure as Code Terraform is a tool to Build, Change, and Version Control your infrastructure. Automation Is the most powerful tool that the developers we have! Under the hood ... Terraform can do API calls to any technology that allow to CRUD resources through an API.
26. Automate Chaos GameDays with Terraform
27. Provision with Terraform
28. Provision with Terraform
29. Provision with Terraform
30. Provision Chaos with Terraform
31. for each
32. Modules are reusable Terraform configurations that can be called and configured by other configurations. Most modules manage a few closely related resources from a single provider. Modules can be used to abstract the logic associated to the use case for Terraform. Terraform Modules 
33. Terraform Modules
34. Terraform Modules for Chaos TERMINAL
35. Changing the chaos provider TERMINAL
36. Terraform Registry
37. Terraform Registry
38. Run Chaos GameDays! Everything fails all the time! Werner Vogels.
39. Thank You! 39 @yurynino