Tweet
Tweet

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Automating Chaos Engineering Gamedays with Terraform Yury Niño Roa DevOps Engineer & Chaos Engineering Advocate ADL Digital Labs Copyright © 2019 HashiCorp

Slide 5

Slide 5 text

Don’t point at people, it is rude! Although our parents teach us not to do it, we finger point to others when something went wrong! Who is to blame?

Slide 6

Slide 6 text

Finger-pointing is self-preservation

Slide 7

Slide 7 text

How to avoid a blame culture in your teams?

Slide 8

Slide 8 text

Agenda ● Chaos Engineering. ● Chaos Principles. ● Chaos Culture with Gamedays. ● Planning a manual Gameday is hard! ● Automate Chaos Gamedays with Terraform! ● Terraform Modules.

Slide 9

Slide 9 text

Chaos Engineering Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production. https://principlesofchaos.org

Slide 10

Slide 10 text

Chaos Engineering Principles Injecting failure to achieve resilience! Hypothesize about Steady State Run Experiments Vary Real-World Events Automate Experiments

Slide 11

Slide 11 text

Chaos Culture Chaos Maturity Model Taken from Chaos Engineering Book

Slide 12

Slide 12 text

How to achieve Sophistication and Adoption?

Slide 13

Slide 13 text

Organization is sophisticated when ... Setup, analysis and termination are automated. ● Experiments run in production. ● Setup and analysis are automated. ● Continuous experimentation. ● Experimentation business metrics. ● Results are tracked over time. ● Tooling to support the chaos practice..

Slide 14

Slide 14 text

The organization is in adoption when ... Experimental verifications are performed in GameDays ● A team is dedicated to Chaos Engineering. ● Regression experiments. ● Most services practice chaos experimentation. ● Verifications are performed in GameDays.

Slide 15

Slide 15 text

Running Chaos GameDays!

Slide 16

Slide 16 text

Interactive, real-world and learning exercises. They are designed to give players a chance to put their skills in a technology to test. GameDays were created by Jesse Robbins inspired by his experience & training as a firefighter. GameDays are ...

Slide 17

Slide 17 text

Chaos GameDays Chaos Engineering promote Chaos Days! A Chaos GameDay is an event hosted to conduct chaos experiments against components of your system to validate or invalidate a hypothesis about a system’s resilience. Chaos

Slide 18

Slide 18 text

Chaos GameDays Chaos Engineering promote Chaos Days! First on Call member sees, triages, and tries to mitigate whatever failure the MoD has caused. Master of Disaster Decides the failure and declares start of incident and attack!!! Team will find and solve the issue in less than 75% of the allocated time. Finally they write up a Postmortem! Inspired in the James Burns’s work

Slide 19

Slide 19 text

How to plan a Chaos GameDay?

Slide 20

Slide 20 text

Planning and Running a GameDay 1. Pick a Hypothesis. 2. Pick a Style of GameDay. 3. Decide Who Participates. 4. Decide Who Observes. 5. Decide Where. 6. Decide When and For How Long. 7. Describe Your GameDay Experiment. 8. Get Approval. 9. Running the GameDay! 10. Consider a Safety Monitor.

Slide 21

Slide 21 text

Planning a GameDay is a lot of work!

Slide 22

Slide 22 text

Manual provisioning involve ...

Slide 23

Slide 23 text

Automate with Terraform ...

Slide 24

Slide 24 text

Terraform Infrastructure as Code Terraform is a tool to Build, Change, and Version Control your infrastructure. Automation Is the most powerful tool that the developers we have! Under the hood ... Terraform makes API calls on your behalf to one or more providers, such as AWs, Azure, Google Cloud, Datadog and Chaos Agents!

Slide 25

Slide 25 text

Terraform Infrastructure as Code Terraform is a tool to Build, Change, and Version Control your infrastructure. Automation Is the most powerful tool that the developers we have! Under the hood ... Terraform can do API calls to any technology that allow to CRUD resources through an API.

Slide 26

Slide 26 text

Automate Chaos GameDays with Terraform

Slide 27

Slide 27 text

Provision with Terraform

Slide 28

Slide 28 text

Provision with Terraform

Slide 29

Slide 29 text

Provision with Terraform

Slide 30

Slide 30 text

Provision Chaos with Terraform

Slide 31

Slide 31 text

for each

Slide 32

Slide 32 text

Modules are reusable Terraform configurations that can be called and configured by other configurations. Most modules manage a few closely related resources from a single provider. Modules can be used to abstract the logic associated to the use case for Terraform. Terraform Modules 

Slide 33

Slide 33 text

Terraform Modules

Slide 34

Slide 34 text

Terraform Modules for Chaos TERMINAL

Slide 35

Slide 35 text

Changing the chaos provider TERMINAL

Slide 36

Slide 36 text

Terraform Registry

Slide 37

Slide 37 text

Terraform Registry

Slide 38

Slide 38 text

Run Chaos GameDays! Everything fails all the time! Werner Vogels.

Slide 39

Slide 39 text

Thank You! 39 @yurynino