Chaos Engineering in 5 Minutes

A very short introduction to Chaos Engineering. Presented in March 2018 at HolidayCheck's Lightning Talks.

Transcript

1. Chaos Engineering in 5 Minutes Pavlos Ratis (@dastergon)
2. N-Tier Architecture Credits: https://appybistro.com/2015/07/27/salad-lovers-multi-layered-creamy-salad/
3. CC BY-SA 3.0 - Sylvain Pedneault - Self-photographed Outage
4. ¯\_(ツ)_/¯
5. Public Domain - https://en.wikipedia.org/wiki/Montparnasse_derailment Failure is inevitable
6. Systems are fragile Courtesy of SpaceX
7. How to make them more reliable?
8. Testing
9. – Edsger W. Dijkstra “Testing shows the presence, not the absence of bugs” CC BY-SA 3.0 - https://en.wikiquote.org/wiki/Edsger_W._Dijkstra#Quotes_about_Dijkstra
10. Credits: http://corgibytes.com/blog/2016/03/28/pyramid-of-tests/
11. BUT
12. We cannot anticipate all failures https://www.theguardian.com/technology/2014/aug/14/google-undersea-fibre-optic-cables-shark-attacks https://www.theregister.co.uk/2017/03/01/aws_s3_outage/
13. Resiliency (noun) - re·sil·ien·cy • The ability to become strong, healthy, or successful again after something bad happens •The ability of something to return to its original shape after it has been pulled, stretched, pressed, bent, etc.
14. – Hsueh, M.C., Tsai, T.K. and Iyer, R.K., 1997. Fault injection techniques and tools. Computer, 30(4), pp.75-82. “Fault injection is important to evaluating the dependability of computer systems.” Fault Injection Public Domain - https://en.wikipedia.org/wiki/NASA
15. This is NOT Rocket Science!
16. ENTER CHAOS ENGINEERING
17. – Principles of Chaos Engineering (http://principlesofchaos.org/) “Chaos Engineering is the discipline of experimenting on a distributed system
 in order to build confidence in the system’s capability
 to withstand turbulent conditions in production.”
18. Chaos as in… • Killing random cloud VM Instances or containers • Killing random Kubernetes pods • Killing MySQL Master or Slaves • Introducing extra latency or packet loss between micro- services • Killing a critical supporting service (i.e logging server) while serving traffic • “Unplugging” a whole datacenter or availability zone
19. Process • Define a steady state hypothesis (i.e HTTP 2xx & 3xx) • Run experiments in production environment (i.e. Destroy a critical service) * • Try to disprove the hypothesis * Minimize the Blast Radius
20. Courtesy of SpaceX Confidence
21. Credits: https://www.netflix.com
22. The Tools https://github.com/Netflix/SimianArmy https://github.com/Netflix/chaosmonkey https://www.gremlin.com/
23. Summary • Uncovers the weaknesses of your system. • Builds confidence in your infrastructure. Chaos Engineering…
24. More Resources (Shameless plug) : Github: dastergon/awesome-chaos-engineering Thank you!