Chaos Engineering Considering Failures from Development

Transcript

1. Chaos Engineering Considering Failures from Development
2. Site Reliability Engineer Chaos Engineering Advocate @yurynino https://www.yurynino.dev YURY NIÑO ROA
3. ▪ Chaos Engineering ▪ Software Development Lifecycle ▪ Designing for Chaos ▪ Coding for Chaos ▪ Deploying for Chaos ▪ Operating with Chaos
4. Chaos Engineering It is the discipline of experimenting failures in production in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
5. Principles Hypothesize about Steady State Run Experiments Vary Real-World Events Automate Experiments
6. 2008 Chaos Engineering was born at Netflix 2010 Chaos Monkey & Simian Army were launched 2016 Gremlin was born 2019 Chaos Massification 2017 SRE USenix Chaos IQ ChaosConf 2018 Book Chaos Eng 2020 Book Chaos Eng History
7. Chaos Monkey Chaos Toolkit Gremlin Chaos Mesh Chaos for Spring Boot Chaos Tools
8. How did we build software?
9. With Software Development Life Cycles Plan Requirements Design Code Deploy Maintain
10. With Software Development Life Cycles Deploy Maintain Plan Requirements Design Code Deploy Maintain
11. Designing for Chaos
13. Designing Requirements focus on general attributes rather than specific behaviors. How to guarantee resilience? It’s not easy to design and build Nonfunctional Requirements, since they are primarily emergent properties! Design to fail
14. Principles Design • Design for Least Privilege • Design for Understandability • Design for Changing Landscape • Design for Resilience • Design for Recovery Where are failures?
15. Principles Design • Design for Least Privilege • Design for Understandability • Design for Changing Landscape • Design for Chaos • Design for Resilience • Design for Recovery
16. By anticipating mistakes like failing, SDLC can eliminate redundant rework and after-the-fact fixes.
17. Coding for Chaos
19. Google reported that 85% of all bugs in Android were caused by memory management errors. How to guarantee resilience? They concluded that “they need to move towards memory safe languages”. Code to fail
20. Principles Coding • Programming Language Choice • Complexity vs Understandability • Securing Third-Party Software • Testing Code • Data Validation Where is Resilience?
21. Principles Coding • Programming Language Choice • Complexity vs Understandability • Securing Third-Party Software • Functions based Chaos • Testing Code • Data Validation
22. Code will inevitably include bugs. You can avoid them using hardened frameworks to resilience.
23. Deploying for Chaos
25. Deploy to fail
26. Deployment Practices • Require Code Reviews • Rely on Automation • Verify Artifacts, Not Just People • Treat Configuration as Code • Securing Against the Threat Model • Policies Verifiable Builds • Post-Deployment Verification
27. Despite your best efforts, your code probably won’t always behave as expected.
28. Operating for Chaos
30. Complex systems can fail in both simple and complex ways, ranging from unexpected service outages to attacks by malicious actors to gain unauthorized access Netflix Twitter
31. Principles Operating • Define a disaster • Prepare a Disaster Planning • Identify Team and Roles • Establish a Team Charter • Establish Severity Models • Develop Response Plans • Create Detailed Playbooks It considers Resilience :)
32. Chaos Engineering Book 2020 Chaos Maturity Model
33. https://www.gremlin.com https://chaosengineering.slack.com https://github.com/dastergon/awesome-ch aos-engineering https://www.infoq.com/chaos-engineering
34. @yurynino