Chaos Engineering, Community, and AWS

Transcript

1. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chaos Engineering, Community and AWS Adrian Hornsby Principal Technical Evangelist Amazon Web Services
2. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A scientific method
5. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A scientific method
6. What if…? “What if this load balancer breaks?” “What if Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the master database stops?” Make it everyone’s problem!
7. Failure injection • Start small & build confidence • Application level (exceptions, errors, …) • Host level (services, processes, …) • Resource attacks (CPU, memory, IO, …) • Network attacks (dependencies, latency, packet loss…) • AZ attack • Region attack • People attack
8. Postmortems – COE (Correction of Errors) • What happened? • What was the impact on customers and your business? • What were the contributing factors? • What data do you have to support this? • especially metrics and graphs • What lessons did you learn? • What corrective actions are you taking? • Actions items • Related items (trouble tickets etc.)
9. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
11. Start simple and local!! $ docker stop 94a214bbeebd
12. DDoS yourself $ wrk -t12 -c400 -d30s http://127.0.0.1/api/health
13. Injecting Chaos to Amazon EC2 using AWS System Manager https://medium.com/@adhorn/injecting-chaos-to-amazon-ec2-using-amazon-system-manager-ca95ee7878f5
14. https://github.com/adhorn/chaos-ssm-documents
15. Injecting Chaos to AWS Lambda $ pip install chaos-lambda https://github.com/adhorn/aws-lambda-chaos-injection
16. https://github.com/adhorn/aws-chaos-scripts
17. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
18. Participate Be kind Join Community
19. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adrian Hornsby https://medium.com/@adhorn adhorn