Improve resiliency and performance with controlled chaos engineering

Presented at I T.A.K.E. Unconference, May 27th, 2021.

@gunnargrosch
LinkedIn

While the benefits of distributed computing are undeniable and have revolutionized IT, they do have some challenges. Architectures have grown increasingly complex and can be hard to understand and difficult to debug and test. Chaos engineering is the process of stressing an application in testing or production environments by creating disruptive events, such as server outages or API throttling, observing how the system responds, and implementing improvements. In this session, we'll see how chaos engineering can help improve our application resiliency and performance, uncover hidden issues, expose blind spots, and more.

AWS Well-Architected Framework
https://aws.amazon.com/architecture/well-architected/

AWS Fault Injection Simulator
https://aws.amazon.com/fis/

AWS FIS Documentation
https://docs.aws.amazon.com/fis/

AWS FIS Samples
https://github.com/aws-samples/aws-fault-injection-simulator-samples

Transcript

1. © 2021, Amazon Web Services, Inc. or its Affiliates. Gunnar Grosch @gunnargrosch Improve resiliency and performance with controlled chaos engineering
2. © 2021, Amazon Web Services, Inc. or its Affiliates. Agenda • Challenges with distributed systems • Why is chaos engineering hard? • Introducing AWS Fault Injection Simulator (FIS) • Key features and use cases • Automated chaos experiments • Multiple demos along the way
3. © 2021, Amazon Web Services, Inc. or its Affiliates. Challenges with distributed systems
4. © 2021, Amazon Web Services, Inc. or its Affiliates. Distributed systems are complex Message Message Reply Reply Server Network Client https://aws.amazon.com/builders-library/challenges-with-distributed-systems/
5. © 2021, Amazon Web Services, Inc. or its Affiliates. Traditional testing is not enough TESTING = VERIFYING A KNOWN CONDITION Unit testing of components Tested in isolation to ensure function meets expectations Functional testing of integrations Each execution path tested to assure expected results
6. © 2021, Amazon Web Services, Inc. or its Affiliates. And it can get more complicated… IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device logfile ROTATE logfile.0 ROTATE logfile.1 ROTATE logfile.2 ROTATE logfile.3 ROTATE logfile.n ROTATE
7. © 2021, Amazon Web Services, Inc. or its Affiliates. S O I T R E S S B S E R V E M P R O V E Chaos engineering Improve resilience and performance Uncover hidden issues Expose blind spots Monitoring, observability, and alarm And more
8. © 2021, Amazon Web Services, Inc. or its Affiliates. Phases of chaos engineering Steady state Hypothesis Run experiment Verify Improve
9. © 2021, Amazon Web Services, Inc. or its Affiliates. Why is chaos engineering difficult? Difficult to ensure safety Stitch together different tools and homemade scripts 1 Agents or libraries required to get started 3 2 Difficult to reproduce “real-world” events (multiple failures at once) 4
10. © 2021, Amazon Web Services, Inc. or its Affiliates. AWS Fault Injection Simulator Safeguards Real-world conditions Easy to get started Fully managed chaos engineering service
11. © 2021, Amazon Web Services, Inc. or its Affiliates. No need to integrate multiple tools and homemade scripts or install agents Use the AWS Management Console or the AWS CLI Use pre-existing experiment templates and get started in minutes Easily share it with others Easy to get started
12. © 2021, Amazon Web Services, Inc. or its Affiliates. Real-world conditions Run experiments in sequence of events or in parallel Target all levels of the system (host, infrastructure, network, etc.) Real faults injected at the service control plane level!
13. © 2021, Amazon Web Services, Inc. or its Affiliates. Safeguards “Stop conditions” alarms Integration with Amazon CloudWatch Built-in rollbacks Fine-grain IAM controls
14. © 2021, Amazon Web Services, Inc. or its Affiliates. Components Experiment templates Experiments Actions Targets
15. © 2021, Amazon Web Services, Inc. or its Affiliates. Actions are the fault injection actions executed during an experiment aws:<service-name>:<action-type> Actions include: • Fault type • Targeted resources • Timing relative to any other actions • Fault-specific parameters, such as duration, rollback behavior, or the portion of requests to throttle Actions
16. © 2021, Amazon Web Services, Inc. or its Affiliates. Targets define one or more AWS resources on which to carry out an action Targets include: • Resource type • Resource IDs, tags, and filters • Selection mode (e.g., ALL, RANDOM) Targets
17. © 2021, Amazon Web Services, Inc. or its Affiliates. Experiment templates define an experiment and are used in the start-experiment request Experiment templates include: • Actions • Targets • Stop condition alarms • IAM role • Description • Tags Experiment templates
18. © 2021, Amazon Web Services, Inc. or its Affiliates. Experiments are snapshot of the experiment template when it was first launched with couple additions Experiments include: • Snapshot of the experiment • Creation and start time • Status • Execution ID • Experiment template ID • IAM role ARN Experiments
19. © 2021, Amazon Web Services, Inc. or its Affiliates. Use cases
20. © 2021, Amazon Web Services, Inc. or its Affiliates. Use cases One-off experiments Periodic game days Automated experiments
21. © 2021, Amazon Web Services, Inc. or its Affiliates. Use cases One-off experiments Periodic game days Automated experiments
22. © 2021, Amazon Web Services, Inc. or its Affiliates. Use cases One-off experiments Periodic game days Automated experiments
23. © 2021, Amazon Web Services, Inc. or its Affiliates. Use cases One-off experiments Periodic game days Automated experiments
24. © 2021, Amazon Web Services, Inc. or its Affiliates. Automated experiments Recurring scheduled experiments Event-triggered experiments Continuous delivery experiments
25. © 2021, Amazon Web Services, Inc. or its Affiliates. Automated experiments Recurring scheduled experiments Event-triggered experiments Continuous delivery experiments
26. © 2021, Amazon Web Services, Inc. or its Affiliates. Automated experiments Recurring scheduled experiments Event-triggered experiments Continuous delivery experiments
27. © 2021, Amazon Web Services, Inc. or its Affiliates. Automated experiments Recurring scheduled experiments Event-triggered experiments Continuous delivery experiments
28. © 2021, Amazon Web Services, Inc. or its Affiliates. Automated experiments Recurring scheduled experiments Event-triggered experiments Continuous delivery experiments
29. © 2021, Amazon Web Services, Inc. or its Affiliates. Resources AWS Well-Architected Framework https://aws.amazon.com/architecture/well-architected/ AWS Fault Injection Simulator https://aws.amazon.com/fis/ AWS FIS Documentation https://docs.aws.amazon.com/fis/ AWS FIS Samples https://github.com/aws-samples/aws-fault-injection-simulator-samples
30. © 2021, Amazon Web Services, Inc. or its Affiliates. Thank you! Gunnar Grosch @gunnargrosch