Improve resiliency and performance with controlled chaos engineering - Stockholm Chaos & Reliability Engineering June 8 2021

Slide 1

Slide 1 text

Slide 2

Slide 2 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Agenda • Challenges with distributed systems • Why is chaos engineering hard? • Introducing AWS Fault Injection Simulator (FIS) • Key features • Use cases • Multiple demos along the way

Slide 3

Slide 3 text

Slide 4

Slide 4 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Distributed systems are complex Message Message Reply Reply Server Network Client https://aws.amazon.com/builders-library/challenges-with-distributed-systems/

Slide 5

Slide 5 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Traditional testing is not enough TESTING = VERIFYING A KNOWN CONDITION Unit testing of components Tested in isolation to ensure function meets expectations Functional testing of integrations Each execution path tested to assure expected results

Slide 6

Slide 6 text

© 2021, Amazon Web Services, Inc. or its Affiliates. And it can get more complicated… IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device close failed in file object destructor: IOError: No space left on device logfile ROTATE logfile.0 ROTATE logfile.1 ROTATE logfile.2 ROTATE logfile.3 ROTATE logfile.n ROTATE

Slide 7

Slide 7 text

© 2021, Amazon Web Services, Inc. or its Affiliates. S O I T R E S S B S E R V E M P R O V E Chaos engineering Improve resilience and performance Uncover hidden issues Expose blind spots Monitoring, observability, and alarm And more

Slide 8

Slide 8 text

Slide 9

Slide 9 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Why is chaos engineering difficult? Difficult to ensure safety Stitch together different tools and homemade scripts 1 Agents or libraries required to get started 3 2 Difficult to reproduce “real-world” events (multiple failures at once) 4

Slide 10

Slide 10 text

Slide 11

Slide 11 text

© 2021, Amazon Web Services, Inc. or its Affiliates. No need to integrate multiple tools and homemade scripts or install agents Use the AWS Management Console or the AWS CLI Use pre-existing experiment templates and get started in minutes Easily share it with others Easy to get started

Slide 12

Slide 12 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Real-world conditions Run experiments in sequence of events or in parallel Target all levels of the system (host, infrastructure, network, etc.) Real faults injected at the service control plane level!

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Actions are the fault injection actions executed during an experiment aws:: Actions include: • Fault type • Targeted resources • Timing relative to any other actions • Fault-specific parameters, such as duration, rollback behavior, or the portion of requests to throttle Actions

Slide 16

Slide 16 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Actions "actions": { "StopInstances": { "actionId": "aws:ec2:stop-instances", "parameters": { "startInstancesAfterDuration": "PT2M" }, "targets": { "Instances": "RandomInstancesInAZ" } }, ”Wait": { "actionId": " aws:fis:wait", "parameters": { "duration": "PT1M", }, "startAfter": [ "StopInstances" ] }, }

Slide 17

Slide 17 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Targets define one or more AWS resources on which to carry out an action Targets include: • Resource type • Resource IDs, tags, and filters • Selection mode (e.g., ALL, RANDOM) Targets

Slide 18

Slide 18 text

© 2021, Amazon Web Services, Inc. or its Affiliates. "targets": { ”RandomInstancesInAZ": { "resourceType": "aws:ec2:instance", "resourceTags": { ”Env": ”test" }, "filters" : [ { "path": "Placement.AvailabilityZone", "values": ["us.east.1a"] }, { "path": "State.Name", "values": ["running"] }, { "path": "VpcId", "values": ["vpc-0123456789"] } ] "selectionMode": ”COUNT(2)" } Targets

Slide 19

Slide 19 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Experiment templates define an experiment and are used in the start-experiment request Experiment templates include: • Actions • Targets • Stop condition alarms • IAM role • Description • Tags Experiment templates

Slide 20

Slide 20 text

© 2021, Amazon Web Services, Inc. or its Affiliates. { "tags”: { "Name": "StopAndRestartRandomInstance" }, "description": "Stop and Restart One Random Instance", "roleArn": "arn:aws:iam::0123456789:role/MyFISExperimentRole”, "stopConditions": [ { "source": "aws:cloudwatch:alarm", "value": " "arn:aws:cloudwatch:us-east-1:0123456789:alarm:No_Traffic" } ], "targets": { "myInstance": { "resourceTags": { ”Env": ”test" }, "resourceType": "aws:ec2:instance", "selectionMode": ”COUNT(1)" } }, "actions": { "StopInstances": { "actionId": "aws:ec2:stop-instances", "description": "stop the instances", "parameters": { "startInstancesAtEnd": "true”, "duration": "PT2M", }, "targets": { "Instances": "myInstance" } } } } Experiment templates Description IAM role Stop conditions Targets Actions Name

Slide 21

Slide 21 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Experiment template A Stop conditions Targets Actions Action 1 Action 2 Amazon CloudWatch alarm i-aaaa i-bbbb i-cccc Specific EC2 instances Experiment template B Stop conditions Targets Actions Action 3 Action 1 Action 2 Amazon CloudWatch alarms All EC2 instances with “chaos-ready” tag Experiment templates

Slide 22

Slide 22 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Experiments are snapshot of the experiment template when it was first launched with couple additions Experiments include: • Snapshot of the experiment • Creation and start time • Status • Execution ID • Experiment template ID • IAM role ARN Experiments

Slide 23

Slide 23 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Server error (EC2) Stop, reboot, and terminate instance(s) (EC2) API throttling Increased memory or CPU load (EC2) Kill process (EC2) Latency injection (EC2) Container instance termination (ECS) Increase memory or CPU consumption per task (ECS) Terminate nodes (EKS) Database stop, reboot, and failover (RDS) And more to come in 2021 Supported fault injections

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

© 2021, Amazon Web Services, Inc. or its Affiliates. Resources AWS Well-Architected Framework https://aws.amazon.com/architecture/well-architected/ AWS Fault Injection Simulator https://aws.amazon.com/fis/ AWS FIS Documentation https://docs.aws.amazon.com/fis/ AWS FIS Samples https://github.com/aws-samples/aws-fault-injection-simulator-samples

Slide 36

Slide 36 text