Tweet
Tweet

Slide 1

Slide 1 text

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chaos Engineering, Community and AWS Adrian Hornsby Principal Technical Evangelist Amazon Web Services

Slide 2

Slide 2 text

STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A scientific method

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A scientific method

Slide 6

Slide 6 text

What if…? “What if this load balancer breaks?” “What if Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the master database stops?” Make it everyone’s problem!

Slide 7

Slide 7 text

Failure injection • Start small & build confidence • Application level (exceptions, errors, …) • Host level (services, processes, …) • Resource attacks (CPU, memory, IO, …) • Network attacks (dependencies, latency, packet loss…) • AZ attack • Region attack • People attack

Slide 8

Slide 8 text

Postmortems – COE (Correction of Errors) • What happened? • What was the impact on customers and your business? • What were the contributing factors? • What data do you have to support this? • especially metrics and graphs • What lessons did you learn? • What corrective actions are you taking? • Actions items • Related items (trouble tickets etc.)

Slide 9

Slide 9 text

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Start simple and local!! $ docker stop 94a214bbeebd

Slide 12

Slide 12 text

DDoS yourself $ wrk -t12 -c400 -d30s http://127.0.0.1/api/health

Slide 13

Slide 13 text

Injecting Chaos to Amazon EC2 using AWS System Manager https://medium.com/@adhorn/injecting-chaos-to-amazon-ec2-using-amazon-system-manager-ca95ee7878f5

Slide 14

Slide 14 text

https://github.com/adhorn/chaos-ssm-documents

Slide 15

Slide 15 text

Injecting Chaos to AWS Lambda $ pip install chaos-lambda https://github.com/adhorn/aws-lambda-chaos-injection

Slide 16

Slide 16 text

https://github.com/adhorn/aws-chaos-scripts

Slide 17

Slide 17 text

© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

Slide 18

Slide 18 text

Participate Be kind Join Community

Slide 19

Slide 19 text

Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adrian Hornsby https://medium.com/@adhorn adhorn