Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chaos Engineering, Community, and AWS

Chaos Engineering, Community, and AWS

E6c942c0f8e6042fbd47fcd3c4113b90?s=128

Adrian Hornsby

February 08, 2020
Tweet

Transcript

  1. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Chaos Engineering, Community and AWS Adrian Hornsby Principal Technical Evangelist Amazon Web Services
  2. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A

    scientific method
  3. None
  4. None
  5. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE Chaos Engineering A

    scientific method
  6. What if…? “What if this load balancer breaks?” “What if

    Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the master database stops?” Make it everyone’s problem!
  7. Failure injection • Start small & build confidence • Application

    level (exceptions, errors, …) • Host level (services, processes, …) • Resource attacks (CPU, memory, IO, …) • Network attacks (dependencies, latency, packet loss…) • AZ attack • Region attack • People attack
  8. Postmortems – COE (Correction of Errors) • What happened? •

    What was the impact on customers and your business? • What were the contributing factors? • What data do you have to support this? • especially metrics and graphs • What lessons did you learn? • What corrective actions are you taking? • Actions items • Related items (trouble tickets etc.)
  9. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  10. None
  11. Start simple and local!! $ docker stop 94a214bbeebd

  12. DDoS yourself $ wrk -t12 -c400 -d30s http://127.0.0.1/api/health

  13. Injecting Chaos to Amazon EC2 using AWS System Manager https://medium.com/@adhorn/injecting-chaos-to-amazon-ec2-using-amazon-system-manager-ca95ee7878f5

  14. https://github.com/adhorn/chaos-ssm-documents

  15. Injecting Chaos to AWS Lambda $ pip install chaos-lambda https://github.com/adhorn/aws-lambda-chaos-injection

  16. https://github.com/adhorn/aws-chaos-scripts

  17. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  18. Participate Be kind Join Community

  19. Thank you! © 2020, Amazon Web Services, Inc. or its

    affiliates. All rights reserved. Adrian Hornsby https://medium.com/@adhorn adhorn