Creating Resiliency Through Destruction

Creating Resiliency Through Destruction

With the wide adoption of micro-services and large-scale distributed systems, architectures have grown increasingly complex and hard to understand. Worse, the software systems running them have become extremely difficult to debug and test, increasing the risk of outages. With these new challenges, new tools are required and since failures have become more and more chaotic in nature, we must turn to chaos engineering in order to reveal failures before they become outages. In this talk, we will deep dive into chaos engineering, a discipline that promotes breaking things on purpose in order to learn how to build more robust systems.

E6c942c0f8e6042fbd47fcd3c4113b90?s=128

Adrian Hornsby

May 08, 2019
Tweet

Transcript

  1. 1.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Creating Resiliency Through Destruction Adrian Hornsby Sr. Technical Evangelist Amazon Web Services adhorn
  2. 2.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Can you guess what will happen?
  3. 3.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Distributed Systems are hard Amazon Twitter Netflix
  4. 4.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Failures are a given and everything will eventually fail over time. Werner Vogels CTO – Amazon.com “ “
  5. 5.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Resiliency: Ability for a system to handle and eventually recover from unexpected conditions
  6. 6.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Resiliency at work
  7. 7.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T How do we build resilient software systems?
  8. 8.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T People Application Network & Data Infrastructure
  9. 9.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Building confidence through testing Unit testing of components: • Tested in isolation to ensure function meets expectations. Functional testing of integrations: • Each execution path tested to assure expected results. Is it enough???
  10. 10.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T GameDay at Amazon Creating Resiliency Through Destruction https://www.youtube.com/watch?v=zoz0ZjfrQ9s
  11. 11.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Chaos engineering https://github.com/Netflix/SimianArmy
  12. 12.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Failure injection • Start small & build confidence • Application level • Host failure • Resource attacks (CPU, memory, …) • Network attacks (dependencies, latency, …) • Region attack • Human attack
  13. 13.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Break your systems on purpose. Find out their weaknesses and fix them before they break when least expected.
  14. 14.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Chaos engineering is NOT about breaking things randomly without a purpose, chaos engineering is about breaking things in a controlled environment and through well- planned experiments in order to build confidence in your application to withstand turbulent conditions.
  15. 15.

    S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved. Chaos Engineering
  16. 16.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Steady State Hypothesis Design & Run Experiment Fix Build Resilient Systems Verify & Learn
  17. 17.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Build Resilient Systems
  18. 19.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Cascading and Overload Failures
  19. 22.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T https://aws.amazon.com/wellarchitected
  20. 23.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Steady State
  21. 24.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T What is steady state? • ”normal” behavior of your system https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero
  22. 25.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T What is steady state? • ”normal” behavior of your system • Business Metric https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a
  23. 26.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Business metrics at work Amazon: 100 ms of extra load time caused a 1% drop in sales (Greg Linden). Google: 500 ms of extra load time caused 20% fewer searches (Marissa Mayer). Yahoo!: 400 ms of extra load time caused a 5–9% increase in the number of people who clicked “back” before the page even loaded (Nicole Sullivan).
  24. 27.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Hypothesis
  25. 28.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T What if…? “What if this load balancer breaks?” “What if Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the database stops?” Make it everyone’s problem!
  26. 29.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Disclaimer! Don’t make an hypothesis that you know will break you!
  27. 30.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Design & Run Experiment
  28. 31.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Designing experiment • Pick hypothesis • Scope the experiment • Identify metrics • Notify the organization
  29. 32.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Rules of thumbs • Start with very small • As close as possible to production • Minimize the blast radius. • Have an emergency STOP! • Careful with state that can’t be rolled back (corrupt or incorrect data)
  30. 33.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Running Chaos Experiment Users Canary deployment Normal Version 99% Users 1% Users Start with ..
  31. 34.

    S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  32. 35.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Fault Injection Queries for Amazon Aurora SQL commands issued to simulate: • A crash of the master instance or an Aurora Replica • A failure of an Aurora Replica • A disk failure • Disk congestion https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.FaultInjectionQueries.html
  33. 36.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Fault Injection Queries for Amazon Aurora SQL commands issued to simulate: • A crash of the master instance or an Aurora Replica • A failure of an Aurora Replica • A disk failure • Disk congestion ALTER SYSTEM SIMULATE percentage_of_failure PERCENT DISK FAILURE [ IN DISK index | NODE index ] FOR INTERVAL quantity { YEAR | QUARTER | MONTH | WEEK | DAY | HOUR | MINUTE | SECOND };
  34. 37.

    S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved. How to DDoS yourself ~ wrk -t12 -c400 -d30s http://127.0.0.1/api/health
  35. 38.

    S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved. Come late to work ~ tc qdisc add dev eth0 root netem delay 200ms
  36. 39.

    https://github.com/Netflix/SimianArmy Set of scheduled agent: • shuts down services randomly

    • slows down performances • checks conformity • breaks an entire region • More …
  37. 40.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T The Chaos Toolkit • Simplifying Adoption of Chaos Engineering • An Open API to Chaos Engineering • Open source extensions for • Infrastructure/Platform Fault Injections • Application Fault Injections • Observability • Integrates easily into CI/CD pipelines
  38. 45.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T ToxiProxy • HTTP API • Build for Automated testing in mind • Not for production environment • Fast • Toxics for: • Timeouts, latency, connections and bandwidth limitation, etc.. • CLI • Stable and well tested (used for 3 years at Shopify) • Open Source: https://github.com/Shopify/toxiproxy
  39. 46.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T https://atscaleconference.com/videos/resiliency-testing-with-toxiproxy/
  40. 47.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T https://github.com/asobti/kube-monkey
  41. 48.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Pumba https://github.com/alexei-led/pumba/
  42. 49.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T https://blog.thundra.io/chaos-test-your-lambda-functions-with-thundra
  43. 50.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T https://medium.com/@adhorn/injecting-chaos-to-aws-lambda-functions-using-lambda-layers-2963f996e0ba Injecting Chaos to AWS Lambda functions using Lambda Layers
  44. 51.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Verify & Learn
  45. 52.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Quantifying the result of the experiment • Time to detect? • Time for notification? And escalation? • Time to public notification? • Time for graceful degradation to kick-in? • Time for self healing to happen? • Time to recovery – partial and full? • Time to all-clear and stable?
  46. 53.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T PostMortems – COE (Correction of Errors) The 5 WHYs Outage Because of … Because of … Because of … Because of …
  47. 54.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Rules to remember! There is no isolated ‘cause’ of an accident.
  48. 55.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T More questions to ask • Can you clarify if there were any preceding events? • Why would they believe acting in this way was the best course of action to deliver the desired outcome? • Is there another failure mode that could present here? • What decisions or events prior to this made this work before? • Why stop there – are there places to dig deeper that could shine a light more on this? • Did others step in to help, to advise, or to intercede?
  49. 56.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T DON’T blame that one person …
  50. 57.
  51. 58.
  52. 59.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Big challenges to chaos engineering Mostly Cultural • no time or flexibility to simulate disasters. • teams already spending all of its time fixing things. • can be very political. • might force deep conversations. • deeply invested in a specific technical roadmap (micro-services) that chaos engineering tests show is not as resilient to failures as originally predicted.
  53. 60.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Big challenges to chaos engineering • Chaos Engineering won’t make your system more robust, People will. • Chaos Engineering won’t replace __all__ the rest (test, quality, …) • Chaos Engineering is NOT the only way to learn from failure • Rollbacks are HARD because of state. • Your systems will continue to fail, sorry.
  54. 61.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T Changing culture takes time! Be patient…
  55. 63.

    © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S U M M I T More Resources • https://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf • https://www.gremlin.com • https://queue.acm.org/detail.cfm?id=2353017 • https://softwareengineeringdaily.com/ • https://github.com/dastergon/awesome-sre • https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf • https://medium.com/@NetflixTechBlog • http://principlesofchaos.org • https://speakerdeck.com/tammybutow/chaos-engineering-bootcamp • https://github.com/adhorn/awesome-chaos-engineering • https://www.infoq.com/presentations/netflix-chaos-microservices • http://royal.pingdom.com/wp-content/uploads/2015/04/pingdom_uptime_cheat_sheet.pdf • http://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallacy • https://medium.com/@adhorn
  56. 64.

    S U M M I T © 2019, Amazon Web

    Services, Inc. or its affiliates. All rights reserved.
  57. 65.

    Thank you! S U M M I T © 2019,

    Amazon Web Services, Inc. or its affiliates. All rights reserved. Adrian Hornsby https://medium.com/@adhorn