Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chaos Engineering: getting out of the starting blocks

Chaos Engineering: getting out of the starting blocks

Architectures are growing increasingly distributed and hard to understand. As a result, software systems have become extremely difficult to debug and test, which increases the risk of failure. With these new challenges, chaos engineering ha become attractive to many organizations as a mechanism for underling the behavior of systems under expected circumstances.

Whilst interest is growing, few have managed to build sustainable chaos engineering practices. In this talk, I will review the state of chaos engineering, the issues customers are facing, based on my learning as an AWS Solution Architect and Technologist focusing on Chaos Engineering and explain why I started to build tools to help with failure injection.

E6c942c0f8e6042fbd47fcd3c4113b90?s=128

Adrian Hornsby

January 23, 2020
Tweet

Transcript

  1. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Chaos Engineering: Getting out of the starting blocks Adrian Hornsby Principal Technical Evangelist Amazon Web Services
  2. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. What currently prevents the wide adoption of chaos engineering in your organization?
  3. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  4. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Why is production chaos?
  5. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  6. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #0 - DON’T CALL IT CHAOS ENGINEERING.
  7. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #0 - DON’T CALL IT CHAOS ENGINEERING.
  8. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #1 - DON’T FOCUS ON CHAOS ENGINEERING, LOOK AT THE BIGGER PICTURE.
  9. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Good intentions never work [...]
  10. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Because people already had good intentions
  11. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  12. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. If good intentions don’t work, what does?
  13. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. The Andon Cord
  14. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. 1902
  15. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Toyota will not allow any defect that they know about to go down the manufacturing line.
  16. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Source: http://www.autoexpress.co.uk/toyota/prius/34615/japanese-earthquake-hits-car-production
  17. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Andon Customer Service
  18. • Erroneously listed recharge cable as included • Andon cord

    pulled and page corrected • Contacts per unit go from 33% to 3.7%
  19. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  20. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. "Good intentions never work, you need good mechanisms to make anything happen." Jeff Bezos
  21. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. People have good intention to start with!
  22. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Good Mechanisms ≈ Complete Processes Tools Adoption Audit
  23. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #2 - CHANGE BEGINS WITH UNDERSTANDING.
  24. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. What are the top 5 “painful” reasons for your fires?
  25. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. 1. It is always DNS 2. Configuration drift 3. SSL Certificate expiration 4. Deployment failure 5. Failed link to 3rd party provider
  26. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Anatomy of a COE • What happened? • What was the impact on customers and your business? • What were the contributing factors? • What data do you have to support this? • What lessons did you learn? • What corrective actions are you taking?
  27. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Audit Weekly Operational Metrics Review • Continuous inspection mechanism • Maintains focus on operations • Foundation of a healthy operations program Typical Agenda - typically divided into fifteen-minute slots • Share successes and failings • Action items follow up • Review COEs • Review key service metrics • Identify new best practices
  28. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Policy Engine • Automated risk and opportunity analyzer • Identifies potential risks to availability, infrastructure, security and more • Highlights opportunities to optimize resource utilization • Extensible and configurable • Provides a view into policy compliance • Allows acknowledgment • Reports roll-up the organization hierarchy Mechanism to propagate local learnings globally
  29. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #3 - CHOOSE YOUR TROJAN HORSE.
  30. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Find the right team to start with: Not the best (improvements are harder) Not the worse (they have bigger problems)
  31. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Choose the metrics to measure improvement: MTTR is __always__ a good default.
  32. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #4 - OVER-INDEX ON THE HYPOTHESIS.
  33. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. STEADY STATE HYPOTHESIS RUN EXPERIMENT VERIFY IMPROVE
  34. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  35. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  36. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #5 - INTRODUCE CHAOS ENGINEERING EARLY IN THE JOURNEY.
  37. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Start simple and local!! $ docker stop 94a214bbeebd
  38. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. DDoS yourself $ wrk -t12 -c400 -d30s http://127.0.0.1/api/health
  39. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Burn CPU with Stress(–ng) $ stress-ng --cpu 0--cpu-method matrixprod -t 60s https://kernel.ubuntu.com/~cking/stress-ng/
  40. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Adding latency to the network $ tc qdisc add dev eth0 root netem delay 300ms
  41. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Blocks DNS resolution $ iptables -A INPUT -p tcp -m tcp --dport 53 -j DROP
  42. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. #6 - BLAST-RADIUS REDUCTION MINDSET.
  43. #7 - IF YOU HAVEN’T VERIFIED IT, IT’S PROBABLY BROKEN.

  44. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Verification: 1. Disaster Recovery & backups 2. Auto scaling 3. Multi-AZ 3. Fault tolerance & self healing 4. People
  45. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved.
  46. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Getting out of the starting blocks.
  47. © 2020, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. Tools Processes Culture Technology
  48. Thank you! © 2019, Amazon Web Services, Inc. or its

    affiliates. All rights reserved. Adrian Hornsby https://medium.com/@adhorn adhorn