Chaos Engineering: Why Breaking Things Should Be Practiced

As presented at the AWS Summit in Madrid - with Iñaki Alzorriz, Director Platform Engineering at Adidas

With the wide adoption of micro-services and large-scale distributed systems, architectures have grown increasingly complex and hard to understand. Worse, the software systems running them have become extremely difficult to debug and test, increasing the risk of outages. With these new challenges, new tools are required and since failures have become more and more chaotic in nature, we must turn to chaos engineering in order to reveal failures before they become outages. In this talk, we will make an introduction to chaos engineering, a discipline that promotes breaking things on purpose in order to learn how to build more robust systems.

Transcript

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Chaos Engineering:
   Why breaking things should be practiced
   Adrian Hornsby
   Sr. Technical Evangelist
   Amazon Web Services
   Iñaki Alzorriz
   Director Platform Engineering
   Adidas
   @adhorn
   

   View Slide

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Been there?
   

   View Slide

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Distributed Systems are hard
   Amazon Twitter Netflix
   

   View Slide

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Failures are a given and
   everything will eventually fail
   over time.
   Werner Vogels
   CTO – Amazon.com
   “
   “
   

   View Slide

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Resiliency: Ability for a system to handle and
   eventually recover from unexpected conditions
   

   View Slide

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Partial failure mode
   

   View Slide

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   How do we build resilient software
   systems?
   

   View Slide

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   People
   Application
   Network & Data
   Infrastructure
   

   View Slide

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   S U M M I T
   Building confidence through testing
   Unit testing of components:
   • Tested in isolation to ensure function meets expectations.
   Functional testing of integrations:
   • Each execution path tested to assure expected results.
   Is it enough???
   

   View Slide

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    GameDay at Amazon
    Creating Resiliency Through Destruction
    https://www.youtube.com/watch?v=zoz0ZjfrQ9s
    

    View Slide

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Chaos engineering
    https://github.com/Netflix/SimianArmy
    

    View Slide

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Failure injection
    • Start small & build confidence
    • Application level
    • Host failure
    • Resource attacks (CPU, memory, …)
    • Network attacks (dependencies, latency, …)
    • Region attack
    • Human attack
    https://www.gremlin.com
    https://github.com/Netflix/SimianArmy https://chaostoolkit.org
    

    View Slide

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Break your systems on purpose.
    Find out their weaknesses and fix
    them before they break when
    least expected.
    

    View Slide

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Chaos engineering is NOT about breaking
    things randomly without a purpose, chaos
    engineering is about breaking things in a
    controlled environment and through well-
    planned experiments in order to build
    confidence in your application to withstand
    turbulent conditions.
    

    View Slide

15. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Chaos Engineering
    

    View Slide

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Steady
    State
    Hypothesis
    Design & Run
    Experiment
    Fix
    Build Resilient
    Systems
    Verify & Learn
    

    View Slide

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Build Resilient
    Systems
    

    View Slide

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Operations
    Infrastructure
    Application
    Software
    

    View Slide

19. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Iñaki Alzorriz
    Director Platform Engineering
    Adidas
    

    View Slide

20. IÑAKI ALZORRIZ
    DIRECTOR PLATFORM ENGINEERING
    FOSTERING A
    RESILIENCE &
    QUALITY MINDSET
    

    View Slide

21. BUILDING THE DIGITAL
    TOMORROW FOR THE BEST
    SPORTS COMPANY IN THE
    WORLD
    15-MAY-19 21
    

    View Slide

22. WE NEED TO BUILD RESILIENT SYSTEMS
    15-MAY-19 22
    

    View Slide

23. 15-MAY-19 23
    

    View Slide

24. ADIDAS RUNNING ON AWS
    AWS Glue
    Amazon Athena
    Amazon QuickSight
    AWS IAM
    Amazon S3
    Amazon SQS
    Amazon Aurora
    Amazon SNS
    AWS Systems Manager
    AWS Trusted Advisor
    Amazon DynamoDB
    Amazon API Gateway
    AWS Step Functions
    Amazon Elasticsearch
    Amazon EBS
    AWS CloudTrail
    Amazon EFS
    Amazon EC2
    Amazon Route 53
    Amazon SES
    Amazon RDS
    Amazon EMR
    Amazon ECS
    AWS Lambda
    Elastic Load Balancing
    Amazon CloudWatch
    Amazon GuardDuty
    AWS KMS
    

    View Slide

25. GREAT TECHNOLOGY REQUIRES GREAT TEAMS
    15-MAY-19 25
    technical enablement
    quality culture
    

    View Slide

26. OUR CONTAINER ORCHESTRATION BASED ON KUBERNETES
    15-MAY-19 26
    self-healing
    autoscaling
    multi-region
    Ideal platform for building resilient applications
    self-service
    replication
    

    View Slide

27. OUR CONTAINER ORCHESTRATION BASED ON KUBERNETES
    15-MAY-19 27
    no readiness/liveness probes
    autoscaling not configured
    incorrect usage of request/limits
    The reality?
    

    View Slide

28. 15-MAY-19 28
    project-seeds
    runbooks
    guidelines/whitepapers
    training
    awareness
    guide & consult
    HOW TO MAKE SURE THIS DOESN’T HAPPEN?
    

    View Slide

29. FAST DATA PLATFORM
    make data easily consumable to provide speed
    15-MAY-19 29
    replication
    self-service
    cataloguing by default
    BDP integrated
    integration patterns
    seeds
    consult
    

    View Slide

30. OUR MONITORING PLATFORM
    15-MAY-19 30
    observability by default
    self-service
    centralized
    default-views
    

    View Slide

31. HELP THE TEAMS MEASURE THEIR SDLC MATURITY
    15-MAY-19 31
    

    View Slide

32. HOW TO SPREAD THE QUALITY CULTURE AND MINDSET?
    15-MAY-19 32
    We can provide the best platforms and
    enablers, but the teams are the ones that need
    to make sure that they build their systems in
    the way we need them to.
    

    View Slide

33. WE DON'T ENFORCE, WE ENABLE OUR TEAMS
    15-MAY-19 33
    SPEED
    AUTONOMY
    FREEDOM
    DECOUPLE
    EXPERIMENT
    STANDARD
    SUSTAINABLE
    RESILIENCE
    SCALE
    

    View Slide

34. DEMOCRATIZATION OF TECHNOLOGY MANAGEMENT
    15-MAY-19 34
    

    View Slide

35. THE INTRODUCTION OF NEW TECHONOLOGIES
    IS ALWAYS A CO-CREATION OPPORTUNITY
    15-MAY-19 35
    We explore
    together
    Build & Evaluate in
    a real case
    Make sure is
    mature enough
    Guidelines &
    Enablers
    

    View Slide

36. AS AN EXAMPLE OF DEMOCRATIZATION – ALL THE ENABLERS ARE SHARED AS AN ”INNERSOURCE”
    INITIATIVE
    15-MAY-19 36
    

    View Slide

37. GAMIFICATION AS A WAY TO ENCOURAGE QUALITY
    15-MAY-19 37
    

    View Slide

38. OUR CHAOS ENVIRONMENT – TEST YOUR APPLICATION FOR RESILIENCE
    15-MAY-19 38
    

    View Slide

39. COMMUNITY EVENTS
    • Technical onboardings
    • More than 10 active communities of practices
    • Biweekly meetings of the engineering community.
    • adidas tech summit 2019 in Herzogenaurach
    15-MAY-19 39
    

    View Slide

40. DEVOPS CUP
    15-MAY-19 40
    6 month
    competition
    DevOps adoption adapt
    creativity
    

    View Slide

41. ENABLE OUR TEAMS & SPREAD THE
    QUALITY CULTURE
    TOGETHER WE PLAY TO WIN
    

    View Slide

42. THANK YOU!
    

    View Slide

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Steady
    State
    

    View Slide

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    What is steady state?
    • ”normal” behavior of your system
    https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero
    

    View Slide

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    What is steady state?
    • ”normal” behavior of your system
    • Business Metric
    https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a
    

    View Slide

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Business metrics at work
    Amazon: 100 ms of extra load time caused a 1% drop in sales (Greg Linden).
    Google: 500 ms of extra load time caused 20% fewer searches (Marissa Mayer).
    Yahoo!: 400 ms of extra load time caused a 5–9% increase in the number
    of people who clicked “back” before the page even loaded (Nicole Sullivan).
    

    View Slide

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Hypothesis
    

    View Slide

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    What if…?
    “What if this load balancer breaks?”
    “What if Redis becomes slow?”
    “What if a host on Cassandra goes away?”
    ”What if latency increases by 300ms?”
    ”What if the database stops?”
    Make it everyone’s problem!
    

    View Slide

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Disclaimer!
    Don’t make an hypothesis that you know
    will break you!
    

    View Slide

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Design & Run
    Experiment
    

    View Slide

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Designing experiment
    • Pick hypothesis
    • Scope the experiment
    • Identify metrics
    • Notify the organization
    

    View Slide

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Rules of thumbs
    • Start with very small
    • As close as possible to production
    • Minimize the blast radius.
    • Have an emergency STOP!
    

    View Slide

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Running Chaos Experiment
    Users
    Canary deployment
    Normal Version
    99%
    Users
    1%
    Users
    Start with ..
    

    View Slide

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Verify & Learn
    

    View Slide

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Quantifying the result of the experiment
    • Time to detect?
    • Time for notification? And escalation?
    • Time to public notification?
    • Time for graceful degradation to kick-in?
    • Time for self healing to happen?
    • Time to recovery – partial and full?
    • Time to all-clear and stable?
    

    View Slide

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    PostMortems – COE (Correction of Errors)
    The 5 WHYs
    Outage
    Because
    of …
    Because
    of …
    Because
    of …
    Because
    of …
    NOT
    ENOUGH
    

    View Slide

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    More questions to ask
    • Can you clarify if there were any preceding events?
    • Why would they believe acting in this way was the best course of action to
    deliver the desired outcome?
    • Is there another failure mode that could present here?
    • What decisions or events prior to this made this work before?
    • Why stop there – are there places to dig deeper that could shine a light more
    on this?
    • Did others step in to help, to advise, or to intercede?
    

    View Slide

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Rules to remember!
    1. Failure requires multiple faults
    2. There is no isolated ‘cause’ of an accident.
    3. There are multiple contributors to accidents.
    

    View Slide

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    DON’T blame that one person …
    

    View Slide

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Fix
    

    View Slide

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Fix
    

    View Slide

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Big challenges to chaos engineering
    Mostly Cultural
    • no time or flexibility to simulate disasters.
    • teams already spending all of its time fixing things.
    • can be very political.
    • might force deep conversations.
    • deeply invested in a specific technical roadmap (micro-services) that
    chaos engineering tests show is not as resilient to failures as originally
    predicted.
    

    View Slide

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Big challenges to chaos engineering
    • Chaos Engineering won’t make your system more robust,
    People will.
    • Chaos Engineering won’t replace __all__ the rest (test, quality, …)
    • Chaos Engineering is NOT the only way to learn from failure
    • Rollbacks are HARD because of state.
    • Your systems will continue to fail, sorry.
    

    View Slide

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    “Quality is not an act, it is a habit”
    Aristotle, some time around 350BC
    

    View Slide

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    Changing culture takes time!
    Be patient…
    

    View Slide

66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    

    View Slide

67. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    S U M M I T
    More Resources
    • https://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf
    • https://www.gremlin.com
    • https://queue.acm.org/detail.cfm?id=2353017
    • https://softwareengineeringdaily.com/
    • https://github.com/dastergon/awesome-sre
    • https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
    • https://medium.com/@NetflixTechBlog
    • http://principlesofchaos.org
    • https://speakerdeck.com/tammybutow/chaos-engineering-bootcamp
    • https://github.com/adhorn/awesome-chaos-engineering
    • https://www.infoq.com/presentations/netflix-chaos-microservices
    • http://royal.pingdom.com/wp-content/uploads/2015/04/pingdom_uptime_cheat_sheet.pdf
    • http://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallacy
    • https://medium.com/@adhorn
    

    View Slide

68. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    

    View Slide

69. Thank you!
    S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Adrian Hornsby
    @adhorn
    https://medium.com/@adhorn
    

    View Slide