Chaos Engineering: Why Breaking Things Should Be Practiced

Slide 1

Slide 1 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Chaos Engineering: Why breaking things should be practiced Adrian Hornsby Sr. Technical Evangelist Amazon Web Services Iñaki Alzorriz Director Platform Engineering Adidas @adhorn

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Building confidence through testing Unit testing of components: • Tested in isolation to ensure function meets expectations. Functional testing of integrations: • Each execution path tested to assure expected results. Is it enough???

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Failure injection • Start small & build confidence • Application level • Host failure • Resource attacks (CPU, memory, …) • Network attacks (dependencies, latency, …) • Region attack • Human attack https://www.gremlin.com https://github.com/Netflix/SimianArmy https://chaostoolkit.org

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Chaos engineering is NOT about breaking things randomly without a purpose, chaos engineering is about breaking things in a controlled environment and through well- planned experiments in order to build confidence in your application to withstand turbulent conditions.

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Slide 20

Slide 20 text

IÑAKI ALZORRIZ DIRECTOR PLATFORM ENGINEERING FOSTERING A RESILIENCE & QUALITY MINDSET

Slide 21

Slide 21 text

BUILDING THE DIGITAL TOMORROW FOR THE BEST SPORTS COMPANY IN THE WORLD 15-MAY-19 21

Slide 22

Slide 22 text

WE NEED TO BUILD RESILIENT SYSTEMS 15-MAY-19 22

Slide 23

Slide 23 text

15-MAY-19 23

Slide 24

Slide 24 text

ADIDAS RUNNING ON AWS AWS Glue Amazon Athena Amazon QuickSight AWS IAM Amazon S3 Amazon SQS Amazon Aurora Amazon SNS AWS Systems Manager AWS Trusted Advisor Amazon DynamoDB Amazon API Gateway AWS Step Functions Amazon Elasticsearch Amazon EBS AWS CloudTrail Amazon EFS Amazon EC2 Amazon Route 53 Amazon SES Amazon RDS Amazon EMR Amazon ECS AWS Lambda Elastic Load Balancing Amazon CloudWatch Amazon GuardDuty AWS KMS

Slide 25

Slide 25 text

GREAT TECHNOLOGY REQUIRES GREAT TEAMS 15-MAY-19 25 technical enablement quality culture

Slide 26

Slide 26 text

OUR CONTAINER ORCHESTRATION BASED ON KUBERNETES 15-MAY-19 26 self-healing autoscaling multi-region Ideal platform for building resilient applications self-service replication

Slide 27

Slide 27 text

OUR CONTAINER ORCHESTRATION BASED ON KUBERNETES 15-MAY-19 27 no readiness/liveness probes autoscaling not configured incorrect usage of request/limits The reality?

Slide 28

Slide 28 text

15-MAY-19 28 project-seeds runbooks guidelines/whitepapers training awareness guide & consult HOW TO MAKE SURE THIS DOESN’T HAPPEN?

Slide 29

Slide 29 text

FAST DATA PLATFORM make data easily consumable to provide speed 15-MAY-19 29 replication self-service cataloguing by default BDP integrated integration patterns seeds consult

Slide 30

Slide 30 text

OUR MONITORING PLATFORM 15-MAY-19 30 observability by default self-service centralized default-views

Slide 31

Slide 31 text

HELP THE TEAMS MEASURE THEIR SDLC MATURITY 15-MAY-19 31

Slide 32

Slide 32 text

HOW TO SPREAD THE QUALITY CULTURE AND MINDSET? 15-MAY-19 32 We can provide the best platforms and enablers, but the teams are the ones that need to make sure that they build their systems in the way we need them to.

Slide 33

Slide 33 text

WE DON'T ENFORCE, WE ENABLE OUR TEAMS 15-MAY-19 33 SPEED AUTONOMY FREEDOM DECOUPLE EXPERIMENT STANDARD SUSTAINABLE RESILIENCE SCALE

Slide 34

Slide 34 text

DEMOCRATIZATION OF TECHNOLOGY MANAGEMENT 15-MAY-19 34

Slide 35

Slide 35 text

THE INTRODUCTION OF NEW TECHONOLOGIES IS ALWAYS A CO-CREATION OPPORTUNITY 15-MAY-19 35 We explore together Build & Evaluate in a real case Make sure is mature enough Guidelines & Enablers

Slide 36

Slide 36 text

AS AN EXAMPLE OF DEMOCRATIZATION – ALL THE ENABLERS ARE SHARED AS AN ”INNERSOURCE” INITIATIVE 15-MAY-19 36

Slide 37

Slide 37 text

GAMIFICATION AS A WAY TO ENCOURAGE QUALITY 15-MAY-19 37

Slide 38

Slide 38 text

OUR CHAOS ENVIRONMENT – TEST YOUR APPLICATION FOR RESILIENCE 15-MAY-19 38

Slide 39

Slide 39 text

COMMUNITY EVENTS • Technical onboardings • More than 10 active communities of practices • Biweekly meetings of the engineering community. • adidas tech summit 2019 in Herzogenaurach 15-MAY-19 39

Slide 40

Slide 40 text

DEVOPS CUP 15-MAY-19 40 6 month competition DevOps adoption adapt creativity

Slide 41

Slide 41 text

ENABLE OUR TEAMS & SPREAD THE QUALITY CULTURE TOGETHER WE PLAY TO WIN

Slide 42

Slide 42 text

THANK YOU!

Slide 43

Slide 43 text

Slide 44

Slide 44 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T What is steady state? • ”normal” behavior of your system https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero

Slide 45

Slide 45 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T What is steady state? • ”normal” behavior of your system • Business Metric https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a

Slide 46

Slide 46 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Business metrics at work Amazon: 100 ms of extra load time caused a 1% drop in sales (Greg Linden). Google: 500 ms of extra load time caused 20% fewer searches (Marissa Mayer). Yahoo!: 400 ms of extra load time caused a 5–9% increase in the number of people who clicked “back” before the page even loaded (Nicole Sullivan).

Slide 47

Slide 47 text

Slide 48

Slide 48 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T What if…? “What if this load balancer breaks?” “What if Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the database stops?” Make it everyone’s problem!

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Rules of thumbs • Start with very small • As close as possible to production • Minimize the blast radius. • Have an emergency STOP!

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Slide 55

Slide 55 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Quantifying the result of the experiment • Time to detect? • Time for notification? And escalation? • Time to public notification? • Time for graceful degradation to kick-in? • Time for self healing to happen? • Time to recovery – partial and full? • Time to all-clear and stable?

Slide 56

Slide 56 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T PostMortems – COE (Correction of Errors) The 5 WHYs Outage Because of … Because of … Because of … Because of … NOT ENOUGH

Slide 57

Slide 57 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T More questions to ask • Can you clarify if there were any preceding events? • Why would they believe acting in this way was the best course of action to deliver the desired outcome? • Is there another failure mode that could present here? • What decisions or events prior to this made this work before? • Why stop there – are there places to dig deeper that could shine a light more on this? • Did others step in to help, to advise, or to intercede?

Slide 58

Slide 58 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Rules to remember! 1. Failure requires multiple faults 2. There is no isolated ‘cause’ of an accident. 3. There are multiple contributors to accidents.

Slide 59

Slide 59 text

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Big challenges to chaos engineering Mostly Cultural • no time or flexibility to simulate disasters. • teams already spending all of its time fixing things. • can be very political. • might force deep conversations. • deeply invested in a specific technical roadmap (micro-services) that chaos engineering tests show is not as resilient to failures as originally predicted.

Slide 63

Slide 63 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T Big challenges to chaos engineering • Chaos Engineering won’t make your system more robust, People will. • Chaos Engineering won’t replace __all__ the rest (test, quality, …) • Chaos Engineering is NOT the only way to learn from failure • Rollbacks are HARD because of state. • Your systems will continue to fail, sorry.

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. S U M M I T More Resources • https://mvdirona.com/jrh/talksAndPapers/JamesRH_Lisa.pdf • https://www.gremlin.com • https://queue.acm.org/detail.cfm?id=2353017 • https://softwareengineeringdaily.com/ • https://github.com/dastergon/awesome-sre • https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf • https://medium.com/@NetflixTechBlog • http://principlesofchaos.org • https://speakerdeck.com/tammybutow/chaos-engineering-bootcamp • https://github.com/adhorn/awesome-chaos-engineering • https://www.infoq.com/presentations/netflix-chaos-microservices • http://royal.pingdom.com/wp-content/uploads/2015/04/pingdom_uptime_cheat_sheet.pdf • http://willgallego.com/2018/04/02/no-seriously-root-cause-is-a-fallacy • https://medium.com/@adhorn