Chaos Engineering: Why breaking things should be practiced

As presented at the WebSummit in Lisbon 2018

Ever wondered how companies delivering global services like Amazon or Netflix are architecting and testing their software systems? If you're curious and want to learn how they do it - this session is for you! With the rise of micro-services and large-scale distributed architectures, software systems have grown increasingly complex and hard to understand. Adding to that complexity, the velocity of software delivery has also dramatically increased, resulting in failures being harder to predict and contain. While the cloud allows for high availability, redundancy and fault-tolerance, no single component can guarantee 100% uptime. Therefore, we have to understand availability but especially learn how to design architectures with failure in mind. And since failures have become more and more chaotic in nature, we must turn to chaos engineering in order to identify failures before they become outages. In this talk, I'll deep dive into availability, reliability and large-scale architectures and make an introduction to chaos engineering, a discipline that promotes breaking things on purpose in order to learn how to build more resilient systems.

Transcript

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Adrian Hornsby, Cloud Architecture Evangelist @adhorn Chaos Engineering: Why Breaking Things Should Be Practiced.
2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. https://xkcd.com/1428/
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Complex systems Amazon Twitter Netflix
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partial failure mode
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Resiliency: Ability for a system to handle and eventually recover from unexpected conditions
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. People Application Network & Data Infrastructure
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Building confidence through testing Unit testing of components: • Tested in isolation to ensure function meets expectations. Functional testing of integrations: • Each execution path tested to assure expected results. Is it enough???
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GameDay at Amazon Creating Resiliency Through Destruction https://www.youtube.com/watch?v=zoz0ZjfrQ9s
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Chaos engineering https://github.com/Netflix/SimianArmy
10. Failure injection • Start small & build confidence • Application level • Host failure • Resource attacks (CPU, memory, …) • Network attacks (dependencies, latency, …) • Region attacks • “Paul” attack https://www.gremlin.com https://github.com/Netflix/SimianArmy https://chaostoolkit.org
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in production.” http://principlesofchaos.org
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Break your systems on purpose. Find out their weaknesses and fix them before they break when least expected.
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Chaos doesn’t cause problems. It reveals them.” Nora Jones Senior Chaos Engineer, Netflix
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. https://bit.ly/2uKOJMQ
16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Phases of Chaos Engineering
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Steady State Hypothesis Design & Run Experiment Verify & Learn Fix
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Steady State
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is steady state? • ”normal” behavior of your system https://www.elastic.co/blog/timelion-tutorial-from-zero-to-hero
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is steady state? • ”normal” behavior of your system • Business Metric https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming-ae4db0e05f8a
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Business metrics at work Amazon: 100 ms of extra load time caused a 1% drop in sales (Greg Linden). Google: 500 ms of extra load time caused 20% fewer searches (Marissa Mayer). Yahoo!: 400 ms of extra load time caused a 5–9% increase in the number of people who clicked “back” before the page even loaded (Nicole Sullivan).
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hypothesis
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What if…? “What if this load balancer breaks?” “What if Redis becomes slow?” “What if a host on Cassandra goes away?” ”What if latency increases by 300ms?” ”What if the database stops?” Make it everyone’s problem!
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Disclaimer! Don’t make an hypothesis that you know will break you!
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Design & Run Experiment
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Designing experiment • Pick hypothesis • Scope the experiment • Identify metrics • Notify the organization
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Rules of thumbs • Start with very small • As close as possible to production • Minimize the blast radius. • Have an emergency STOP!
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Running Chaos Experiment Users Canary deployment Normal Version 99% Users 1% Users Start with ..
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Verify & Learn
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Quantifying the result of the experiment • Time to detect? • Time for notification? And escalation? • Time to public notification? • Time for graceful degradation to kick-in? • Time for self healing to happen? • Time to recovery – partial and full? • Time to all-clear and stable?
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DON’T blame that one person …
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PostMortems – COE (Correction of Errors) The 5 WHYs Outage Because of … Because of … Because of … Because of … NOT ENOUGH
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. More questions to ask • Can you clarify if there were any preceding events? • Why would they believe acting in this way was the best course of action to deliver the desired outcome? • Is there another failure mode that could present here? • What decisions or events prior to this made this work before? • Why stop there – are there places to dig deeper that could shine a light more on this? • Did others step in to help, to advise, or to intercede?
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Rules to remember! 1. Failure requires multiple faults 2. There is no isolated ‘cause’ of an accident. 3. There are multiple contributors to accidents.
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fix
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fix
37. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Patterns for Resilient Architecture
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability in parallel A = 1 – (1 – Ax)2 Part X Part X
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Availability in parallel Component Availability Downtime X 99% (2-nines) 3 days 15 hours Two X in parallel 99.99% (4-nines) 52 minutes Three X in parallel 99.9999% (6-nines) 31 seconds
40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-AZ architecture Region Availability zone a Availability zone b Availability zone c Application
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling group Service A Availability zone 1 Auto Scaling group AWS Region Service A Availability zone 2 Service B Service B Stateless Services
42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto-Scaling • Compute efficiency • Node failure • Traffic spikes • Performance bugs
43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling group Service A Availability zone 1 Auto Scaling group AWS Region Service A Availability zone 2 Service B Service B Auto-scaling
44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Decoupling with async pattern Listener Pub-Sub Queue Queue A A B B
45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. API: {DO foo} PUT JOB: {JobID: 0001, Task: DO foo} API: {JobID: 0001} GET JOB: {JobID: 0001, Task: DO foo} {JobID: 0001, Result: bar} Cache node Worker Instance Worker Instance Queue API Instance API Instance API Instance
46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Push Notification User Worker Instance Worker Instance Queue API Instance API Instance Cache node Fetch results API Instance
47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Degrade & prioritize traffic with queues Worker Instance Worker Instance API Instance API Instance API Instance High Priority Queue Low Priority Queue
48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Read / Write Sharding DB Instance DB instance read replica DB instance read replica DB instance read replica Instance Instance Instance
49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Database Federation Users DB Products DB Instance Instance Instance DB Instance DB instance read replica DB Instance DB instance read replica
50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Database Sharding User ShardID 002345 A 002346 B 002347 C 002348 B 002349 A C B A Instance Instance Instance DB Instance DB instance read replica DB Instance DB instance read replica DB Instance DB instance read replica
51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Transient state does not belong in the database.
52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Cascading Failures
53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Let’s talk about timeouts & retries!
54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Users App DB Conn Pool INSERT INSERT INSERT INSERT What happens if the DB “slows down”? Timeout client side Timeout backend side ? ?
55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. User 1 App DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = Not implemented Retry INSERT Retry INSERT ERROR: Failed to get connection from pool Retry
56. http://docs.python-requests.org/en/master/user/advanced/#timeouts
57. http://docs.python-requests.org/en/master/user/advanced/#timeouts
58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @timeout_decorator.timeout(5, timeout_exception=StopIteration) def timed_get(url): return requests.get(url) https://pypi.org/project/timeout-decorator/
59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Timeouts
60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How else could we have prevented the error? User 1 DB Conn Pool INSERT Retry INSERT Retry INSERT Retry ERROR: Failed to get connection from pool
61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. User 1 DB Conn Pool INSERT Timeout client side = 10s Timeout backend side = 10s Wait 2s before Retry INSERT INSERT Wait 4s before Retry Wait 8s before Retry Wait 16s before Retry Backing off between retries Releasing connections Backoff
62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. No jitter With jitter https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/ Simple Exponential Backoff is not enough: Add Jitter
63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: add jitter 0-1000ms def get_item(self, url, n=1): MAX_TRIES = 12 try: res = requests.get(url) except: if n > MAX_TRIES: return None n += 1 time.sleep((2 ** n) + (random.randint(0, 1000) / 1000.0)) return self.get_item(url, n) else: return res
64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. @backoff.on_exception(backoff.full_jitter, max_time=60) def poll_for_message(queue): return queue.get() https://pypi.org/project/backoff/ As of version 1.2, the default jitter function backoff.full_jitter implements the ‘Full Jitter’ algorithm as defined in the AWS Architecture Blog’s Exponential Backoff And Jitter post.
65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Idempotent operation No additional effect if it is called more than once with the same input parameters.
66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Service Degradation & Fallbacks
67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Circuit Breaker • Wrap a protected function call in a circuit breaker object, which monitors for failures. • If failures reach a certain threshold, the circuit breaker trips. Producer Circuit Breaker Consumer Connection Monitoring Timeouts Breaking Circuit
68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Non-blocking UI https://medium.com/@sophie_paxtonUX/stop-getting-in-my-way-non-blocking-ux-5cbbfe0f0158
69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Fire Drills
70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Big challenges to chaos engineering Mostly Cultural • no time or flexibility to simulate disasters. • teams already spending all of its time fixing things. • can be very political. • might force deep conversations. • deeply invested in a specific technical roadmap (micro-services) that chaos engineering tests show is not as resilient to failures as originally predicted.
71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Changing culture takes time! Be patient…
73. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark Thanks you! @adhorn https://medium.com/@adhorn
74. Did We Scan Your Badge? Remember to opt-in to AWS communications and you will receive a post-event email with a link to: • AWS Developer Workshop Slides • $200 in AWS Credits