Using Chaos To Build Resilient Systems

https://qconnewyork.com/ny2018/speakers/tammy-butow

There are those of us that are motivated to build resilient systems, improve uptime, move fast and keep systems reliable. Then there are those of us who feel overwhelmed by our to-do lists and the features or projects we feel we need to get out the door.

The world needs more resilient systems because the world needs engineers in this for the long haul. We can create a better future for ourselves, those who come after us, our customers and our wider teams by focusing on building resilient systems. How do we make it easier for everyone to build resilient systems?

It is not easy to build resilient systems, but that doesn’t mean we shouldn’t try. Engineers love a technical challenge. In this talk I will explain how focusing on the detection, mitigation, resolution and prevention of incidents is a great place to start. I will share my experiences using chaos engineering to build resilient systems... even when you can’t build your systems from scratch.

Transcript

1. USING CHAOS TO BUILD RESILIENT SYSTEMS @tammybütow, Gremlin
2. What’s the scale of your infra? @tammybütow #QCONNYC
3. How many services do you have running in production? @tammybütow #QCONNYC
4. How many engineers do you have at your company? @tammybütow #QCONNYC
5. A Common Chaos Engineering Journey @tammybütow, Gremlin @tammybütow #QCONNYC
6. TOP 5 MOST POPULAR WAYS TO USE CHAOS ENGINEERING IN 2018 @tammybütow #QCONNYC
7. ADVANCED USES OF CHAOS ENGINEERING @tammybütow #QCONNYC
8. @tammybütow, Gremlin @tammybütow #QCONNYC What happened this week: June 2018 Slack Outage
9. @tammybütow, Gremlin @tammybütow #QCONNYC
10. TAMMY BÜTOW Principal SRE, Gremlin Causing chaos in prod since 2009. Previously SRE Manager @ Dropbox leading Databases, Block Storage and Code Workflows for 500 million users and 800 engineers. @tammybütow % @tammybütow #QCONNYC
11. GREMLIN • We are practitioners of Chaos Engineering • We build software that helps engineers build resilient systems in a safe, secure and simple way. • We offer 11 ways to inject chaos for your Chaos Engineering experiments (e.g. host/container packet loss and shutdown) @tammybütow #QCONNYC
12. PART 1: LAYING THE FOUNDATION @tammybütow #QCONNYC
13. • A resilient system is a highly available and durable system. • A resilient system can maintain an acceptable level of service in the face of failure. • A resilient system can weather the storm (a misconfiguration, a large scale natural disaster or controlled chaos engineering). @tammybütow #QCONNYC Let’s Define A Resilient System:
14. It would be silly to give an Olympic pole-vaulter a broom and ban them from practicing! @tammybütow #QCONNYC
15. “Thoughtful planned experiments designed to reveal the weaknesses in our systems” - Kolton Andrus, Gremlin CEO @tammybütow, Gremlin @tammybütow #QCONNYC
16. Inject something harmful in order to build an immunity. @tammybütow, Gremlin @tammybütow #QCONNYC Think of it like a vaccination:
17. Eventually systems will break in many undesired ways. Break them first on purpose with controlled chaos! @tammybütow #QCONNYC
18. DOGFOODING • Using your own product. • For us that means using Gremlin for our Chaos Engineering experiments. • Failure Fridays @tammybütow #QCONNYC
19. Failure Fridays are dedicated time for teams to collaboratively focus on using Chaos Engineering practices to reveal weaknesses in your services. @tammybütow #QCONNYC
20. WHY DO DISTRIBUTED SYSTEMS NEED CHAOS? • Unusual hard to debug failures are common • Systems & companies scale rapidly and Chaos Engineering helps you learn along the way @tammybütow #QCONNYC
21. FULL-STACK CHAOS ENGINEERING • You can inject chaos at any layer. • API, App, Cache, Database, OS, Host, Network, Power & more. @tammybütow #QCONNYC
22. WHY RUN CHAOS ENGINEERING EXPERIMENTS? @tammybütow #QCONNYC
23. Are you confident that your metrics and alerting are as good as they should be? #pagerpain @tammybütow #QCONNYC
24. Are you confident your customers are getting as good an experience as they should be? #customerpain @tammybütow #QCONNYC
25. Are you losing money due to downtime and broken features? #businesspain @tammybütow #QCONNYC
26. HOW DO YOU RUN CHAOS ENGINEERING EXPERIMENTS? @tammybütow #QCONNYC
27. HOW TO RUN A CHAOS ENGINEERING EXPERIMENT • Form a hypothesis • Consider blast radius • Run experiment • Measure results • Find & fix issues or scale ⚡ @tammybütow #QCONNYC
28. Don’t run before you can walk @tammybütow, Gremlin @tammybütow #QCONNYC
29. The 3 Prerequisites for Chaos Engineering @tammybütow, Gremlin @tammybütow #QCONNYC 1. Monitoring & Observability 2. On-Call & Incident Management 3. Know Your Cost of Downtime Per Hour
30. What Do I Use For Monitoring & Observability? @tammybütow, Gremlin @tammybütow #QCONNYC
31. We All Need To Know The Cost Of Downtime @tammybütow, Gremlin @tammybütow #QCONNYC
32. We All Need Incident Management @tammybütow, Gremlin @tammybütow #QCONNYC
33. HOW TO CHOOSE A CHAOS EXPERIMENT • Identify top 5 critical systems • Choose 1 system • Whiteboard the system • Select attack: resource/ state/network • Determine scope ⚡ @tammybütow #QCONNYC
34. WHAT SHOULD WE MEASURE? • Availability — 500s • Service specific KPIs • System metrics: CPU, IO, Disk • Customer complaints @tammybütow #QCONNYC
35. HOW TO RUN YOUR OWN GAMEDAY! @tammybütow #QCONNYC gremlin.com/gameday
36. HOW TO RUN YOUR OWN GAMEDAY! @tammybütow #QCONNYC gremlin.com/gameday
37. EXAMPLE SYSTEM: KUBERNETES RETAIL STORE User Primary: kube-01 Node: kube-02 Node: kube-03 Node: kube-04 @tammybütow #QCONNYC
38. PART 2: RESOURCE CHAOS ENGINEERING @tammybütow #QCONNYC
39. We can increase CPU, Disk, IO & Memory consumption to ensure monitoring is setup to catch problems. Important to catch issues before they turn into high severity incidents (unable to purchase new product!) and downtime for customers. RESOURCE CHAOS @tammybütow #QCONNYC
40. CPU CHAOS @tammybütow #QCONNYC
41. https://github.com/tammybutow/chaosengineeringbootcamp LET’S CREATE A “KNOWN-KNOWN” EXPERIMENT @tammybütow #QCONNYC
42. CHAOS IN TOP @tammybütow #QCONNYC
43. LET’S KILL THE CHAOS NOW @tammybütow #QCONNYC
44. NO MORE CHAOS IN TOP @tammybütow #QCONNYC
45. DISK CHAOS @tammybütow #QCONNYC
46. DISK CHAOS @tammybütow #QCONNYC
47. MEMORY CHAOS @tammybütow #QCONNYC
48. MEMORY CHAOS free -m @tammybütow #QCONNYC
49. PART 3: STATE CHAOS ENGINEERING @tammybütow #QCONNYC
50. PROCESS CHAOS @tammybütow #QCONNYC
51. Ways to create process chaos on purpose: PROCESS CHAOS • Kill one process • Loop kill a process • Spawn new processes • Fork bomb @tammybütow #QCONNYC
52. PROCESS CHAOS pkill -u chaos @tammybütow #QCONNYC
53. SHUTDOWN CHAOS @tammybütow #QCONNYC
54. SHUTDOWN CHAOS shutdown -h @tammybütow #QCONNYC
55. WHAT ARE OTHER WAYS YOU CAN TURN OFF A SERVER? WHAT IF YOU WANT TO TURN OFF EVERY SERVER WHEN IT’S ONE WEEK OLD? @tammybütow #QCONNYC
56. HALT, REBOOT & POWEROFF CHAOS halt @tammybütow #QCONNYC
57. WHAT ABOUT SHUTTING DOWN
 CONTAINERS AND K8’S PODS? @tammybütow #QCONNYC
58. THE MANY WAYS TO KILL CONTAINERS • Kill self • Kill a container from the host • Use one container to kill another • Use one container to kills several containers • Use several containers to kill several @tammybütow #QCONNYC
59. The average lifespan of a container is 2.5 days And they fail in many unexpected ways. @tammybütow #QCONNYC
60. TIME TRAVEL CHAOS @tammybütow #QCONNYC
61. TIME TRAVEL CHAOS AKA CLOCK SKEW ntpq @tammybütow #QCONNYC
62. PART 4: NETWORK CHAOS ENGINEERING @tammybütow #QCONNYC
63. BLACKHOLE CHAOS @tammybütow #QCONNYC
64. BLACKHOLE CHAOS ip route show @tammybütow #QCONNYC
65. DNS CHAOS @tammybütow #QCONNYC
66. DNS CHAOS @tammybütow #QCONNYC
67. DNS CHAOS @tammybütow #QCONNYC
68. LATENCY CHAOS @tammybütow #QCONNYC
69. LATENCY CHAOS mtr google.com @tammybütow #QCONNYC
70. PACKET LOSS CHAOS @tammybütow #QCONNYC
71. PACKET LOSS CHAOS @tammybütow #QCONNYC
72. PART 5: COMPLEX OUTAGES @tammybütow #QCONNYC
73. We can combine different types of chaos engineering experiments to reproduce complicated outages. Reproducing outages gives you confidence you can handle it if/when it happens again. @tammybütow #QCONNYC
74. Let’s go back in time to look at some of the worst outage stories that kicked off the introduction of chaos engineering. @tammybütow #QCONNYC
75. DROPBOX’S WORST OUTAGE EVER Some master-replica pairs were impacted which resulted in the site going down. https://blogs.dropbox.com/tech/2014/01/outage-post-mortem/ @tammybütow #QCONNYC
76. UBER’S DATABASE OUTAGE 1.Master log replication to S3 failed 2.Logs backed up on the primary 3.Alerts fired to engineer but they are ignored 4.Disk fills up on database primary 5.Engineer deletes unarchived WAL files 6.Error in config prevents promotion — Matt Ranney, Uber, 2015 @tammybütow #QCONNYC
77. OUTAGES HAPPEN. @tammybütow #QCONNYC
78. THERE ARE MANY MORE OUTAGES YOU CAN READ ABOUT HERE: https://github.com/danluu/post-mortems @tammybütow #QCONNYC
79. HOW CAN YOU CONTINUE YOUR CHAOS ENGINEERING JOURNEY? @tammybütow #QCONNYC
80. JOIN THE CHAOS SLACK GREMLIN.COM/SLACK @tammybütow #QCONNYC
81. LEARN WITH THE GREMLIN COMMUNITY GREMLIN.COM/COMMUNITY @tammybütow #QCONNYC
82. THE FIRST CHAOS ENGINEERING CONFERENCE! CHAOSCONF.IO @tammybütow #QCONNYC
83. THANK YOU QCON NYC @tammybütow #CHAOSENGINEERING