Using Chaos To Build Resilient Systems

Using Chaos To Build Resilient Systems

https://qconnewyork.com/ny2018/speakers/tammy-butow

There are those of us that are motivated to build resilient systems, improve uptime, move fast and keep systems reliable. Then there are those of us who feel overwhelmed by our to-do lists and the features or projects we feel we need to get out the door.

The world needs more resilient systems because the world needs engineers in this for the long haul. We can create a better future for ourselves, those who come after us, our customers and our wider teams by focusing on building resilient systems. How do we make it easier for everyone to build resilient systems?

It is not easy to build resilient systems, but that doesn’t mean we shouldn’t try. Engineers love a technical challenge. In this talk I will explain how focusing on the detection, mitigation, resolution and prevention of incidents is a great place to start. I will share my experiences using chaos engineering to build resilient systems... even when you can’t build your systems from scratch.

203e64aeb53ae59b2b4dcf923c163c23?s=128

Tammy Bütow

June 29, 2018
Tweet

Transcript

  1. USING CHAOS TO BUILD RESILIENT SYSTEMS @tammybütow, Gremlin

  2. What’s the scale of your infra? @tammybütow #QCONNYC

  3. How many services do you have running in production? @tammybütow

    #QCONNYC
  4. How many engineers do you have at your company? @tammybütow

    #QCONNYC
  5. A Common Chaos Engineering Journey @tammybütow, Gremlin @tammybütow #QCONNYC

  6. TOP 5 MOST POPULAR WAYS TO USE CHAOS ENGINEERING IN

    2018 @tammybütow #QCONNYC
  7. ADVANCED USES OF CHAOS ENGINEERING @tammybütow #QCONNYC

  8. @tammybütow, Gremlin @tammybütow #QCONNYC What happened this week: June 2018

    Slack Outage
  9. @tammybütow, Gremlin @tammybütow #QCONNYC

  10. TAMMY BÜTOW Principal SRE, Gremlin Causing chaos in prod since

    2009. Previously SRE Manager @ Dropbox leading Databases, Block Storage and Code Workflows for 500 million users and 800 engineers. @tammybütow % @tammybütow #QCONNYC
  11. GREMLIN • We are practitioners of Chaos Engineering • We

    build software that helps engineers build resilient systems in a safe, secure and simple way. • We offer 11 ways to inject chaos for your Chaos Engineering experiments (e.g. host/container packet loss and shutdown) @tammybütow #QCONNYC
  12. PART 1: LAYING THE FOUNDATION @tammybütow #QCONNYC

  13. • A resilient system is a highly available and durable

    system. • A resilient system can maintain an acceptable level of service in the face of failure. • A resilient system can weather the storm (a misconfiguration, a large scale natural disaster or controlled chaos engineering). @tammybütow #QCONNYC Let’s Define A Resilient System:
  14. It would be silly to give an Olympic pole-vaulter a

    broom and ban them from practicing! @tammybütow #QCONNYC
  15. “Thoughtful planned experiments designed to reveal the weaknesses in our

    systems” - Kolton Andrus, Gremlin CEO @tammybütow, Gremlin @tammybütow #QCONNYC
  16. Inject something harmful in order to build an immunity. @tammybütow,

    Gremlin @tammybütow #QCONNYC Think of it like a vaccination:
  17. Eventually systems will break in many undesired ways. Break them

    first on purpose with controlled chaos! @tammybütow #QCONNYC
  18. DOGFOODING • Using your own product. • For us that

    means using Gremlin for our Chaos Engineering experiments. • Failure Fridays @tammybütow #QCONNYC
  19. Failure Fridays are dedicated time for teams to collaboratively focus

    on using Chaos Engineering practices to reveal weaknesses in your services. @tammybütow #QCONNYC
  20. WHY DO DISTRIBUTED SYSTEMS NEED CHAOS? • Unusual hard to

    debug failures are common • Systems & companies scale rapidly and Chaos Engineering helps you learn along the way @tammybütow #QCONNYC
  21. FULL-STACK CHAOS ENGINEERING • You can inject chaos at any

    layer. • API, App, Cache, Database, OS, Host, Network, Power & more. @tammybütow #QCONNYC
  22. WHY RUN CHAOS ENGINEERING EXPERIMENTS? @tammybütow #QCONNYC

  23. Are you confident that your metrics and alerting are as

    good as they should be? #pagerpain @tammybütow #QCONNYC
  24. Are you confident your customers are getting as good an

    experience as they should be? #customerpain @tammybütow #QCONNYC
  25. Are you losing money due to downtime and broken features?

    #businesspain @tammybütow #QCONNYC
  26. HOW DO YOU RUN CHAOS ENGINEERING EXPERIMENTS? @tammybütow #QCONNYC

  27. HOW TO RUN A CHAOS ENGINEERING EXPERIMENT • Form a

    hypothesis • Consider blast radius • Run experiment • Measure results • Find & fix issues or scale ⚡ @tammybütow #QCONNYC
  28. Don’t run before you can walk @tammybütow, Gremlin @tammybütow #QCONNYC

  29. The 3 Prerequisites for Chaos Engineering @tammybütow, Gremlin @tammybütow #QCONNYC

    1. Monitoring & Observability 2. On-Call & Incident Management 3. Know Your Cost of Downtime Per Hour
  30. What Do I Use For Monitoring & Observability? @tammybütow, Gremlin

    @tammybütow #QCONNYC
  31. We All Need To Know The Cost Of Downtime @tammybütow,

    Gremlin @tammybütow #QCONNYC
  32. We All Need Incident Management @tammybütow, Gremlin @tammybütow #QCONNYC

  33. HOW TO CHOOSE A CHAOS EXPERIMENT • Identify top 5

    critical systems • Choose 1 system • Whiteboard the system • Select attack: resource/ state/network • Determine scope ⚡ @tammybütow #QCONNYC
  34. WHAT SHOULD WE MEASURE? • Availability — 500s • Service

    specific KPIs • System metrics: CPU, IO, Disk • Customer complaints @tammybütow #QCONNYC
  35. HOW TO RUN YOUR OWN GAMEDAY! @tammybütow #QCONNYC gremlin.com/gameday

  36. HOW TO RUN YOUR OWN GAMEDAY! @tammybütow #QCONNYC gremlin.com/gameday

  37. EXAMPLE SYSTEM: KUBERNETES RETAIL STORE User Primary: kube-01 Node: kube-02

    Node: kube-03 Node: kube-04 @tammybütow #QCONNYC
  38. PART 2: RESOURCE CHAOS ENGINEERING @tammybütow #QCONNYC

  39. We can increase CPU, Disk, IO & Memory consumption to

    ensure monitoring is setup to catch problems. Important to catch issues before they turn into high severity incidents (unable to purchase new product!) and downtime for customers. RESOURCE CHAOS @tammybütow #QCONNYC
  40. CPU CHAOS @tammybütow #QCONNYC

  41. https://github.com/tammybutow/chaosengineeringbootcamp LET’S CREATE A “KNOWN-KNOWN” EXPERIMENT @tammybütow #QCONNYC

  42. CHAOS IN TOP @tammybütow #QCONNYC

  43. LET’S KILL THE CHAOS NOW @tammybütow #QCONNYC

  44. NO MORE CHAOS IN TOP @tammybütow #QCONNYC

  45. DISK CHAOS @tammybütow #QCONNYC

  46. DISK CHAOS @tammybütow #QCONNYC

  47. MEMORY CHAOS @tammybütow #QCONNYC

  48. MEMORY CHAOS free -m @tammybütow #QCONNYC

  49. PART 3: STATE CHAOS ENGINEERING @tammybütow #QCONNYC

  50. PROCESS CHAOS @tammybütow #QCONNYC

  51. Ways to create process chaos on purpose: PROCESS CHAOS •

    Kill one process • Loop kill a process • Spawn new processes • Fork bomb @tammybütow #QCONNYC
  52. PROCESS CHAOS pkill -u chaos @tammybütow #QCONNYC

  53. SHUTDOWN CHAOS @tammybütow #QCONNYC

  54. SHUTDOWN CHAOS shutdown -h @tammybütow #QCONNYC

  55. WHAT ARE OTHER WAYS YOU CAN TURN OFF A SERVER?

    WHAT IF YOU WANT TO TURN OFF EVERY SERVER WHEN IT’S ONE WEEK OLD? @tammybütow #QCONNYC
  56. HALT, REBOOT & POWEROFF CHAOS halt @tammybütow #QCONNYC

  57. WHAT ABOUT SHUTTING DOWN
 CONTAINERS AND K8’S PODS? @tammybütow #QCONNYC

  58. THE MANY WAYS TO KILL CONTAINERS • Kill self •

    Kill a container from the host • Use one container to kill another • Use one container to kills several containers • Use several containers to kill several @tammybütow #QCONNYC
  59. The average lifespan of a container is 2.5 days And

    they fail in many unexpected ways. @tammybütow #QCONNYC
  60. TIME TRAVEL CHAOS @tammybütow #QCONNYC

  61. TIME TRAVEL CHAOS AKA CLOCK SKEW ntpq @tammybütow #QCONNYC

  62. PART 4: NETWORK CHAOS ENGINEERING @tammybütow #QCONNYC

  63. BLACKHOLE CHAOS @tammybütow #QCONNYC

  64. BLACKHOLE CHAOS ip route show @tammybütow #QCONNYC

  65. DNS CHAOS @tammybütow #QCONNYC

  66. DNS CHAOS @tammybütow #QCONNYC

  67. DNS CHAOS @tammybütow #QCONNYC

  68. LATENCY CHAOS @tammybütow #QCONNYC

  69. LATENCY CHAOS mtr google.com @tammybütow #QCONNYC

  70. PACKET LOSS CHAOS @tammybütow #QCONNYC

  71. PACKET LOSS CHAOS @tammybütow #QCONNYC

  72. PART 5: COMPLEX OUTAGES @tammybütow #QCONNYC

  73. We can combine different types of chaos engineering experiments to

    reproduce complicated outages. Reproducing outages gives you confidence you can handle it if/when it happens again. @tammybütow #QCONNYC
  74. Let’s go back in time to look at some of

    the worst outage stories that kicked off the introduction of chaos engineering. @tammybütow #QCONNYC
  75. DROPBOX’S WORST OUTAGE EVER Some master-replica pairs were impacted which

    resulted in the site going down. https://blogs.dropbox.com/tech/2014/01/outage-post-mortem/ @tammybütow #QCONNYC
  76. UBER’S DATABASE OUTAGE 1.Master log replication to S3 failed 2.Logs

    backed up on the primary 3.Alerts fired to engineer but they are ignored 4.Disk fills up on database primary 5.Engineer deletes unarchived WAL files 6.Error in config prevents promotion — Matt Ranney, Uber, 2015 @tammybütow #QCONNYC
  77. OUTAGES HAPPEN. @tammybütow #QCONNYC

  78. THERE ARE MANY MORE OUTAGES YOU CAN READ ABOUT HERE:

    https://github.com/danluu/post-mortems @tammybütow #QCONNYC
  79. HOW CAN YOU CONTINUE YOUR CHAOS ENGINEERING JOURNEY? @tammybütow #QCONNYC

  80. JOIN THE CHAOS SLACK GREMLIN.COM/SLACK @tammybütow #QCONNYC

  81. LEARN WITH THE GREMLIN COMMUNITY GREMLIN.COM/COMMUNITY @tammybütow #QCONNYC

  82. THE FIRST CHAOS ENGINEERING CONFERENCE! CHAOSCONF.IO @tammybütow #QCONNYC

  83. THANK YOU QCON NYC @tammybütow #CHAOSENGINEERING