Increasing confidence in your system before the next outage

As our systems are growing more complex, we’ve come up with a standard set of ideas to help manage the dependencies between components. Ideas like timeouts, concurrency bounds, and circuit breakers let us control these interactions.

However, these ideas are complex too. Each of them has a variety of knobs to tune, and they’re rarely exercised in steady-state. So there’s a lot of opportunity for them to behave poorly when called upon, and especially when a few of them kick in at once.

My talk will show how to use chaos engineering to increase confidence in your usage and tuning of timeouts, concurrency bounds, and circuit breakers. You can use these ideas to understand your system better and hopefully make future incidents more manageable.

Transcript

17. recommendations-demo
23. • Fail Recommendations Service at 10% • Expectation is that overall service will fail at 10%, since we haven’t done any work to remediate (yet)
31. •
33. •
36. • Now fallback data from S3 is being served whenever a Users OR Recommendations failure is encountered • Fail Recommendations Service at 10% • Expectation is that customers get 90% personalized responses, 10% unpersonalized responses, and no errors
40. • •
43. • What’s the maximum we want the customer to wait? • How many resources can we spend per-request waiting?
44. • Threads ◦ • File Descriptors • Locks held • Memory • CPU scheduler
49. •
52. • Add 7500ms to 100% of Recommendations Service • Expectation is that resources will be spent waiting for response - some will succeed and some will fail
60. • Add concurrency bounds to users and recommendations • Add 7500ms to 100% of Recommendations Service • Expectation is that resources will be spent waiting for response - but service will remain healthy. • No errors, but fallback logic should get hit
65. • •
66. •
67. • Explicit state transitions are a useful way to think about the problem. • Can fail even faster than timeouts • Manual triggering of circuit open/closed is really valuable for operators • Relieves pressure on downstreams
68. • Lot of configuration to set and validate • If timeouts are already being used, that should be sufficient for user-facing latency • If concurrency-bounds are already being used, that should be sufficient for system stability • Can introduce many more failures that would happen naturally ◦ • Which failures open? ◦ ◦
69. • ◦ ◦ ◦
72. • UI resiliency - what happens when certain endpoints fail or are slow? • Finding hidden dependencies • Testing your monitoring • Testing your incident response • Testing your autoscaling rules • Testing your region evacuation strategy • Onboarding new SREs • (many more)
74. • Gremlin.com • https://github.com/dastergon/awesome-chaos-engineering •