The Shocking Truth About Circuit Breakers

Circuit Breakers: we know them & love them, right? Honestly, I'm not sure. For those of us lucky enough to have used them, we understand why we need them. But do we fully understand how they work? What problem are they actually trying to solve? Can we communicate this to our organization in way that they can jump on board? What happens when they go off? Shocking plot twist: I love Circuit Breakers. They are a secret sauce to solving the murder mystery that is your microservices architecture.

The Demo Project can be found at https://github.com/anfleene/circuit_breaker_demo

Transcript

1. @andyfleener The Shocking Truth about Circuit Breakers By Andy Fleener
2. @andyfleener I work at We use Circuit Breakers
3. @andyfleener WTF is a Circuit Breaker?
4. @andyfleener I’m so glad you asked
5. @andyfleener Resilience is the Future* *it might not be but I’m an optimist
6. @andyfleener Network Latency is a Distributed System’s Kryptonite It’s a thing because: physics
7. @andyfleener Circuit Breakers are the Lead Box to contain the Kryptonite
8. @andyfleener Cue Martin Fowler wrap a protected function call in a circuit breaker object Monitors for failures Failures reach a certain threshold, the circuit breaker trips Calls to the circuit breaker return with an error, without the protected call being made at all Eventually a timer expires and attempts the call again If success go back to 1 If failure go back to 3
9. @andyfleener The pattern was first named in Michael Nygard’s Release It in 2007 Yes, that’s 10 years ago.
10. @andyfleener Cool User Story Bro
11. @andyfleener You’re probably thinking one of 3 things right now.
12. @andyfleener 1. I should wrap every network call in a Circuit Breaker I’m going to start that tomorrow!
13. @andyfleener 2. I don’t need this, my network is reliable
14. @andyfleener 3. Was that a Mr. Robot Gif on the title slide?
15. @andyfleener Sure those are all perfectly valid thoughts. I’m here to tell you it’s not that simple
16. @andyfleener At SportsEngine we started using Circuit Breakers just last year.
17. @andyfleener We’re still trying to figure it out
18. @andyfleener But Andy it’s been out for 10 years what were you doing for the other 9?
19. @andyfleener I was learning hard lessons about Complex Systems
20. @andyfleener Complex Systems Cliff Notes They are unsafe They will always be unsafe Failure is inevitable Embrace the failure
21. @andyfleener We started our journey from Monolith to Microservices in 2011
22. @andyfleener We learned quickly that services became highly dependent on each other
23. @andyfleener Enter the Burning Ring of SOA
24. @andyfleener It works for Netflix! Boom! Let’s use circuit breakers!
25. @andyfleener Realtalk: It’s hard for a Systems Thinker to communicate with Non Systems Thinkers
26. @andyfleener What seems like an obvious solution to us doesn’t sound obvious to everyone
27. @andyfleener The hardest part about using circuit breakers was explaining why we needed to use them
28. @andyfleener The biggest reason is that Resilience is counterintuitive.
29. @andyfleener You have to fail to succeed? WTF are you talking about?
30. @andyfleener Talk in terms of outcomes: How many outages happened because of a cascading failure?
31. @andyfleener What if only that first services failed, how many customers would that have impacted
32. Ok awesome, lets fail on purpose! Now What? Prioritize based on impact by identifying central services that would amplify a cascade Circuits are only effective at the service level picking and choosing still leads to #cryparty Use an abstraction, make the right thing easy. You shouldn’t have to reinvent the wheel on every api call
33. @andyfleener Ok sweet what does that look like?
34. @andyfleener Demo Time!
35. Finding a Library: Someone has implemented this pattern in any language used for the web They may or may not be using it in production Test. Test. Test. Timeouts are a primitive of a Circuit Breaker they must be built into it.
36. Testing Tools: Comcast: https://github.com/tylertreat/comcast Toxiproxy: https://github.com/Shopify/toxiproxy Iptables + tc Sleeps ¯\_(ツ)_/¯
37. @andyfleener Thanks!