The Shocking Truth About Circuit Breakers

Slide 1

Slide 1 text

@andyﬂeener The Shocking Truth about Circuit Breakers By Andy Fleener

Slide 2

Slide 2 text

@andyﬂeener I work at We use Circuit Breakers

Slide 3

Slide 3 text

@andyﬂeener WTF is a Circuit Breaker?

Slide 4

Slide 4 text

@andyﬂeener I’m so glad you asked

Slide 5

Slide 5 text

@andyﬂeener Resilience is the Future* *it might not be but I’m an optimist

Slide 6

Slide 6 text

@andyﬂeener Network Latency is a Distributed System’s Kryptonite It’s a thing because: physics

Slide 7

Slide 7 text

@andyﬂeener Circuit Breakers are the Lead Box to contain the Kryptonite

Slide 8

Slide 8 text

@andyﬂeener Cue Martin Fowler wrap a protected function call in a circuit breaker object Monitors for failures Failures reach a certain threshold, the circuit breaker trips Calls to the circuit breaker return with an error, without the protected call being made at all Eventually a timer expires and attempts the call again If success go back to 1 If failure go back to 3

Slide 9

Slide 9 text

@andyﬂeener The pattern was ﬁrst named in Michael Nygard’s Release It in 2007 Yes, that’s 10 years ago.

Slide 10

Slide 10 text

@andyﬂeener Cool User Story Bro

Slide 11

Slide 11 text

@andyﬂeener You’re probably thinking one of 3 things right now.

Slide 12

Slide 12 text

@andyﬂeener 1. I should wrap every network call in a Circuit Breaker I’m going to start that tomorrow!

Slide 13

Slide 13 text

@andyﬂeener 2. I don’t need this, my network is reliable

Slide 14

Slide 14 text

@andyﬂeener 3. Was that a Mr. Robot Gif on the title slide?

Slide 15

Slide 15 text

@andyﬂeener Sure those are all perfectly valid thoughts. I’m here to tell you it’s not that simple

Slide 16

Slide 16 text

@andyﬂeener At SportsEngine we started using Circuit Breakers just last year.

Slide 17

Slide 17 text

@andyﬂeener We’re still trying to ﬁgure it out

Slide 18

Slide 18 text

@andyﬂeener But Andy it’s been out for 10 years what were you doing for the other 9?

Slide 19

Slide 19 text

@andyﬂeener I was learning hard lessons about Complex Systems

Slide 20

Slide 20 text

@andyﬂeener Complex Systems Cliff Notes They are unsafe They will always be unsafe Failure is inevitable Embrace the failure

Slide 21

Slide 21 text

@andyﬂeener We started our journey from Monolith to Microservices in 2011

Slide 22

Slide 22 text

@andyﬂeener We learned quickly that services became highly dependent on each other

Slide 23

Slide 23 text

@andyﬂeener Enter the Burning Ring of SOA

Slide 24

Slide 24 text

@andyﬂeener It works for Netﬂix! Boom! Let’s use circuit breakers!

Slide 25

Slide 25 text

@andyﬂeener Realtalk: It’s hard for a Systems Thinker to communicate with Non Systems Thinkers

Slide 26

Slide 26 text

@andyﬂeener What seems like an obvious solution to us doesn’t sound obvious to everyone

Slide 27

Slide 27 text

@andyﬂeener The hardest part about using circuit breakers was explaining why we needed to use them

Slide 28

Slide 28 text

@andyﬂeener The biggest reason is that Resilience is counterintuitive.

Slide 29

Slide 29 text

@andyﬂeener You have to fail to succeed? WTF are you talking about?

Slide 30

Slide 30 text

@andyﬂeener Talk in terms of outcomes: How many outages happened because of a cascading failure?

Slide 31

Slide 31 text

@andyﬂeener What if only that ﬁrst services failed, how many customers would that have impacted

Slide 32

Slide 32 text

Ok awesome, lets fail on purpose! Now What? Prioritize based on impact by identifying central services that would amplify a cascade Circuits are only effective at the service level picking and choosing still leads to #cryparty Use an abstraction, make the right thing easy. You shouldn’t have to reinvent the wheel on every api call

Slide 33

Slide 33 text

@andyﬂeener Ok sweet what does that look like?

Slide 34

Slide 34 text

@andyﬂeener Demo Time!

Slide 35

Slide 35 text

Finding a Library: Someone has implemented this pattern in any language used for the web They may or may not be using it in production Test. Test. Test. Timeouts are a primitive of a Circuit Breaker they must be built into it.

Slide 36

Slide 36 text

Testing Tools: Comcast: https://github.com/tylertreat/comcast Toxiproxy: https://github.com/Shopify/toxiproxy Iptables + tc Sleeps ¯\_(ツ)_/¯

Slide 37

Slide 37 text

@andyﬂeener Thanks!