Resilient Systems with Circuit Breaker 

Guilherme Cavalcanti github.com/guiocavalcanti 

No content

– Release It! “A resilient system keeps processing transactions, even when there are transient impulses, persistent stresses, or component failures disrupting normal processing.” 

Transactions { Not DB transactions Checkout process User signup 3rd party service communication 

KEEPS Processing { Failures in one transaction affects the others 3rd party service failures 

Transient impulses { Transaction peak Appear on reddid 

Persistent stress { High transaction per minute Christmas, Black Friday, etc 

Release It • Michael Nygard — @mtnygard • Resilience • Capacity • Deployment e Monitoring 

Resilience patterns • Timeouts • Bulkheads • Fail fast • Handshking • Circuit Breaker 

Circuit Breaker • Nowadays it’s common to have apps composed by many other apps communicating via API • 3rd party services call will fail • Failure chain 

Example App App App NGINX Service x x x 

Client Circuit Breaker Service Timeout Timeout 

Naive implementation 

Estados Closed Open Threshold reached Success one time failure 

Initialization class CircuitBreaker! ! def initialize(options = {})! @timeout = options.fetch(:timeout, 0.01)! @failure_threshold = options.fetch(:failure_threshold, 5)! @silent = false! @failure_count = 0! end 

#handle • Strategy pattern • Circuit state • Try to execute and capture exceptions def handle(&block)! case state! when :closed then try_to_execute &block! when :open then handle_open! end! end 

#try to… • Failure counter • Tri timeout • Reset counter def try_to_execute(&block)! begin! yield_with_timeout(&block)! reset! rescue Timeout::Error! record_failure! raise $!! end! end! ! def yield_with_timeout(&b)! Timeout::timeout(@timeout, &b)! end! ! def reset! @failure_count = 0! end! ! def record_failure! @failure_count += 1! end! 

#STATE ! def state! (@failure_count >= @failure_threshold) ? :open : :closed! end 

Using require "circuit_breaker"! ! circuit = CircuitBreaker.new! conn = Faraday.new! ! circuit.handle do! conn.get('http://api.foo.com/me.json')! end! ! circuit.handle do! conn.get('http://api.foo.com/accounts.json')! end 

Problem Closed Open Falhas consecutivas Sucesso Falha esporádica Success 

Strategy • Half open state • Come back to closed after a while 

Closed Open threshold reached Success One time failure Half 
 Open Timeout reset Success Failure 

.new class CircuitBreaker! ! def initialize(options = {})! @invocation_timeout = options.fetch(:timeout, 0.01)! @failure_threshold = options.fetch(:failure_threshold, 5)! @silent = false! @failure_count = 0! @reset_timeout = 0.1! @last_failure = nil ! end 

Mudanças • Try to execute • Last failure time • Reset last failure time def handle(&b)! case state! when :closed, :half_open! try_to_execute &b! when :open then handle_open! end! end! ! def try_to_execute(&block)! begin! yield_with_timeout(&block)! reset! rescue Timeout::Error! record_failure! raise $!! end! end! ! def yield_with_timeout(&b)! Timeout::timeout(@timeout, &b)! end! ! def reset! @failure_count = 0! @last_failure_time = nil! end! ! def record_failure! @last_failure_time = Time.now! @failure_count += 1! end! 

#STate def state! case! when (@failure_count >= @failure_threshold) &&! (Time.now - @last_failure_time) > @reset_timeout! :half_open! when (@failure_count >= @failure_threshold)! :open! else! :closed! end! end 

Improvements • Use AAsm or Statesman to manage state machine • Monitoring infrastructure using dependency injection • Distributed Circuit breaker • Detect errors other than timeout 

References • Circuit Breaker, Martin Fowler • Release It, Michael Nygard • Gem Circuit Breaker • Faraday