Resilient Systems with Circuit Breaker

It's common for software systems to make remote calls to software running in different processes, probably on different machines across a network. One of the big differences between in-memory calls and remote calls is that remote calls can fail, or hang without a response until some timeout limit is reached. What's worse if you have many callers on a unresponsive supplier, then you can run out of critical resources leading to cascading failures across multiple systems. In his excellent book Release It, Michael Nygard popularized the Circuit Breaker pattern to prevent this kind of catastrophic cascade.

Transcript

1. Resilient Systems with Circuit Breaker
2. Guilherme Cavalcanti github.com/guiocavalcanti
4. – Release It! “A resilient system keeps processing transactions, even when there are transient impulses, persistent stresses, or component failures disrupting normal processing.”
5. Transactions { Not DB transactions Checkout process User signup 3rd party service communication
6. KEEPS Processing { Failures in one transaction affects the others 3rd party service failures
7. Transient impulses { Transaction peak Appear on reddid
8. Persistent stress { High transaction per minute Christmas, Black Friday, etc
9. Release It • Michael Nygard — @mtnygard • Resilience • Capacity • Deployment e Monitoring
10. Resilience patterns • Timeouts • Bulkheads • Fail fast • Handshking • Circuit Breaker
11. Circuit Breaker • Nowadays it’s common to have apps composed by many other apps communicating via API • 3rd party services call will fail • Failure chain
12. Example App App App NGINX Service x x x
13. Client Circuit Breaker Service Timeout Timeout
14. Naive implementation
15. Estados Closed Open Threshold reached Success one time failure
16. Initialization class CircuitBreaker! ! def initialize(options = {})! @timeout = options.fetch(:timeout, 0.01)! @failure_threshold = options.fetch(:failure_threshold, 5)! @silent = false! @failure_count = 0! end
17. #handle • Strategy pattern • Circuit state • Try to execute and capture exceptions def handle(&block)! case state! when :closed then try_to_execute &block! when :open then handle_open! end! end
18. #try to… • Failure counter • Tri timeout • Reset counter def try_to_execute(&block)! begin! yield_with_timeout(&block)! reset! rescue Timeout::Error! record_failure! raise $!! end! end! ! def yield_with_timeout(&b)! Timeout::timeout(@timeout, &b)! end! ! def reset! @failure_count = 0! end! ! def record_failure! @failure_count += 1! end!
19. #STATE ! def state! (@failure_count >= @failure_threshold) ? :open : :closed! end
20. Using require "circuit_breaker"! ! circuit = CircuitBreaker.new! conn = Faraday.new! ! circuit.handle do! conn.get('http://api.foo.com/me.json')! end! ! circuit.handle do! conn.get('http://api.foo.com/accounts.json')! end
21. Problem Closed Open Falhas consecutivas Sucesso Falha esporádica Success
22. Strategy • Half open state • Come back to closed after a while
23. Closed Open threshold reached Success One time failure Half 
 Open Timeout reset Success Failure
24. .new class CircuitBreaker! ! def initialize(options = {})! @invocation_timeout = options.fetch(:timeout, 0.01)! @failure_threshold = options.fetch(:failure_threshold, 5)! @silent = false! @failure_count = 0! @reset_timeout = 0.1! @last_failure = nil ! end
25. Mudanças • Try to execute • Last failure time • Reset last failure time def handle(&b)! case state! when :closed, :half_open! try_to_execute &b! when :open then handle_open! end! end! ! def try_to_execute(&block)! begin! yield_with_timeout(&block)! reset! rescue Timeout::Error! record_failure! raise $!! end! end! ! def yield_with_timeout(&b)! Timeout::timeout(@timeout, &b)! end! ! def reset! @failure_count = 0! @last_failure_time = nil! end! ! def record_failure! @last_failure_time = Time.now! @failure_count += 1! end!
26. #STate def state! case! when (@failure_count >= @failure_threshold) &&! (Time.now - @last_failure_time) > @reset_timeout! :half_open! when (@failure_count >= @failure_threshold)! :open! else! :closed! end! end
27. Improvements • Use AAsm or Statesman to manage state machine • Monitoring infrastructure using dependency injection • Distributed Circuit breaker • Detect errors other than timeout
28. References • Circuit Breaker, Martin Fowler • Release It, Michael Nygard • Gem Circuit Breaker • Faraday