Remote Calls != Local Calls @ DevOps Days Boston 2015

Graceful degradation when services fail Remote Calls != Local Calls

@danriti Dan Riti Senior Software Engineer @ Dan Riti @danriti
[email protected] github.com/danriti

@danriti Dan Riti Monolithic

@danriti Dan Riti Monolithic Services

@danriti Dan Riti Dependencies

@danriti Dan Riti Points of Failure

@danriti Dan Riti Remote Calls != Local Calls

@danriti Dan Riti Q: What approaches support graceful degradation when
(services, networks, data_stores) fail?

@danriti Dan Riti Q: What approaches support graceful degradation when
(services, networks, data_stores) fail? A: 1. Timeouts 2. Circuit Breaker Pattern 3. Retries 4. Bulkhead Pattern

@danriti Dan Riti 1. Timeouts ◦ Forcing an error when
a dependency is unhealthy 2. Circuit Breaker Pattern ◦ Prevent operations when a dependency is unhealthy 3. Retries ◦ Forcing extra attempts where extra latency is acceptable if a recovery provides more value 4. Bulkhead Pattern ◦ Partitioning a system to enforce the principle of damage containment

@danriti Dan Riti Time Service User Service Web App

@danriti Dan Riti Time Service User Service Web App RESTful
RESTful

@danriti Dan Riti Time Service User Service Web App RESTful
RESTful HTTP HTTP

@danriti Dan Riti Time Service Response { "time": "2015-09-11T17:33:48.940483" }
User Service Response { "name": "Dan Riti" }

@danriti Dan Riti

@danriti Dan Riti web app

@danriti Dan Riti web app time service

@danriti Dan Riti web app time service user service

@danriti Dan Riti Q: How should we degrade when the
time_service is unavailable?

@danriti Dan Riti Q: How should we degrade when the
time_service is unavailable? A: Present “Unavailable” to user Give up on requests after 3 seconds Provide fault isolation

@danriti Dan Riti https://github.com/danriti/short-circuit

@danriti Dan Riti “Your code can't just wait forever for
a response that might never come sooner or late, it needs to give up. Hope is not a design method.” - Michael T. Nygard, Release It! Timeouts

@danriti Dan Riti

@danriti Dan Riti response = requests.get('http://localhost:3001/time')

@danriti Dan Riti response = requests.get('http://localhost:3001/time', timeout=3.0)

@danriti Dan Riti RTFM

@danriti Dan Riti Time Service User Service https://github.com/danriti/short-circuit The time
service is unhealthy Web App

@danriti Dan Riti 3.01 s

@danriti Dan Riti 3.01 s Broken Pipe

@danriti Dan Riti Timeouts are not perfect Easy to get
started with Provides some fault isolation Response bound to timeout value Still applying load to unhealthy service(s)

@danriti Dan Riti Circuit Breaker Pattern

@danriti Dan Riti Circuit Breaker Pattern • “Allow one subsystem
(an electrical circuit) to fail (excessive current draw) without destroying the entire system (the house)” • “Once the danger has passed, the circuit breaker can be reset to restore full function to the system” • “This differs from retries, in that circuit breakers exist to prevent operations rather then re-execute them” - Michael T. Nygard, Release It!

@danriti Dan Riti

@danriti Dan Riti Circuit Breaker Pattern • Release It! by
Michael T. Nygard (2007) ◦ https://pragprog.com/book/mnee/release-it • Netflix (2011) ◦ http://techblog.netflix.com/2011/12/making-netflix-api-more-resilient.html • Martin Fowler (2014) ◦ http://martinfowler.com/bliki/CircuitBreaker.html

@danriti Dan Riti

@danriti Dan Riti time_breaker = pybreaker.CircuitBreaker(fail_max=3, reset_timeout=30)

@danriti Dan Riti time_breaker = pybreaker.CircuitBreaker( fail_max=3, reset_timeout=30)

@danriti Dan Riti Time Service User Service https://github.com/danriti/short-circuit The system
is healthy Web App

@danriti Dan Riti # Request Time Service User Service Circuit
Breaker State 1. GET / 50 ms 50 ms Closed

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure)

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure)

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure) 6. GET / 3 s (timeout) 50 ms Closed => Open (3 Failure)

@danriti Dan Riti Time Service User Service https://github.com/danriti/short-circuit The time
service is unhealthy Web App

@danriti Dan Riti Time Service User Service https://github.com/danriti/short-circuit The circuit
breaker is open Web App

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure) 6. GET / 3 s (timeout) 50 ms Closed => Open (3 Failure)

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure) 6. GET / 3 s (timeout) 50 ms Closed => Open (3 Failure) 7. GET / Prevented 50 ms Open

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure) 6. GET / 3 s (timeout) 50 ms Closed => Open (3 Failure) 7. GET / Prevented 50 ms Open 8. GET / Prevented 50 ms Open

@danriti Dan Riti 6 ms

Breaker State 8. GET / Prevented 50 ms Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open ~30 seconds passed, so reset timeout triggers Open => Half-Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 10. GET / 3 s (timeout) 50 ms Half-Open => Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 10. GET / 3 s (timeout) 50 ms Half-Open => Open 11. GET / Prevented 50 ms Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 10. GET / 3 s (timeout) 50 ms Half-Open => Open 11. GET / Prevented 50 ms Open 12. GET / Prevented 50 ms Open

Breaker State 13. GET / Prevented 50 ms Open

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open ~30 seconds passed, so reset timeout triggers Open => Half-Open

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 14. GET / 50 ms 50 ms Half-Open => Closed

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 14. GET / 50 ms 50 ms Half-Open => Closed 15. GET / 50 ms 50 ms Closed

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 14. GET / 50 ms 50 ms Half-Open => Closed 15. GET / 50 ms 50 ms Closed 16. GET / 50 ms 50 ms Closed

@danriti Dan Riti 14 ms

@danriti Dan Riti Timeouts + Circuit Breaker Pattern • Graceful
degradation of user experience • Fail fast and rapidly recover • Reduces load on unhealthy service • Avoid unhealthy service affecting the system • (Bonus) Interface to monitor and measure points of integration in a system

@danriti Dan Riti Timeouts + Circuit Breaker Pattern • Under
provisioned services can cause “flapping” ◦ Service is overwhelmed due to load ◦ Circuit breaker trips to “open” ◦ Service no longer receiving requests, so it “recovers” ◦ Circuit breaker “closes” • Important to understand service performance constraints

@danriti Dan Riti Parameters for Timeouts + Circuit Breaker •
What do you present during failure? • How many times do you accept failure? (max_fails) • How long until you attempt reset? (reset_timeout) • How long will you wait? (timeout)

@danriti Dan Riti

@danriti Dan Riti Holiday Weekend

@danriti Dan Riti Circuit Breaker Libraries • Python - https://github.com/danielfm/pybreaker
• Go - https://github.com/rubyist/circuitbreaker • Java - https://github.com/Netflix/Hystrix • Ruby - https://github.com/wsargent/circuit_breaker • .NET - https://github.com/michael-wolfenden/Polly • Javascript - https://github.com/yammer/circuit-breaker-js • PHP - https://github.com/ejsmont-artur/php-circuit-breaker

@danriti Dan Riti https://github.com/Netflix/Hystrix/wiki/Dashboard

@danriti Dan Riti Thank You @danriti

@danriti Dan Riti Backup

@danriti Dan Riti Source: https://github.com/Netflix/Hystrix/wiki

Breaker State 1. GET / 50 ms 50 ms Closed 2. GET / 1 s 50 ms Closed 3. GET / 2 s 50 ms Closed 4. GET / 3 s (timeout) 50 ms Closed (1 Failure) 5. GET / 3 s (timeout) 50 ms Closed (2 Failure) 6. GET / 3 s (timeout) 50 ms Closed => Open (3 Failure) 7. GET / Skipped 50 ms Open 8. GET / Skipped 50 ms Open

Breaker State 8. GET / Prevented 50 ms Open 9. GET / Prevented 50 ms Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 10. GET / 3 s (timeout) 50 ms Half-Open => Open 11. GET / Prevented 50 ms Open 12. GET / Prevented 50 ms Open

Breaker State 13. GET / Prevented 50 ms Open Ops saves the day, so time service is healthy Open ~30 seconds passed, so reset timeout triggers Open => Half-Open 14. GET / 50 ms 50 ms Half-Open => Closed 15. GET / 50 ms 50 ms Closed 16. GET / 50 ms 50 ms Closed

@danriti Dan Riti Fallacies of Distributed Computing 1. The network
is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4. The network is secure. 5. Topology doesn't change. 6. There is one administrator. 7. Transport cost is zero. 8. The network is homogeneous. https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

@danriti Dan Riti

Remote Calls != Local Calls @ DevOps Days Bosto...

Remote Calls != Local Calls @ DevOps Days Boston 2015

More Decks by Dan Riti

Other Decks in Programming

Featured

Transcript