Circuit Breaking in Distributed Systems

Presented at ProgSCon 2015

Design patterns such as the CircuitBreaker can be used to prevent cascading failures across multiple systems caused by failing components. These features are especially important in microservice architectures, where requests may traverse many systems, and services may retry calls to other services, potentially resulting in failure cascades which multiply traffic to failing systems and hinder their recovery. In this talk we look at how circuit breakers and other similar approaches can be used to increase reliability in distributed systems such as Go based microservice platforms.

Image Credits:
Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq
Network Cables - Steven Kreuzer: http://goo.gl/QS0Q0K
MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk
Stopwatch - William Warby: http://goo.gl/LZZBb6
Shoot Again - Stéfan: http://goo.gl/TCgg8R
Hystrix diagrams via Netflix: http://goo.gl/Gul9n4

Transcript

1. CIRCUIT
 BREAKING
2. @mattheath
4. DATABASE APPLICATION
5. DATABASE APPLICATION
6. DATABASE DATABASES APPLICATION CACHE SEARCH CAT GIFS
7. ALL HAIL THE MONOLITH
9. MONOLITH
21. FALLACIES OF DISTRIBUTED COMPUTING
22. DEALING
 WITH FAILURE
23. TIMING OUT AND MOVING ON
24. SENSIBLE
 TIMEOUTS?
25. MEASURE EVERYTHING
27. TIMEOUT?
28. api service.signup service.profile api service.signup service.profile
29. api service.signup service.profile api service.signup service.profile
30. api service.signup service.profile api service.signup service.profile
31. api service.signup service.profile api service.signup service.profile ? ??
32. api service.signup service.profile api service.signup service.profile ?
35. FAST RETRY
 FAILURES?
36. MAKE PROBLEMS WORSE.
37. api service.signup service.profile api service.signup service.profile
38. api service.signup service.profile api service.signup service.profile
39. api service.signup service.profile api service.signup service.profile
40. DEFERRED RETRIES?
41. EVENT DRIVEN
 ARCHITECTURES
42. CIRCUIT
 BREAKERS
44. api service.signup service.profile api service.signup service.profile
45. api service.signup service.profile api service.signup service.profile Server Client Logic
46. api service.signup service.profile api service.signup service.profile Server Client Logic
47. Client Remote Server Circuit Breaker
48. Client Remote Server Circuit Breaker Error! Error!
49. Client Remote Server Circuit Breaker Timeout! Timeout!
50. Client Remote Server Circuit Breaker Circuit Open! OPEN
58. client := circuit.NewHTTPClient( time.Second * 5, // timeout 10, // threshold nil // http.Client ) resp, err := client.Get( “http://beesbeesbees.com/" )
59. cb := circuit.NewRateBreaker( 0.95, // trip at 95% errors 100, // with 100 samples ) cb.Call(func() error { // do something }, time.Second * 5)
60. MONITORING
61. DISTRIBUTED
 TRACING
62. Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq Network Cables - Steven Kreuzer: http://goo.gl/QS0Q0K MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk Stopwatch - William Warby: http://goo.gl/LZZBb6 Shoot Again - Stéfan: http://goo.gl/TCgg8R Hystrix diagrams via Netflix: http://goo.gl/Gul9n4 THANKS