Pro Yearly is on sale from $80 to $50! »

Circuit Breaking in Distributed Systems

67f4a8f2a209a38d7242829947b26ba3?s=47 mattheath
April 17, 2015

Circuit Breaking in Distributed Systems

Presented at ProgSCon 2015

Design patterns such as the CircuitBreaker can be used to prevent cascading failures across multiple systems caused by failing components. These features are especially important in microservice architectures, where requests may traverse many systems, and services may retry calls to other services, potentially resulting in failure cascades which multiply traffic to failing systems and hinder their recovery. In this talk we look at how circuit breakers and other similar approaches can be used to increase reliability in distributed systems such as Go based microservice platforms.

Image Credits:
Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq
Network Cables - Steven Kreuzer: http://goo.gl/QS0Q0K
MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk
Stopwatch - William Warby: http://goo.gl/LZZBb6
Shoot Again - Stéfan: http://goo.gl/TCgg8R
Hystrix diagrams via Netflix: http://goo.gl/Gul9n4

67f4a8f2a209a38d7242829947b26ba3?s=128

mattheath

April 17, 2015
Tweet

Transcript

  1. CIRCUIT
 BREAKING

  2. @mattheath

  3. None
  4. DATABASE APPLICATION

  5. DATABASE APPLICATION

  6. DATABASE DATABASES APPLICATION CACHE SEARCH CAT GIFS

  7. ALL HAIL THE MONOLITH

  8. None
  9. MONOLITH

  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. None
  19. None
  20. None
  21. FALLACIES OF DISTRIBUTED COMPUTING

  22. DEALING
 WITH FAILURE

  23. TIMING OUT AND MOVING ON

  24. SENSIBLE
 TIMEOUTS?

  25. MEASURE EVERYTHING

  26. None
  27. TIMEOUT?

  28. api service.signup service.profile api service.signup service.profile

  29. api service.signup service.profile api service.signup service.profile

  30. api service.signup service.profile api service.signup service.profile

  31. api service.signup service.profile api service.signup service.profile ? ??

  32. api service.signup service.profile api service.signup service.profile ?

  33. None
  34. None
  35. FAST RETRY
 FAILURES?

  36. MAKE PROBLEMS WORSE.

  37. api service.signup service.profile api service.signup service.profile

  38. api service.signup service.profile api service.signup service.profile

  39. api service.signup service.profile api service.signup service.profile

  40. DEFERRED RETRIES?

  41. EVENT DRIVEN
 ARCHITECTURES

  42. CIRCUIT
 BREAKERS

  43. None
  44. api service.signup service.profile api service.signup service.profile

  45. api service.signup service.profile api service.signup service.profile Server Client Logic

  46. api service.signup service.profile api service.signup service.profile Server Client Logic

  47. Client Remote Server Circuit Breaker

  48. Client Remote Server Circuit Breaker Error! Error!

  49. Client Remote Server Circuit Breaker Timeout! Timeout!

  50. Client Remote Server Circuit Breaker Circuit Open! OPEN

  51. None
  52. None
  53. None
  54. None
  55. None
  56. None
  57. None
  58. client := circuit.NewHTTPClient( time.Second * 5, // timeout 10, //

    threshold nil // http.Client ) resp, err := client.Get( “http://beesbeesbees.com/" )
  59. cb := circuit.NewRateBreaker( 0.95, // trip at 95% errors 100,

    // with 100 samples ) cb.Call(func() error { // do something }, time.Second * 5)
  60. MONITORING

  61. DISTRIBUTED
 TRACING

  62. Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq Network Cables

    - Steven Kreuzer: http://goo.gl/QS0Q0K MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk Stopwatch - William Warby: http://goo.gl/LZZBb6 Shoot Again - Stéfan: http://goo.gl/TCgg8R Hystrix diagrams via Netflix: http://goo.gl/Gul9n4 THANKS