Circuit Breaking in Distributed Systems

Presented at ProgSCon 2015

Design patterns such as the CircuitBreaker can be used to prevent cascading failures across multiple systems caused by failing components. These features are especially important in microservice architectures, where requests may traverse many systems, and services may retry calls to other services, potentially resulting in failure cascades which multiply traffic to failing systems and hinder their recovery. In this talk we look at how circuit breakers and other similar approaches can be used to increase reliability in distributed systems such as Go based microservice platforms.

Image Credits:
Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq
Network Cables - Steven Kreuzer: http://goo.gl/QS0Q0K
MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk
Stopwatch - William Warby: http://goo.gl/LZZBb6
Shoot Again - Stéfan: http://goo.gl/TCgg8R
Hystrix diagrams via Netflix: http://goo.gl/Gul9n4

Transcript

1. CIRCUIT

   BREAKING
   

   View Slide

2. @mattheath
   

   View Slide

3. View Slide

4. DATABASE
   APPLICATION
   

   View Slide

5. DATABASE
   APPLICATION
   

   View Slide

6. DATABASE
   DATABASES
   APPLICATION
   CACHE
   SEARCH
   CAT GIFS
   

   View Slide

7. ALL
   HAIL
   THE
   MONOLITH
   

   View Slide

8. View Slide

9. MONOLITH
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. FALLACIES OF
    DISTRIBUTED
    COMPUTING
    

    View Slide

22. DEALING

    WITH FAILURE
    

    View Slide

23. TIMING OUT
    AND MOVING ON
    

    View Slide

24. SENSIBLE

    TIMEOUTS?
    

    View Slide

25. MEASURE
    EVERYTHING
    

    View Slide

26. View Slide

27. TIMEOUT?
    

    View Slide

28. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

29. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

30. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

31. api service.signup service.profile
    api service.signup service.profile
    ?
    ??
    

    View Slide

32. api service.signup service.profile
    api service.signup service.profile
    ?
    

    View Slide

33. View Slide

34. View Slide

35. FAST RETRY

    FAILURES?
    

    View Slide

36. MAKE
    PROBLEMS
    WORSE.
    

    View Slide

37. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

38. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

39. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

40. DEFERRED
    RETRIES?
    

    View Slide

41. EVENT DRIVEN

    ARCHITECTURES
    

    View Slide

42. CIRCUIT

    BREAKERS
    

    View Slide

43. View Slide

44. api service.signup service.profile
    api service.signup service.profile
    

    View Slide

45. api service.signup service.profile
    api service.signup service.profile
    Server
    Client
    Logic
    

    View Slide

46. api service.signup service.profile
    api service.signup service.profile
    Server
    Client
    Logic
    

    View Slide

47. Client
    Remote Server
    Circuit Breaker
    

    View Slide

48. Client
    Remote Server
    Circuit Breaker
    Error! Error!
    

    View Slide

49. Client
    Remote Server
    Circuit Breaker
    Timeout! Timeout!
    

    View Slide

50. Client
    Remote Server
    Circuit Breaker
    Circuit
    Open!
    OPEN
    

    View Slide

51. View Slide

52. View Slide

53. View Slide

54. View Slide

55. View Slide

56. View Slide

57. View Slide

58. client := circuit.NewHTTPClient(
    time.Second * 5, // timeout
    10, // threshold
    nil // http.Client
    )
    resp, err := client.Get(
    “http://beesbeesbees.com/"
    )
    

    View Slide

59. cb := circuit.NewRateBreaker(
    0.95, // trip at 95% errors
    100, // with 100 samples
    )
    cb.Call(func() error {
    // do something
    }, time.Second * 5)
    

    View Slide

60. MONITORING
    

    View Slide

61. DISTRIBUTED

    TRACING
    

    View Slide

62. Switch & Fuses - The Q Speaks: http://goo.gl/RVglEq
    Network Cables - Steven Kreuzer: http://goo.gl/QS0Q0K
    MSC Napoli - Gary Tanner: http://goo.gl/Yyoatk
    Stopwatch - William Warby: http://goo.gl/LZZBb6
    Shoot Again - Stéfan: http://goo.gl/TCgg8R
    Hystrix diagrams via Netflix: http://goo.gl/Gul9n4
    THANKS
    

    View Slide