Building Adaptive Systems

Transcript

1. 1.

   Chris Keathley / @ChrisKeathley / c@keathley.io Building Adaptive Systems

2. 2.

   Server Server

3. 3.

   Server Server I have a request

4. 4.

   Server Server

5. 5.

   Server Server

6. 6.

   Server Server No Problem!

7. 7.

   Server Server

8. 8.

   Server Server Thanks!

9. 9.

   Server Server

10. 10.

    Server Server I have a request

11. 11.

    Server Server

12. 12.

    Server Server

13. 13.

    Server Server I’m a little busy

14. 14.

    Server Server I’m a little busy I have more requests!

15. 15.

    Server Server I’m a little busy I have more requests!

16. 16.

    Server Server I’m a little busy I have more requests!

17. 17.

    Server Server I’m a little busy I have more requests!

18. 18.

    Server Server I’m a little busy I have more requests!

19. 19.

    Server Server I’m a little busy I have more requests!

20. 20.

    Server Server I’m a little busy I have more requests!

21. 21.

    Server Server I’m a little busy I have more requests!

22. 22.

    Server Server I don’t feel so good

23. 23.

    Server

24. 24.

    Server Welp

25. 25.

    Server Welp

26. 26.

    All services have objectives

27. 27.

    A resilient service should be able to withstand a 10x

    traffic spike and continue to meet those objectives
28. 28.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

29. 29.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

30. 30.

    What causes overload?

31. 31.

    What causes overload? Server Queue

32. 32.

    What causes overload? Server Queue Processing Time Arrival Rate >

33. 33.

    Little’s Law Elements in the queue = Arrival Rate *

    Processing Time
34. 34.

    Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms
35. 35.

    Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms
36. 36.

    Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms
37. 37.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms
38. 38.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms
39. 39.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms
40. 40.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms
41. 41.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms
42. 42.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms BEAM Processes
43. 43.

    Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms BEAM Processes CPU Pressure
44. 44.

    Little’s Law Server 3 requests = 10 rps * 300

    ms 300ms BEAM Processes CPU Pressure
45. 45.

    Little’s Law Server 30 requests = 10 rps * 3000

    ms 3000ms BEAM Processes CPU Pressure
46. 46.

    Little’s Law Server 30 requests = 10 rps * ∞

    ms ∞ BEAM Processes CPU Pressure
47. 47.

    Little’s Law 30 requests = 10 rps * ∞ ms

48. 48.

    Little’s Law ∞ requests = 10 rps * ∞ ms

49. 49.

    Little’s Law ∞ requests = 10 rps * ∞ ms

    This is bad
50. 50.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

51. 51.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

52. 52.

    Overload Arrival Rate > Processing Time

53. 53.

    Overload Arrival Rate > Processing Time We need to get

    these under control
54. 54.

    Load Shedding Server Queue Server

55. 55.

    Load Shedding Server Queue Server Drop requests

56. 56.

    Load Shedding Server Queue Server Drop requests Stop sending

57. 57.

    Autoscaling

58. 58.

    Autoscaling

59. 59.

    Autoscaling Server DB Server

60. 60.

    Autoscaling Server DB Server Requests start queueing

61. 61.

    Autoscaling Server DB Server Server

62. 62.

    Autoscaling Server DB Server Server Now its worse

63. 63.

    Autoscaling needs to be in response to load shedding

64. 64.

    Circuit Breakers

65. 65.

    Circuit Breakers

66. 66.

    Circuit Breakers Server Server

67. 67.

    Circuit Breakers Server Server

68. 68.

    Circuit Breakers Server Server Shut off traffic

69. 69.

    Circuit Breakers Server Server

70. 70.

    Circuit Breakers Server Server I’m not quite dead yet

71. 71.

    Circuit Breakers are your last line of defense

72. 72.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

73. 73.

    Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

74. 74.

    We want to allow as many requests as we can

    actually handle
75. 75.

    None
76. 76.

    Adaptive Limits Time Concurrency

77. 77.

    Adaptive Limits Actual limit Time Concurrency

78. 78.

    Adaptive Limits Actual limit Dynamic Discovery Time Concurrency

79. 79.

    Load Shedding Server Server

80. 80.

    Load Shedding Server Server Are we at the limit?

81. 81.

    Load Shedding Server Server Am I still healthy?

82. 82.

    Load Shedding Server Server

83. 83.

    Load Shedding Server Server Update Limits

84. 84.

    Adaptive Limits Time Concurrency Increased latency

85. 85.

    Latency Successful vs. Failed requests Signals for Adjusting Limits

86. 86.

    Additive Increase Multiplicative Decrease Success state: limit + 1 Backoff

    state: limit * 0.95 Time Concurrency
87. 87.

    Prior Art/Alternatives https://github.com/ferd/pobox/ https://github.com/fishcakez/sbroker/ https://github.com/heroku/canal_lock https://github.com/jlouis/safetyvalve https://github.com/jlouis/fuse

88. 88.

    Regulator https://github.com/keathley/regulator

89. 89.

    Regulator.install(:service, [ limit: {Regulator.Limit.AIMD, [timeout: 500]} ]) Regulator.ask(:service, fn ->

    {:ok, Finch.request(:get, "https://keathley.io")} end) Regulator
90. 90.

    Conclusion

91. 91.

    Queues are everywhere

92. 92.

    Those queues need to be bounded to avoid overload

93. 93.

    If your system is dynamic, your solution will also need

    to be dynamic
94. 94.

    Go and build awesome stuff

95. 95.

    Thanks Chris Keathley / @ChrisKeathley / c@keathley.io