Save 37% off PRO during our Black Friday Sale! »

Building Adaptive Systems

06f8b41980eb4c577fa40c41d5030c19?s=47 Chris Keathley
May 28, 2020
Programming
25
950

Building Adaptive Systems

06f8b41980eb4c577fa40c41d5030c19?s=128

Chris Keathley

May 28, 2020
Tweet

More Decks by Chris Keathley

See All by Chris Keathley
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
3
370
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
1
790
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
5
1.1k
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
4
890
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
0
260
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
0
97
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
0
380
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
0
180
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
2
320

Other Decks in Programming

See All in Programming
A1792748885fe8f2555d9c718ee7ae53?s=48 joytomo
1
160
6b8e2101579190ad96e747e01c279898?s=48 pauloxnet
1
130
0ecada41f471c45033ee166c37819915?s=48 artoy
0
180
575ca492bac55e895d0e1c86f7d709fe?s=48 hschwentner
2
690
75486cbfd37125f121cf4a6c5614601c?s=48 hideokamoto
PRO
1
600
Ddecbf99faa27ea6257856734fa6f9b8?s=48 whitegreen
0
160
8f84b7d8869ef6005d89b378e8661f7c?s=48 masuda220
PRO
1
950
82d6167c4d14393c2e20b37a74b363c5?s=48 kasacchiful
0
120
C31b5afbe1ffb5588b2fe5a9e39b357b?s=48 takarasawa_
1
610
F34c64bc7a4bced65306809205aca33c?s=48 nologyance
0
210
Dbab5e5a1fd54531e1119ede6b3b9e65?s=48 horie1024
3
2.7k
A10e41b0a61d59f2258d7f6172c33479?s=48 kaityo256
2
100

Featured

See All Featured
1f74b13f1e5c6c69cb5d7fbaabb1e2cb?s=48 sferik
612
56k
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
11
1.2k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
236
18k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
32
1.2k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
318
19k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
778
200k
E9221b172e78e888355fa2fe4541b595?s=48 mclloyd
PRO
7
7.7k
016edace78ffe826f2348d1e0c504afd?s=48 maggiecrowley
16
950
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
195
9.1k
Af6a12710770ad9a7cfd08c97efd40d3?s=48 pedronauck
655
120k
Ecdea9b9714877b86cee08458f085481?s=48 trallard
27
1.2k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
373
43k

Transcript

  1. Chris Keathley / @ChrisKeathley / c@keathley.io Building Adaptive Systems

  2. Server Server

  3. Server Server I have a request

  4. Server Server

  5. Server Server

  6. Server Server No Problem!

  7. Server Server

  8. Server Server Thanks!

  9. Server Server

  10. Server Server I have a request

  11. Server Server

  12. Server Server

  13. Server Server I’m a little busy

  14. Server Server I’m a little busy I have more requests!

  15. Server Server I’m a little busy I have more requests!

  16. Server Server I’m a little busy I have more requests!

  17. Server Server I’m a little busy I have more requests!

  18. Server Server I’m a little busy I have more requests!

  19. Server Server I’m a little busy I have more requests!

  20. Server Server I’m a little busy I have more requests!

  21. Server Server I’m a little busy I have more requests!

  22. Server Server I don’t feel so good

  23. Server

  24. Server Welp

  25. Server Welp

  26. All services have objectives

  27. A resilient service should be able to withstand a 10x

    traffic spike and continue to meet those objectives

  28. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  29. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  30. What causes overload?

  31. What causes overload? Server Queue

  32. What causes overload? Server Queue Processing Time Arrival Rate >

  33. Little’s Law Elements in the queue = Arrival Rate *

    Processing Time

  34. Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms

  35. Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms

  36. Little’s Law Server 1 requests = 10 rps * 100

    ms 100ms

  37. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms

  38. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms

  39. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms

  40. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms

  41. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms

  42. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms BEAM Processes

  43. Little’s Law Server 2 requests = 10 rps * 200

    ms 200ms BEAM Processes CPU Pressure

  44. Little’s Law Server 3 requests = 10 rps * 300

    ms 300ms BEAM Processes CPU Pressure

  45. Little’s Law Server 30 requests = 10 rps * 3000

    ms 3000ms BEAM Processes CPU Pressure

  46. Little’s Law Server 30 requests = 10 rps * ∞

    ms ∞ BEAM Processes CPU Pressure

  47. Little’s Law 30 requests = 10 rps * ∞ ms

  48. Little’s Law ∞ requests = 10 rps * ∞ ms

  49. Little’s Law ∞ requests = 10 rps * ∞ ms

    This is bad

  50. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  51. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  52. Overload Arrival Rate > Processing Time

  53. Overload Arrival Rate > Processing Time We need to get

    these under control

  54. Load Shedding Server Queue Server

  55. Load Shedding Server Queue Server Drop requests

  56. Load Shedding Server Queue Server Drop requests Stop sending

  57. Autoscaling

  58. Autoscaling

  59. Autoscaling Server DB Server

  60. Autoscaling Server DB Server Requests start queueing

  61. Autoscaling Server DB Server Server

  62. Autoscaling Server DB Server Server Now its worse

  63. Autoscaling needs to be in response to load shedding

  64. Circuit Breakers

  65. Circuit Breakers

  66. Circuit Breakers Server Server

  67. Circuit Breakers Server Server

  68. Circuit Breakers Server Server Shut off traffic

  69. Circuit Breakers Server Server

  70. Circuit Breakers Server Server I’m not quite dead yet

  71. Circuit Breakers are your last line of defense

  72. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  73. Lets Talk About… Queues Overload Mitigation Adaptive Concurrency

  74. We want to allow as many requests as we can

    actually handle
  75. None

  76. Adaptive Limits Time Concurrency

  77. Adaptive Limits Actual limit Time Concurrency

  78. Adaptive Limits Actual limit Dynamic Discovery Time Concurrency

  79. Load Shedding Server Server

  80. Load Shedding Server Server Are we at the limit?

  81. Load Shedding Server Server Am I still healthy?

  82. Load Shedding Server Server

  83. Load Shedding Server Server Update Limits

  84. Adaptive Limits Time Concurrency Increased latency

  85. Latency Successful vs. Failed requests Signals for Adjusting Limits

  86. Additive Increase Multiplicative Decrease Success state: limit + 1 Backoff

    state: limit * 0.95 Time Concurrency

  87. Prior Art/Alternatives https://github.com/ferd/pobox/ https://github.com/fishcakez/sbroker/ https://github.com/heroku/canal_lock https://github.com/jlouis/safetyvalve https://github.com/jlouis/fuse

  88. Regulator https://github.com/keathley/regulator

  89. Regulator.install(:service, [ limit: {Regulator.Limit.AIMD, [timeout: 500]} ]) Regulator.ask(:service, fn ->

    {:ok, Finch.request(:get, "https://keathley.io")} end) Regulator

  90. Conclusion

  91. Queues are everywhere

  92. Those queues need to be bounded to avoid overload

  93. If your system is dynamic, your solution will also need

    to be dynamic

  94. Go and build awesome stuff

  95. Thanks Chris Keathley / @ChrisKeathley / c@keathley.io