Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
Building Adaptive Systems
Chris Keathley
May 28, 2020
Programming
25
1.1k
Building Adaptive Systems
Chris Keathley
May 28, 2020
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Contracts for building reliable systems
keathley
3
400
Kafka, the hard parts
keathley
1
880
Building Resilient Elixir Systems
keathley
5
1.2k
Consistent, Distributed Elixir
keathley
4
980
Telling stories with data visualization
keathley
0
280
Easing into continuous deployment
keathley
1
110
Leveling up your git skills
keathley
0
410
Generative Testing in Elixir
keathley
0
200
Concurrent Feature tests with Wallaby
keathley
2
400
Other Decks in Programming
See All in Programming
Reinventing the wheel ... as a service
mariofusco
3
260
Kueue入門/Kueue Introduction
bells17
0
510
Loom is Blooming
josepaumard
3
540
JGS594 Lecture 23
javiergs
PRO
0
400
Composing an API with Kotlin (Kotlin Dev Day 2022)
zsmb
0
260
Hapticをカスタマイズしてみよう / ZOZO Tech Talk #6 Customize Haptic
endoumari
0
310
Airflow1=>Airflow2へのupgrade 事例紹介
reizist
0
100
職場にPythonistaを増やす方法
soogie
0
310
Jetpack Compose 頑張らないPreviewParameterProvider
horie23
0
110
Learning DDD輪読会#4 / Learning DDD Book Club #4
suzushin54
1
140
microCMS × Shopifyで、ECサイトがリニューアル後急成長した話
microcms
0
470
The future of trust stores in Python
sethmlarson
0
180
Featured
See All Featured
The Brand Is Dead. Long Live the Brand.
mthomps
45
2.7k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
268
11k
Testing 201, or: Great Expectations
jmmastey
21
5.4k
The Invisible Customer
myddelton
110
11k
Learning to Love Humans: Emotional Interface Design
aarron
261
37k
The Straight Up "How To Draw Better" Workshop
denniskardys
225
120k
Product Roadmaps are Hard
iamctodd
34
6.1k
Intergalactic Javascript Robots from Outer Space
tanoku
261
25k
Optimizing for Happiness
mojombo
365
63k
A Philosophy of Restraint
colly
192
14k
Atom: Resistance is Futile
akmur
255
20k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
3
440
Transcript
Chris Keathley / @ChrisKeathley / c@keathley.io Building Adaptive Systems
Server Server
Server Server I have a request
Server Server
Server Server
Server Server No Problem!
Server Server
Server Server Thanks!
Server Server
Server Server I have a request
Server Server
Server Server
Server Server I’m a little busy
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I don’t feel so good
Server
Server Welp
Server Welp
All services have objectives
A resilient service should be able to withstand a 10x
traffic spike and continue to meet those objectives
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
What causes overload?
What causes overload? Server Queue
What causes overload? Server Queue Processing Time Arrival Rate >
Little’s Law Elements in the queue = Arrival Rate *
Processing Time
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes CPU Pressure
Little’s Law Server 3 requests = 10 rps * 300
ms 300ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * 3000
ms 3000ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * ∞
ms ∞ BEAM Processes CPU Pressure
Little’s Law 30 requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
This is bad
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Overload Arrival Rate > Processing Time
Overload Arrival Rate > Processing Time We need to get
these under control
Load Shedding Server Queue Server
Load Shedding Server Queue Server Drop requests
Load Shedding Server Queue Server Drop requests Stop sending
Autoscaling
Autoscaling
Autoscaling Server DB Server
Autoscaling Server DB Server Requests start queueing
Autoscaling Server DB Server Server
Autoscaling Server DB Server Server Now its worse
Autoscaling needs to be in response to load shedding
Circuit Breakers
Circuit Breakers
Circuit Breakers Server Server
Circuit Breakers Server Server
Circuit Breakers Server Server Shut off traffic
Circuit Breakers Server Server
Circuit Breakers Server Server I’m not quite dead yet
Circuit Breakers are your last line of defense
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
We want to allow as many requests as we can
actually handle
None
Adaptive Limits Time Concurrency
Adaptive Limits Actual limit Time Concurrency
Adaptive Limits Actual limit Dynamic Discovery Time Concurrency
Load Shedding Server Server
Load Shedding Server Server Are we at the limit?
Load Shedding Server Server Am I still healthy?
Load Shedding Server Server
Load Shedding Server Server Update Limits
Adaptive Limits Time Concurrency Increased latency
Latency Successful vs. Failed requests Signals for Adjusting Limits
Additive Increase Multiplicative Decrease Success state: limit + 1 Backoff
state: limit * 0.95 Time Concurrency
Prior Art/Alternatives https://github.com/ferd/pobox/ https://github.com/fishcakez/sbroker/ https://github.com/heroku/canal_lock https://github.com/jlouis/safetyvalve https://github.com/jlouis/fuse
Regulator https://github.com/keathley/regulator
Regulator.install(:service, [ limit: {Regulator.Limit.AIMD, [timeout: 500]} ]) Regulator.ask(:service, fn ->
{:ok, Finch.request(:get, "https://keathley.io")} end) Regulator
Conclusion
Queues are everywhere
Those queues need to be bounded to avoid overload
If your system is dynamic, your solution will also need
to be dynamic
Go and build awesome stuff
Thanks Chris Keathley / @ChrisKeathley / c@keathley.io