Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Adaptive Systems
Search
Chris Keathley
May 28, 2020
Programming
40
2.4k
Building Adaptive Systems
Chris Keathley
May 28, 2020
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
4
1k
Contracts for building reliable systems
keathley
5
830
Kafka, the hard parts
keathley
2
1.6k
Building Resilient Elixir Systems
keathley
6
2.1k
Consistent, Distributed Elixir
keathley
5
1.5k
Telling stories with data visualization
keathley
0
590
Easing into continuous deployment
keathley
2
350
Leveling up your git skills
keathley
0
720
Generative Testing in Elixir
keathley
0
490
Other Decks in Programming
See All in Programming
Datadog DBMでなにができる? JDDUG Meetup#7
nealle
0
160
「その気にさせる」エンジニアが 最強のリーダーになる理由
gimupop
1
260
Django NinjaによるAPI開発の効率化とリプレースの実践
kashewnuts
1
310
Rubyと自由とAIと
yotii23
6
2k
Amazon Bedrockマルチエージェントコラボレーションを諦めてLangGraphに入門してみた
akihisaikeda
1
190
もう少しテストを書きたいんじゃ〜 #phpstudy
o0h
PRO
21
4.5k
読まないコードリーディング術
hisaju
1
150
Functional APIから再考するLangGraphを使う理由
os1ma
4
280
ABEMA iOS 大規模プロジェクトにおける段階的な技術刷新 / ABEMA iOS Technology Upgrade
akkyie
1
270
RecSys2024 参加報告
unonao
1
110
気がついたら子供が社会人になって 自分と同じモバイルアプリエンジニアになった件 / Parent-Child Engineers
koishi
0
150
DevNexus - Create AI Infused Java Apps with LangChain4j
kdubois
0
150
Featured
See All Featured
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Being A Developer After 40
akosma
89
590k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
The Language of Interfaces
destraynor
156
24k
Building Applications with DynamoDB
mza
93
6.3k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
We Have a Design System, Now What?
morganepeng
51
7.4k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
11
1.3k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Side Projects
sachag
452
42k
Music & Morning Musume
bryan
46
6.4k
Transcript
Chris Keathley / @ChrisKeathley / c@keathley.io Building Adaptive Systems
Server Server
Server Server I have a request
Server Server
Server Server
Server Server No Problem!
Server Server
Server Server Thanks!
Server Server
Server Server I have a request
Server Server
Server Server
Server Server I’m a little busy
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I don’t feel so good
Server
Server Welp
Server Welp
All services have objectives
A resilient service should be able to withstand a 10x
traffic spike and continue to meet those objectives
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
What causes overload?
What causes overload? Server Queue
What causes overload? Server Queue Processing Time Arrival Rate >
Little’s Law Elements in the queue = Arrival Rate *
Processing Time
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes CPU Pressure
Little’s Law Server 3 requests = 10 rps * 300
ms 300ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * 3000
ms 3000ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * ∞
ms ∞ BEAM Processes CPU Pressure
Little’s Law 30 requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
This is bad
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Overload Arrival Rate > Processing Time
Overload Arrival Rate > Processing Time We need to get
these under control
Load Shedding Server Queue Server
Load Shedding Server Queue Server Drop requests
Load Shedding Server Queue Server Drop requests Stop sending
Autoscaling
Autoscaling
Autoscaling Server DB Server
Autoscaling Server DB Server Requests start queueing
Autoscaling Server DB Server Server
Autoscaling Server DB Server Server Now its worse
Autoscaling needs to be in response to load shedding
Circuit Breakers
Circuit Breakers
Circuit Breakers Server Server
Circuit Breakers Server Server
Circuit Breakers Server Server Shut off traffic
Circuit Breakers Server Server
Circuit Breakers Server Server I’m not quite dead yet
Circuit Breakers are your last line of defense
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
We want to allow as many requests as we can
actually handle
None
Adaptive Limits Time Concurrency
Adaptive Limits Actual limit Time Concurrency
Adaptive Limits Actual limit Dynamic Discovery Time Concurrency
Load Shedding Server Server
Load Shedding Server Server Are we at the limit?
Load Shedding Server Server Am I still healthy?
Load Shedding Server Server
Load Shedding Server Server Update Limits
Adaptive Limits Time Concurrency Increased latency
Latency Successful vs. Failed requests Signals for Adjusting Limits
Additive Increase Multiplicative Decrease Success state: limit + 1 Backoff
state: limit * 0.95 Time Concurrency
Prior Art/Alternatives https://github.com/ferd/pobox/ https://github.com/fishcakez/sbroker/ https://github.com/heroku/canal_lock https://github.com/jlouis/safetyvalve https://github.com/jlouis/fuse
Regulator https://github.com/keathley/regulator
Regulator.install(:service, [ limit: {Regulator.Limit.AIMD, [timeout: 500]} ]) Regulator.ask(:service, fn ->
{:ok, Finch.request(:get, "https://keathley.io")} end) Regulator
Conclusion
Queues are everywhere
Those queues need to be bounded to avoid overload
If your system is dynamic, your solution will also need
to be dynamic
Go and build awesome stuff
Thanks Chris Keathley / @ChrisKeathley / c@keathley.io