Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Building Adaptive Systems
Search
Chris Keathley
May 28, 2020
Programming
3.3k
44
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Building Adaptive Systems
Chris Keathley
May 28, 2020
Transcript
95 slides
Chris Keathley / @ChrisKeathley /
[email protected]
Building Adaptive Systems
Server Server
Server Server I have a request
Server Server
Server Server
Server Server No Problem!
Server Server
Server Server Thanks!
Server Server
Server Server I have a request
Server Server
Server Server
Server Server I’m a little busy
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I’m a little busy I have more requests!
Server Server I don’t feel so good
Server
Server Welp
Server Welp
All services have objectives
A resilient service should be able to withstand a 10x
traffic spike and continue to meet those objectives
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
What causes overload?
What causes overload? Server Queue
What causes overload? Server Queue Processing Time Arrival Rate >
Little’s Law Elements in the queue = Arrival Rate *
Processing Time
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 1 requests = 10 rps * 100
ms 100ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes
Little’s Law Server 2 requests = 10 rps * 200
ms 200ms BEAM Processes CPU Pressure
Little’s Law Server 3 requests = 10 rps * 300
ms 300ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * 3000
ms 3000ms BEAM Processes CPU Pressure
Little’s Law Server 30 requests = 10 rps * ∞
ms ∞ BEAM Processes CPU Pressure
Little’s Law 30 requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
Little’s Law ∞ requests = 10 rps * ∞ ms
This is bad
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Overload Arrival Rate > Processing Time
Overload Arrival Rate > Processing Time We need to get
these under control
Load Shedding Server Queue Server
Load Shedding Server Queue Server Drop requests
Load Shedding Server Queue Server Drop requests Stop sending
Autoscaling
Autoscaling
Autoscaling Server DB Server
Autoscaling Server DB Server Requests start queueing
Autoscaling Server DB Server Server
Autoscaling Server DB Server Server Now its worse
Autoscaling needs to be in response to load shedding
Circuit Breakers
Circuit Breakers
Circuit Breakers Server Server
Circuit Breakers Server Server
Circuit Breakers Server Server Shut off traffic
Circuit Breakers Server Server
Circuit Breakers Server Server I’m not quite dead yet
Circuit Breakers are your last line of defense
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
Lets Talk About… Queues Overload Mitigation Adaptive Concurrency
We want to allow as many requests as we can
actually handle
None
Adaptive Limits Time Concurrency
Adaptive Limits Actual limit Time Concurrency
Adaptive Limits Actual limit Dynamic Discovery Time Concurrency
Load Shedding Server Server
Load Shedding Server Server Are we at the limit?
Load Shedding Server Server Am I still healthy?
Load Shedding Server Server
Load Shedding Server Server Update Limits
Adaptive Limits Time Concurrency Increased latency
Latency Successful vs. Failed requests Signals for Adjusting Limits
Additive Increase Multiplicative Decrease Success state: limit + 1 Backoff
state: limit * 0.95 Time Concurrency
Prior Art/Alternatives https://github.com/ferd/pobox/ https://github.com/fishcakez/sbroker/ https://github.com/heroku/canal_lock https://github.com/jlouis/safetyvalve https://github.com/jlouis/fuse
Regulator https://github.com/keathley/regulator
Regulator.install(:service, [ limit: {Regulator.Limit.AIMD, [timeout: 500]} ]) Regulator.ask(:service, fn ->
{:ok, Finch.request(:get, "https://keathley.io")} end) Regulator
Conclusion
Queues are everywhere
Those queues need to be bounded to avoid overload
If your system is dynamic, your solution will also need
to be dynamic
Go and build awesome stuff
Thanks Chris Keathley / @ChrisKeathley /
[email protected]
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
5
1.2k
Contracts for building reliable systems
keathley
6
1.1k
Kafka, the hard parts
keathley
3
2.1k
Building Resilient Elixir Systems
keathley
7
2.6k
Consistent, Distributed Elixir
keathley
6
1.7k
Telling stories with data visualization
keathley
1
760
Easing into continuous deployment
keathley
2
500
Leveling up your git skills
keathley
0
910
Generative Testing in Elixir
keathley
0
670
Other Decks in Programming
See All in Programming
Spec-driven Development with AI
simas
PRO
0
2.3k
TanStack Vue Query と Pinia Colada から学ぶ非同期状態の型設計
aoshi
0
680
Kawaiicon: Kubernetes, it's no longer cool, but let's secure it anyway!
barnbarn
0
130
そのデザイントークン、Flutterでも同じ見た目になりますか?
nakamizu
0
370
社内AIエージェント「Askワンバン」における コストチューニングの取り組み
t0yohei
0
170
Vue.jsで作る空間解析アプリとCapacitorプラグインのData Bridge
hal_spidernight
1
170
ローカルLLMの基本と近況 2611 / Local LLM basics and recent situation 2611
kishida
2
490
まずはコードより業務と向き合う:テーブル解体のためにPdMを呼んだ話
sakawe_ee
0
240
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
2
170
Web3Dライブラリを作ってVue Custom Rendererの仕組みを理解しよう / lets learn vue custom renderer by making 3d library
drumath2237
1
140
GraphQL の onError と Nullability のこれから
quramy
0
220
動画プレイヤーはなぜカクつくのか 〜Flutterのレンダリングとネイティブの境界を解剖する〜
naruogram
0
340
Featured
See All Featured
We Analyzed 250 Million AI Search Results: Here's What I Found
joshbly
2
2.1k
SEO in 2025: How to Prepare for the Future of Search
ipullrank
3
3.9k
So, you think you're a good person
axbom
PRO
2
2.3k
Music & Morning Musume
bryan
48
7.5k
Lightning talk: Run Django tests with GitHub Actions
sabderemane
0
320
We Have a Design System, Now What?
morganepeng
55
8.4k
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
How to Ace a Technical Interview
jacobian
280
24k
Utilizing Notion as your number one productivity tool
mfonobong
4
640
Abbi's Birthday
coloredviolet
5
11k
Large-scale JavaScript Application Architecture
addyosmani
516
110k
Keith and Marios Guide to Fast Websites
keithpitt
413
23k