Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scale like a pro
Search
Python Porto
December 14, 2017
Programming
1
270
Scale like a pro
Distributed task processing with Python and Celery
Python Porto
December 14, 2017
Tweet
Share
More Decks by Python Porto
See All by Python Porto
Detecting phishing with Recurrent Neural Networks
pyporto
0
44
Quick and Robust API with Django Rest Framework
pyporto
1
340
Django as your data management framework
pyporto
1
1.2k
Can my computer make jokes
pyporto
0
110
Building a serverless cloud service
pyporto
0
58
Python Porto #10. Past, present and future
pyporto
0
86
Entertaining testing with pytest
pyporto
0
190
Joyful Python Web App development with Appier
pyporto
0
160
Other Decks in Programming
See All in Programming
JetBrainsのAI機能の紹介 #jjug
yusuke
0
190
11年かかって やっとVibe Codingに 時代が追いつきましたね
yimajo
1
240
MCP連携で加速するAI駆動開発/mcp integration accelerates ai-driven-development
bpstudy
0
270
バイブコーディング超えてバイブデプロイ〜CloudflareMCPで実現する、未来のアプリケーションデリバリー〜
azukiazusa1
3
780
MySQL9でベクトルカラム登場!PHP×AWSでのAI/類似検索はこう変わる
suguruooki
1
280
decksh - a little language for decks
ajstarks
4
21k
GUI操作LLMの最新動向: UI-TARSと関連論文紹介
kfujikawa
0
490
Jakarta EE Meets AI
ivargrimstad
0
590
QA x AIエコシステム段階構築作戦
osu
0
240
新しいモバイルアプリ勉強会(仮)について
uetyo
1
250
MCPで実現できる、Webサービス利用体験について
syumai
7
2.4k
大規模FlutterプロジェクトのCI実行時間を約8割削減した話
teamlab
PRO
0
450
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
The Cult of Friendly URLs
andyhume
79
6.5k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6k
Side Projects
sachag
455
43k
Designing for Performance
lara
610
69k
Agile that works and the tools we love
rasmusluckow
329
21k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Mobile First: as difficult as doing things right
swwweet
223
9.9k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
540
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.6k
Transcript
Scale like a pro Distributed computing with message queues and
Python Roman Imankulov | Python Porto | December 2017
Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
browser ./webserver.py request POST /obj/<id> obj.update()
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send
email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” … response
None
browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py
update search indexes send email “object updated” update business analytics
Message queues
None
None
queue
frontends queue
frontends queue workers
frontends queue job 1 workers
frontends queue job 1 workers job 2
frontends queue job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 get() workers job 2 job 3
frontends queue get() workers job 2 job 3 job 1
frontends queue workers job 2 job 3 job 1
frontends queue workers get() job 2 job 3 job 1
frontends queue workers get() job 3 job 1 job 2
frontends queue workers job 3 job 1 job 2
frontends queue workers job 3 job 2
frontends queue workers job 3 job 2 get()
frontends queue workers job 2 job 3 get()
frontends queue workers job 2 job 3
frontends queue workers
Queue in python
Multiprocessing
None
None
None
None
None
None
None
Celery The queue out of the box
None
None
None
None
None
Celery Workflows chains, groups and chords
Task signatures
Task signatures
Task signatures
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Celery Extras out of the box
None
• Different backends
• Different backends • Different serializers
• Different backends • Different serializers • Callbacks / errbacks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
None
facebook.com/pyporto