Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scale like a pro
Search
Python Porto
December 14, 2017
Programming
1
280
Scale like a pro
Distributed task processing with Python and Celery
Python Porto
December 14, 2017
Tweet
Share
More Decks by Python Porto
See All by Python Porto
Detecting phishing with Recurrent Neural Networks
pyporto
0
44
Quick and Robust API with Django Rest Framework
pyporto
1
350
Django as your data management framework
pyporto
1
1.2k
Can my computer make jokes
pyporto
0
120
Building a serverless cloud service
pyporto
0
58
Python Porto #10. Past, present and future
pyporto
0
88
Entertaining testing with pytest
pyporto
0
200
Joyful Python Web App development with Appier
pyporto
0
160
Other Decks in Programming
See All in Programming
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
420
あなたとKaigi on Rails / Kaigi on Rails + You
shimoju
0
180
React Nativeならぬ"Vue Native"が実現するかも?_新世代マルチプラットフォーム開発フレームワークのLynxとLynxのVue.js対応を追ってみよう_Vue Lynx
yut0naga1_fa
1
150
はじめてのDSPy - 言語モデルを『プロンプト』ではなく『プログラミング』するための仕組み
masahiro_nishimi
3
13k
「ちょっと古いから」って避けてた技術書、今だからこそ読もう
mottyzzz
12
7.1k
開発組織の戦略的な役割と 設計スキル向上の効果
masuda220
PRO
8
1.2k
フロントエンド開発のためのブラウザ組み込みAI入門
masashi
7
3.4k
Introduce Hono CLI
yusukebe
6
3.1k
モテるデスク環境
mozumasu
3
1.1k
Software Architecture
hschwentner
6
2.3k
Devoxx BE - Local Development in the AI Era
kdubois
0
140
PHPに関数型の魂を宿す〜PHP 8.5 で実現する堅牢なコードとは〜 #phpcon_hiroshima / phpcon-hiroshima-2025
shogogg
1
320
Featured
See All Featured
Building an army of robots
kneath
305
46k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
A Tale of Four Properties
chriscoyier
161
23k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
990
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
10
610
Practical Orchestrator
shlominoach
190
11k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
16
1.7k
Building a Scalable Design System with Sketch
lauravandoore
463
33k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
249
1.3M
Side Projects
sachag
455
43k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
34
2.5k
Transcript
Scale like a pro Distributed computing with message queues and
Python Roman Imankulov | Python Porto | December 2017
Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
browser ./webserver.py request POST /obj/<id> obj.update()
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send
email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” … response
None
browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py
update search indexes send email “object updated” update business analytics
Message queues
None
None
queue
frontends queue
frontends queue workers
frontends queue job 1 workers
frontends queue job 1 workers job 2
frontends queue job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 get() workers job 2 job 3
frontends queue get() workers job 2 job 3 job 1
frontends queue workers job 2 job 3 job 1
frontends queue workers get() job 2 job 3 job 1
frontends queue workers get() job 3 job 1 job 2
frontends queue workers job 3 job 1 job 2
frontends queue workers job 3 job 2
frontends queue workers job 3 job 2 get()
frontends queue workers job 2 job 3 get()
frontends queue workers job 2 job 3
frontends queue workers
Queue in python
Multiprocessing
None
None
None
None
None
None
None
Celery The queue out of the box
None
None
None
None
None
Celery Workflows chains, groups and chords
Task signatures
Task signatures
Task signatures
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Celery Extras out of the box
None
• Different backends
• Different backends • Different serializers
• Different backends • Different serializers • Callbacks / errbacks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
None
facebook.com/pyporto