Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scale like a pro
Search
Python Porto
December 14, 2017
Programming
1
280
Scale like a pro
Distributed task processing with Python and Celery
Python Porto
December 14, 2017
Tweet
Share
More Decks by Python Porto
See All by Python Porto
Detecting phishing with Recurrent Neural Networks
pyporto
0
44
Quick and Robust API with Django Rest Framework
pyporto
1
350
Django as your data management framework
pyporto
1
1.2k
Can my computer make jokes
pyporto
0
120
Building a serverless cloud service
pyporto
0
58
Python Porto #10. Past, present and future
pyporto
0
88
Entertaining testing with pytest
pyporto
0
190
Joyful Python Web App development with Appier
pyporto
0
160
Other Decks in Programming
See All in Programming
ABEMAモバイルアプリが Kotlin Multiplatformと歩んだ5年 ─ 導入と運用、成功と課題 / iOSDC 2025
akkyie
0
310
Back to the Future: Let me tell you about the ACP protocol
terhechte
0
120
AccessorySetupKitで実現するシームレスなペアリング体験 / Seamless pairing with AccessorySetupKit
nekowen
0
210
10年もののAPIサーバーにおけるCI/CDの改善の奮闘
mbook
0
690
高度なUI/UXこそHotwireで作ろう Kaigi on Rails 2025
naofumi
4
3k
非同期jobをtransaction内で 呼ぶなよ!絶対に呼ぶなよ!
alstrocrack
0
430
CSS Linter の現在地 2025年のベストプラクティスを探る
ryo_manba
10
3.2k
株式会社 Sun terras カンパニーデック
sunterras
0
180
iOSエンジニア向けの英語学習アプリを作る!
yukawashouhei
0
150
AIを活用したレシート読み取り機能の開発から得られた実践知 / AI Receipt Scan Practice
rockname
2
1.5k
プログラマのための作曲入門
cheebow
0
510
フロントエンド開発に役立つクライアントプログラム共通のノウハウ / Universal client-side programming best practices for frontend development
nrslib
7
3.8k
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
A designer walks into a library…
pauljervisheath
208
24k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
890
Become a Pro
speakerdeck
PRO
29
5.5k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
Embracing the Ebb and Flow
colly
88
4.8k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Large-scale JavaScript Application Architecture
addyosmani
514
110k
Transcript
Scale like a pro Distributed computing with message queues and
Python Roman Imankulov | Python Porto | December 2017
Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
browser ./webserver.py request POST /obj/<id> obj.update()
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send
email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” … response
None
browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py
update search indexes send email “object updated” update business analytics
Message queues
None
None
queue
frontends queue
frontends queue workers
frontends queue job 1 workers
frontends queue job 1 workers job 2
frontends queue job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 get() workers job 2 job 3
frontends queue get() workers job 2 job 3 job 1
frontends queue workers job 2 job 3 job 1
frontends queue workers get() job 2 job 3 job 1
frontends queue workers get() job 3 job 1 job 2
frontends queue workers job 3 job 1 job 2
frontends queue workers job 3 job 2
frontends queue workers job 3 job 2 get()
frontends queue workers job 2 job 3 get()
frontends queue workers job 2 job 3
frontends queue workers
Queue in python
Multiprocessing
None
None
None
None
None
None
None
Celery The queue out of the box
None
None
None
None
None
Celery Workflows chains, groups and chords
Task signatures
Task signatures
Task signatures
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Celery Extras out of the box
None
• Different backends
• Different backends • Different serializers
• Different backends • Different serializers • Callbacks / errbacks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
None
facebook.com/pyporto