Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scale like a pro
Search
Python Porto
December 14, 2017
Programming
1
190
Scale like a pro
Distributed task processing with Python and Celery
Python Porto
December 14, 2017
Tweet
Share
More Decks by Python Porto
See All by Python Porto
Detecting phishing with Recurrent Neural Networks
pyporto
0
31
Quick and Robust API with Django Rest Framework
pyporto
1
330
Django as your data management framework
pyporto
1
1k
Can my computer make jokes
pyporto
0
87
Building a serverless cloud service
pyporto
0
40
Python Porto #10. Past, present and future
pyporto
0
56
Entertaining testing with pytest
pyporto
0
160
Joyful Python Web App development with Appier
pyporto
0
130
Other Decks in Programming
See All in Programming
[技育CAMPアカデミア]アイディアを形に!【超入門】スマホアプリ開発〜リリースまでの流れをご紹介
teamlab
PRO
0
360
今、知っておきたい! 生成AIエージェントの世界
elith
3
350
Micro Frontends for Java Microservices - Devnexus 2024
mraible
PRO
0
480
ONE WEDGE_company_guide
1wedge_one
0
450
Tailwind CSSを本気でカスタマイズする方法
fsubal
13
5.1k
"config" ってなんだ? / What is "config"?
okashoi
0
240
Changed Rules: Architectures with Lightweight Stores
manfredsteyer
PRO
0
240
PHPの次期バージョンはこの時期どうなっているのか - Internalsの開発体制について - PHPカンファレンス小田原
youkidearitai
PRO
1
190
Prepare for Jakarta EE 11 - Performance and Developer Productivity
ivargrimstad
0
710
新宿ダンジョンを可視化してみた
satoshi7190
2
240
コードレビューで学ぶ!Kotlinオブジェクト指向デザインパターン
akkie76
2
200
MetricKitで予期せぬ終了を検知する話 / Detect unexpected termination with MetricKit
nekowen
1
180
Featured
See All Featured
Into the Great Unknown - MozCon
thekraken
10
990
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
78
42k
The MySQL Ecosystem @ GitHub 2015
samlambert
243
12k
What's new in Ruby 2.0
geeforr
337
31k
Happy Clients
brianwarren
92
6.4k
Optimising Largest Contentful Paint
csswizardry
8
2.4k
YesSQL, Process and Tooling at Scale
rocio
164
13k
Facilitating Awesome Meetings
lara
42
5.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
320
23k
Stop Working from a Prison Cell
hatefulcrawdad
266
19k
Documentation Writing (for coders)
carmenintech
60
3.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
Transcript
Scale like a pro Distributed computing with message queues and
Python Roman Imankulov | Python Porto | December 2017
Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
browser ./webserver.py request POST /obj/<id> obj.update()
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send
email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” … response
None
browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py
update search indexes send email “object updated” update business analytics
Message queues
None
None
queue
frontends queue
frontends queue workers
frontends queue job 1 workers
frontends queue job 1 workers job 2
frontends queue job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 get() workers job 2 job 3
frontends queue get() workers job 2 job 3 job 1
frontends queue workers job 2 job 3 job 1
frontends queue workers get() job 2 job 3 job 1
frontends queue workers get() job 3 job 1 job 2
frontends queue workers job 3 job 1 job 2
frontends queue workers job 3 job 2
frontends queue workers job 3 job 2 get()
frontends queue workers job 2 job 3 get()
frontends queue workers job 2 job 3
frontends queue workers
Queue in python
Multiprocessing
None
None
None
None
None
None
None
Celery The queue out of the box
None
None
None
None
None
Celery Workflows chains, groups and chords
Task signatures
Task signatures
Task signatures
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Celery Extras out of the box
None
• Different backends
• Different backends • Different serializers
• Different backends • Different serializers • Callbacks / errbacks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
None
facebook.com/pyporto