Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Celery - Distributed task queue system
Search
Mushfiq-E Mahabub
November 28, 2014
Programming
0
53
Celery - Distributed task queue system
Mushfiq-E Mahabub
November 28, 2014
Tweet
Share
More Decks by Mushfiq-E Mahabub
See All by Mushfiq-E Mahabub
Kanban as a Software Development Process
mushfiq
0
150
Kick Start your freelance career.
mushfiq
0
91
Other Decks in Programming
See All in Programming
Kotlin Multiplatform at Stable and Beyond (Android Makers 2024)
zsmb
0
440
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
27
8.5k
Introducing Kotlin Multiplatform in an existing mobile app - Workshop Edition | AndroidMakers Paris
prof18
0
140
PHPの次期バージョンはこの時期どうなっているのか - Internalsの開発体制について - PHPカンファレンス小田原
youkidearitai
PRO
1
220
検証も兼ねて個人開発でHonoとかと向き合った話
hanetsuki
1
1.3k
新宿ダンジョンを可視化してみた
satoshi7190
3
380
0→1と1→10の狭間で Javaという技術選定を振り返る/Reflecting on the Decision to Choose Java Between Scaling from 0 to 1 and 1 to 10
jaguar_imo
2
400
Polars入門
daikikatsuragawa
1
160
Tailwind CSSを本気でカスタマイズする方法
fsubal
14
5.5k
SwiftUIで使いやすいToastの作り方 / How to build a Toast system which is easy to use in SwiftUI
lovee
3
170
2 週間で Twitter Bot を作ってみた
contour_gara
0
760
Micro Frontends for Java Microservices - Utah JUG 2024
mraible
PRO
1
110
Featured
See All Featured
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
What's new in Ruby 2.0
geeforr
337
31k
No one is an island. Learnings from fostering a developers community.
thoeni
16
2.1k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
22
1.6k
Design by the Numbers
sachag
274
18k
Designing with Data
zakiwarfel
96
4.8k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
20
1.7k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
The Mythical Team-Month
searls
216
42k
Building Adaptive Systems
keathley
32
1.9k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
660
120k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
Transcript
Celery Distributed task queue system 1
Outline ! • What ? • Why ? • When
? • Who? • How ? 2
What? “Celery is a simple, flexible and reliable distributed system
to process vast amounts of messages, while providing operations with the tools required to maintain such a system.” Official Celery Docs. ! “Asynchronous task management system” 3
What? • Tasks • Worker • Broker 4
Tasks • Tasks can be created by any callable •
do something (?) when a worker receives the message • do something (?) when it is called 5
Worker • message consumer • execute tasks 6
Broker • message storage • executes task • store the
result (optional) 7
Architecture 8
When? • For background processing • Distributed computing • Eg.
Bulk email sending • Eg. Content aggregation • Eg. Long running data processing jobs • Eg. Periodic task 9
Who uses? 10
Who else? 11
Hello World? 12
Real World? 13
Common Issues • How to lock? • How to have
results? • Pythonic Cron? • How to distribute? • How to execute dependent task? 14
How to lock? 15
How to save result? 16
Cron (Pythonic)? 17
Distribute • start workers in separate machine • use (same/a
new one) message queue 18
Dependent tasks 19
Demo? 20
Image Sources • http://thenewstack.io/wp-content/uploads/2014/06/ redis.png • https://shiori2309.files.wordpress.com/2014/06/ what-not-to-say-to-mixed-race-4.gif • http://3.bp.blogspot.com/-9pKvE5OCXeM/
UtTnAGXrylI/AAAAAAAAQ_0/MlXbQnQqpAk/ s1600/rabbit_header_logo.jpg 21