Celery - Distributed task queue system

Transcript

1. Celery Distributed task queue system 1
2. Outline ! • What ? • Why ? • When ? • Who? • How ? 2
3. What? “Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.” Official Celery Docs. ! “Asynchronous task management system” 3
4. What? • Tasks • Worker • Broker 4
5. Tasks • Tasks can be created by any callable • do something (?) when a worker receives the message • do something (?) when it is called 5
6. Worker • message consumer • execute tasks 6
7. Broker • message storage • executes task • store the result (optional) 7
8. Architecture 8
9. When? • For background processing • Distributed computing • Eg. Bulk email sending • Eg. Content aggregation • Eg. Long running data processing jobs • Eg. Periodic task 9
10. Who uses? 10
11. Who else? 11
12. Hello World? 12
13. Real World? 13
14. Common Issues • How to lock? • How to have results? • Pythonic Cron? • How to distribute? • How to execute dependent task? 14
15. How to lock? 15
16. How to save result? 16
17. Cron (Pythonic)? 17
18. Distribute • start workers in separate machine • use (same/a new one) message queue 18
19. Dependent tasks 19
20. Demo? 20
21. Image Sources • http://thenewstack.io/wp-content/uploads/2014/06/ redis.png • https://shiori2309.files.wordpress.com/2014/06/ what-not-to-say-to-mixed-race-4.gif • http://3.bp.blogspot.com/-9pKvE5OCXeM/ UtTnAGXrylI/AAAAAAAAQ_0/MlXbQnQqpAk/ s1600/rabbit_header_logo.jpg 21