Celery Python user group meet up 2013

Transcript

1. Celery http://www.celeryproject.org/ ( A python Library ) @ronaldlcheung Tuesday, February 25, 14
2. What is Celery? • Celery is Python Library that performs: • Task Queuing • Asynchronously • Distributed across many machines • Works with Django •Celery is just Python Tuesday, February 25, 14
3. When should we use Celery? • For web / real time applications, Celery can be used for any processing that is better executed Asychonously or in a Non Blocking fashion • Periodic Tasks ( more Pythonic than cron! ) • Expensive or heavy processing you want to do on a separate machine or instance Tuesday, February 25, 14
4. Take a use case • Photo sharing app allows users to upload an album of photos • The photos need to be postprocessed by the app ( resized, cropped, filtered, persisted to storage) • This post processing takes a long time to complete Tuesday, February 25, 14
5. Approach #1 Make the user wait • Http request kept open until processing complete • Bad because the user experience slow response time • Blocks the web server thread - with sufficient users web server will run out of threads Tuesday, February 25, 14
6. Approach #2 Use AJAX • Use a AJAX to call expensive operation while initial response returns quickly • User has a better experience but threads are still blocked Tuesday, February 25, 14
7. Approach #3 Cron Jobs • The photo post processing is persisted somewhere to be processed by a cron job that runs every x minutes • Better, because user has a fast response, threads are not blocked • However user’s input is not processed until the next cycle Tuesday, February 25, 14
8. Approach #4 Celery •Post processing are celery tasks placed on distributed messaging queues • User has fast response, threads not blocked • Post processing starts as soon as user input received • Can be scaled as traffic increases by Tuesday, February 25, 14
9. How does Celery work? Tuesday, February 25, 14
10. Distributed Python / Django AMQP Broker [RabbitMQ] Celery Worker Nodes Results Storage Tuesday, February 25, 14
11. Distributed Architecture Python / Django Results Storage Results Storage Results returned to Python Tuesday, February 25, 14
12. Making a Celery Task from celery.task import task @task(ignore_result=True) def generate_resized_page_images(self,page): page.get_picture_thumbnail_url() page.get_picture_large_url() page.get_picture_medium_url() page.get_picture_normal_url() return 0 Tuesday, February 25, 14
13. Making a Celery Task from celery.task import Task from celery.registry import tasks Class ProcessImages(Task): def run(self,title,**kwargs): page.get_picture_thumbnail_url() page.get_picture_large_url() return 0 tasks.register(ProcessImages) Tuesday, February 25, 14
14. Run the task! from someapp.task import ProcessImages result = ProcessImages.delay(‘Cat picture’) generate_resized_page_images.delay(‘Cat picture’) Tuesday, February 25, 14
15. How to Install Celery • Install an AMQP Broker ( RabbitMQ ) • add AMQP config to settings.py AMQP_SERVER = ‘localhost’ AMQP_PORT = ‘5672’ AMQP_USER = ‘myuser’ AMQP_PASSWORD = ‘password’ AMQP_VHOST = ‘myvhost’ • add celery to INSTALLED_APPS manage.py synchdb • Start Celery: manage.py celeryd Tuesday, February 25, 14
16. Try Celery yourself! • Twitter: @ronaldlcheung • Email: ronaldlcheung@gmail.com Tuesday, February 25, 14