Background Task Processing in Celery

Transcript

1. Background Task Processing Using Celery Abu Ashraf Masnun
2. Tasks that take time • The case of image uploading and resizing ◦ We need to upload image ◦ We need to resize it to several different dimensions • Should we handle it in the http handler? ◦ Increases response time ◦ We should focus on more important tasks (ie. authentication) ◦ Let’s defer image resizing to background
3. Solutions? Deferring the task to background • Cron Jobs ◦ Store the image data in database / data store ◦ Run scheduled cron jobs ◦ Scaling could be hard between multiple servers ◦ Overlapping cron jobs could lead to race conditions • Message Queues and Workers ◦ A queue stores the image data ◦ Multiple workers (across different servers) can consume data one by one ◦ Uses pub-sub pattern
4. Resizing images • Web Endpoint ◦ Handles the upload ◦ Sends a message to a defined queue with file location and other parameters • Message Broker ◦ Routes the message to the consumer aka background worker • Background worker ◦ Receives the message ◦ Takes the image file, resizes according to parameters ◦ Puts it back
5. Meet Celery Celery abstracts these things into a convenient setup. • Give celery a message broker (say: redis) • Have celery workers always running • Write the tasks as Python functions / classes • Wrap the function in a decorator • In your web end point, call the function (slightly differently) Celery does what needs to be done behind the scene
6. Github Repository https://github.com/masnun/devcon-celery-demo
7. Demo
8. Learn More • Flower • Periodic Task • Task routing • Designing work flows • Signals There’s so much more you can do: https://docs.celeryproject.org/en/stable/
9. Questions? Ask in Python Bangladesh group - https://www.facebook.com/groups/pythonbd - get faster answers, from people with more advanced knowledge than me. Reach out to me - masnun@gmail.com