Slide 1

Slide 1 text

Background Task Processing Using Celery Abu Ashraf Masnun

Slide 2

Slide 2 text

Tasks that take time ● The case of image uploading and resizing ○ We need to upload image ○ We need to resize it to several different dimensions ● Should we handle it in the http handler? ○ Increases response time ○ We should focus on more important tasks (ie. authentication) ○ Let’s defer image resizing to background

Slide 3

Slide 3 text

Solutions? Deferring the task to background ● Cron Jobs ○ Store the image data in database / data store ○ Run scheduled cron jobs ○ Scaling could be hard between multiple servers ○ Overlapping cron jobs could lead to race conditions ● Message Queues and Workers ○ A queue stores the image data ○ Multiple workers (across different servers) can consume data one by one ○ Uses pub-sub pattern

Slide 4

Slide 4 text

Resizing images ● Web Endpoint ○ Handles the upload ○ Sends a message to a defined queue with file location and other parameters ● Message Broker ○ Routes the message to the consumer aka background worker ● Background worker ○ Receives the message ○ Takes the image file, resizes according to parameters ○ Puts it back

Slide 5

Slide 5 text

Meet Celery Celery abstracts these things into a convenient setup. ● Give celery a message broker (say: redis) ● Have celery workers always running ● Write the tasks as Python functions / classes ● Wrap the function in a decorator ● In your web end point, call the function (slightly differently) Celery does what needs to be done behind the scene

Slide 6

Slide 6 text

Github Repository https://github.com/masnun/devcon-celery-demo

Slide 7

Slide 7 text

Demo

Slide 8

Slide 8 text

Learn More ● Flower ● Periodic Task ● Task routing ● Designing work flows ● Signals There’s so much more you can do: https://docs.celeryproject.org/en/stable/

Slide 9

Slide 9 text

Questions? Ask in Python Bangladesh group - https://www.facebook.com/groups/pythonbd - get faster answers, from people with more advanced knowledge than me. Reach out to me - [email protected]