Scheduling Async Tasks with Python Celery

Slide 1

Slide 1 text

Scheduling Async Task with Python Celery Rain Wu

Slide 2

Slide 2 text

Agenda Introduction to Asynchronous (Async) Tasks How Celery Help Us Manage Async Tasks Deep Dive into Async Tasks Management in a Distributed System

Slide 3

Slide 3 text

Introduction to Asynchronous Tasks Synchronous v.s. Asynchronous Why do we need asynchronous? Drawbacks of Asynchronous Tasks

Slide 4

Slide 4 text

Synchronous v.s. Asynchronous

Slide 5

Slide 5 text

Synchronous Acknowledge Study System design Implement Test Deliver Project Manager

Slide 6

Slide 6 text

Synchronous Acknowledge Study System design Implement Test Deliver Software Engineer … Still designing the manual Designer

Slide 7

Slide 7 text

Asynchronous System design Implement Test Review the PRs Write the documents Chat with your colleague Software Engineer Save the draft Ctrl + S Manual design completed

Slide 8

Slide 8 text

Asynchronous Acknowledge Study System design Implement Test Deliver Save the draft Commit the code Build the artifact Say ok to the PM Store knowledge in mind

Slide 9

Slide 9 text

A Practical Case

Slide 10

Slide 10 text

Youtube-liked Service Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects Youtube-liked Service

Slide 13

Slide 13 text

Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects Store the raw data Youtube-liked Service

Slide 14

Slide 14 text

Why do we need asynchronous?

Slide 15

Slide 15 text

Request Response

Slide 16

Slide 16 text

Request Response

Slide 17

Slide 17 text

Request Response

Slide 18

Slide 18 text

Request Response 危

Slide 19

Slide 19 text

Solutions More Resource Need Money Auto-Scaling Need Time Resource Bidding Not Stable

Slide 20

Slide 20 text

Synchronous Receive the request Execute the logic Reply the response

Slide 21

Slide 21 text

Asynchronous Receive the request Store the Result Reply the response Save the task context Fetch the task and execute Retry if task failed

Slide 22

Slide 22 text

Drawbacks?

Slide 23

Slide 23 text

Users’ Perspective Update my password to 123456 Ok, I’ll do it later …?

Slide 24

Slide 24 text

Developers’ Perspective Client paid NTD$123 to our account Ok, I’ll do it later How is it going now? It’s failed lol~ F**k Today Yesterday

Slide 25

Slide 25 text

Developers’ Perspective Sentry, Redmine Issue Tracker Elasticsearch, AWS Cloudwatch Log Storage Flower Monitor

Slide 26

Slide 26 text

How Celery Help Us Manage Async Tasks Architecture Overview Implement the Configuration and the Tasks A Practical Demo

Slide 27

Slide 27 text

Python Celery Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.

Slide 28

Slide 28 text

$ pipenv install celery Installation $ poetry add celery

Slide 29

Slide 29 text

Request Response Message Broker Server Workers Save the task Fetch the task Result Backend Fetch the result Save the result

Slide 30

Slide 30 text

Message Broker Worker 1 Retrieve Queue 1 Worker 3 Queue 3 Queue 2 Worker 2

Slide 31

Slide 31 text

Source Code https://github.com/RainrainWu/python_async_lab

Slide 32

Slide 32 text

Configuration . ├── __init__.py ├── celery_tasks │ ├── __init__.py │ ├── celery.py │ └── celeryconfig.py

Slide 33

Slide 33 text

External Broker & Storage

Slide 34

Slide 34 text

Message Content Type { “user_uuid”: “09dojq…”, “amount”: “3000”, “channel”: “line_pay”, “fee”: “12” }

Slide 35

Slide 35 text

Routing

Slide 36

Slide 36 text

Annotations

Slide 37

Slide 37 text

Tasks

Slide 38

Slide 38 text

How will it looks like in a project?

Slide 39

Slide 39 text

Deep Dive into Async Tasks Management in a Distributed System Load balancer v.s. message broker Duplicated message Mingle and gossip Thundering herds

Slide 40

Slide 40 text

Load Balancer Load Balancer Servers Load Balancer dispatch workloads to Servers Focus on synchronous, run scheduling algorithm

Slide 41

Slide 41 text

Message Broker Message Broker Workers Servers retrieve workloads from Load Balancer Focus on asynchronous, not actively schedule tasks

Slide 42

Slide 42 text

Load Balancer and Message Broker are not mutually exclusive, they can definitely cooperate very well

Slide 43

Slide 43 text

Understanding the difference between them will be very helpful for you to debug

Slide 44

Slide 44 text

Highest cost and worst performance because it needs complicated mechanism to ensure the delivery status (e.g. Pessimistic lock) Communication in Distributed System Exactly once delivery At most once delivery At least once delivery

Slide 45

Slide 45 text

Highest performance but messages may be lost, not suitable for critical tasks. (Similar to UDP protocol) Communication in Distributed System Exactly once delivery At most once delivery At least once delivery

Slide 46

Slide 46 text

Strike a balance between reliable and performance, but messages may be duplicated. Communication in Distributed System Exactly once delivery At most once delivery At least once delivery

Slide 47

Slide 47 text

Message Broker Message Broker Worker 1 Worker 2 Ready Received Unknown

Slide 48

Slide 48 text

Message Broker Message Broker Worker 1 2 3 1. Retrieve queue for new workload 2. Dispatch message to worker 3. Acknowledge to broker 4. Broker mark the state of message with “received”

Slide 49

Slide 49 text

Message Broker Message Broker Worker 1 1 2 3 1. Worker 1 retrieve queue for new workload 2. Broker dispatch message to worker and mark   it “dispatched” 3. Worker 1 acknowledge to broker, but a   network error occurs Worker 2

Slide 50

Slide 50 text

Message Broker Message Broker Worker 1 1 1. Broker does not receive acknowledgement   until timeout, so the message was revoked Worker 2

Slide 51

Slide 51 text

Message Broker Message Broker Worker 1 1 2 3 Worker 2 1. Worker 2 retrieve queue for new workload 2. Broker dispatch message to worker and mark   it “dispatched” 3. Worker 2 acknowledge to broker 4. Broker mark the state of message with “received”

Slide 52

Slide 52 text

Message Broker Workers Storage (Database) - Maintain anther state machine - Lock the minimal scope

Slide 53

Slide 53 text

Any alternative solution for communication?

Slide 54

Slide 54 text

Mingle & Gossip Message Broker Workers Broadcast vast of event for synchronization - Revoked tasks (Mingle) - Heartbeat (Gossip) - Logical clock (Both)

Slide 55

Slide 55 text

Mingle & Gossip Message Broker Workers All of the event will transmit through the message broker 危

Slide 56

Slide 56 text

CPU RAM Network IO Disk IO

Slide 57

Slide 57 text

The mingle and gossip feature was enabled by default

Slide 58

Slide 58 text

I will suggest you turn off gossip but keep mingle alive since it can avoid duplicated messages via ask revoked tasks

Slide 59

Slide 59 text

And do not make a service serving as broker and result backend at the same time

Slide 60

Slide 60 text

Thundering Herds

Slide 61

Slide 61 text

Thundering Herds Time Task start Lock DB Task finish Unlock DB

Slide 62

Slide 62 text

Thundering Herds Time Task 1~10 start Task 1   lock DB Task 1 finish Task 1   unlock DB Task 2~10 failed to access DB,   schedule for retry after 30 sec 00:00 00:04 00:10 00:06

Slide 63

Slide 63 text

Thundering Herds Time Task 2~10 start Task 2   lock DB Task 2 finish Task 2   unlock DB Task 3~10 failed to access DB,   schedule for retry after 30 sec 00:34 00:38 00:44 00:40

Slide 64

Slide 64 text

Thundering Herds Time Time History will repeat itself Random retry delay can solve this issue

Slide 65

Slide 65 text

You may see something like this on your monitor’s dashboard while monitoring celery tasks amount