Scheduling Async Tasks with Python Celery

- Introduction to Asynchronous (Async) Tasks
- How Celery Help Us Manage Async Tasks
- Deep Dive into Async Tasks Management in a Distributed System

Transcript

1. Scheduling Async Task with Python Celery Rain Wu
2. Agenda Introduction to Asynchronous (Async) Tasks How Celery Help Us Manage Async Tasks Deep Dive into Async Tasks Management in a Distributed System
3. Introduction to Asynchronous Tasks Synchronous v.s. Asynchronous Why do we need asynchronous? Drawbacks of Asynchronous Tasks
4. Synchronous v.s. Asynchronous
5. Synchronous Acknowledge Study System design Implement Test Deliver Project Manager
6. Synchronous Acknowledge Study System design Implement Test Deliver Software Engineer … Still designing the manual Designer
7. Asynchronous System design Implement Test Review the PRs Write the documents Chat with your colleague Software Engineer Save the draft Ctrl + S Manual design completed
8. Asynchronous Acknowledge Study System design Implement Test Deliver Save the draft Commit the code Build the artifact Say ok to the PM Store knowledge in mind
9. A Practical Case
10. Youtube-liked Service Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects
12. Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects Youtube-liked Service
13. Receive video data Respond to client Segmentation Encoding & Compression Post-processing Store playable objects Store the raw data Youtube-liked Service
14. Why do we need asynchronous?
15. Request Response
16. Request Response
17. Request Response
18. Request Response 危
19. Solutions More Resource Need Money Auto-Scaling Need Time Resource Bidding Not Stable
20. Synchronous Receive the request Execute the logic Reply the response
21. Asynchronous Receive the request Store the Result Reply the response Save the task context Fetch the task and execute Retry if task failed
22. Drawbacks?
23. Users’ Perspective Update my password to 123456 Ok, I’ll do it later …?
24. Developers’ Perspective Client paid NTD$123 to our account Ok, I’ll do it later How is it going now? It’s failed lol~ F**k Today Yesterday
25. Developers’ Perspective Sentry, Redmine Issue Tracker Elasticsearch, AWS Cloudwatch Log Storage Flower Monitor
26. How Celery Help Us Manage Async Tasks Architecture Overview Implement the Configuration and the Tasks A Practical Demo
27. Python Celery Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.
28. $ pipenv install celery Installation $ poetry add celery
29. Request Response Message Broker Server Workers Save the task Fetch the task Result Backend Fetch the result Save the result
30. Message Broker Worker 1 Retrieve Queue 1 Worker 3 Queue 3 Queue 2 Worker 2
31. Source Code https://github.com/RainrainWu/python_async_lab
32. Configuration . ├── __init__.py ├── celery_tasks │ ├── __init__.py │ ├── celery.py │ └── celeryconfig.py
33. External Broker & Storage
34. Message Content Type { “user_uuid”: “09dojq…”, “amount”: “3000”, “channel”: “line_pay”, “fee”: “12” }
35. Routing
36. Annotations
37. Tasks
38. How will it looks like in a project?
39. Deep Dive into Async Tasks Management in a Distributed System Load balancer v.s. message broker Duplicated message Mingle and gossip Thundering herds
40. Load Balancer Load Balancer Servers Load Balancer dispatch workloads to Servers Focus on synchronous, run scheduling algorithm
41. Message Broker Message Broker Workers Servers retrieve workloads from Load Balancer Focus on asynchronous, not actively schedule tasks
42. Load Balancer and Message Broker are not mutually exclusive, they can definitely cooperate very well
43. Understanding the difference between them will be very helpful for you to debug
44. Highest cost and worst performance because it needs complicated mechanism to ensure the delivery status (e.g. Pessimistic lock) Communication in Distributed System Exactly once delivery At most once delivery At least once delivery
45. Highest performance but messages may be lost, not suitable for critical tasks. (Similar to UDP protocol) Communication in Distributed System Exactly once delivery At most once delivery At least once delivery
46. Strike a balance between reliable and performance, but messages may be duplicated. Communication in Distributed System Exactly once delivery At most once delivery At least once delivery
47. Message Broker Message Broker Worker 1 Worker 2 Ready Received Unknown
48. Message Broker Message Broker Worker 1 2 3 1. Retrieve queue for new workload 2. Dispatch message to worker 3. Acknowledge to broker 4. Broker mark the state of message with “received”
49. Message Broker Message Broker Worker 1 1 2 3 1. Worker 1 retrieve queue for new workload 2. Broker dispatch message to worker and mark 
 it “dispatched” 3. Worker 1 acknowledge to broker, but a 
 network error occurs Worker 2
50. Message Broker Message Broker Worker 1 1 1. Broker does not receive acknowledgement 
 until timeout, so the message was revoked Worker 2
51. Message Broker Message Broker Worker 1 1 2 3 Worker 2 1. Worker 2 retrieve queue for new workload 2. Broker dispatch message to worker and mark 
 it “dispatched” 3. Worker 2 acknowledge to broker 4. Broker mark the state of message with “received”
52. Message Broker Workers Storage (Database) - Maintain anther state machine - Lock the minimal scope
53. Any alternative solution for communication?
54. Mingle & Gossip Message Broker Workers Broadcast vast of event for synchronization - Revoked tasks (Mingle) - Heartbeat (Gossip) - Logical clock (Both)
55. Mingle & Gossip Message Broker Workers All of the event will transmit through the message broker 危
56. CPU RAM Network IO Disk IO
57. The mingle and gossip feature was enabled by default
58. I will suggest you turn off gossip but keep mingle alive since it can avoid duplicated messages via ask revoked tasks
59. And do not make a service serving as broker and result backend at the same time
60. Thundering Herds
61. Thundering Herds Time Task start Lock DB Task finish Unlock DB
62. Thundering Herds Time Task 1~10 start Task 1 
 lock DB Task 1 finish Task 1 
 unlock DB Task 2~10 failed to access DB, 
 schedule for retry after 30 sec 00:00 00:04 00:10 00:06
63. Thundering Herds Time Task 2~10 start Task 2 
 lock DB Task 2 finish Task 2 
 unlock DB Task 3~10 failed to access DB, 
 schedule for retry after 30 sec 00:34 00:38 00:44 00:40
64. Thundering Herds Time Time History will repeat itself Random retry delay can solve this issue
65. You may see something like this on your monitor’s dashboard while monitoring celery tasks amount