Django tasty salad- DOs and DON'Ts using Celery

Best practices and optimizations when using Celery for background tasks.

Transcript

1. Django tasty salad: DOs and DON'Ts using Celery Roberto Rosario robertorosario.com
2. Who am I? robertorosario.com me@robertorosario.com Roberto Rosario
4. What is Celery?
5. Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.
6. Why do I need it?
10. • For tasks outside the HTTP request/response cycle. • Long running process after an HTTP request. • Distribute a task to better use hardware. • Automatic (scheduled) tasks.
11. Examples
12. Typical webapp cycle
13. HTTP Request HTTP Response
14. Sending email
15. HTTP Request HTTP Response
16. Scheduled data cleanup
18. Thumbnail generation
19. 3 x HTTP Request HTTP Response Webapp Workers
20. OCR processing
21. n x HTTP Request Webapp Workers
22. How Celery works?
23. Broker Result storage Workers Celery App
24. Message broker is an intermediary program module that translates a message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Message brokers are elements in telecommunication networks where software applications communicate by exchanging formally-defined messages. Message brokers are a building block of Message oriented middleware. https://en.wikipedia.org/wiki/Message_broker
25. Broker Celery
26. http://docs.celeryproject.org/en/latest/getting-started/brokers/
27. Result storage
29. Why result storages and brokers?
31. Broker Result storage Workers Celery App
32. Broker Result storage Workers Celery App Computer 1 Computer 2
33. Broker Result storage Workers Celery App Amazon VM1 USA Amazon VM10 EU
34. Broker Result storage Workers Celery App VM Container
35. Broker Result storage Workers Celery App Language A Language B
36. Broker Result storage Workers Celery App Now Future
37. Broker Result storage Workers Celery App 10 records 100 records
39. Never assume the state of things beforehand.
40. Workers must always introspect the world.
41. Task queueing is like multiprocessing.
42. Not really.
43. Caller and task do not share memory.
44. Only pass pointer, not actual objects.
45. Only pass model instance IDs to tasks.
46. Avoid the urge to pass pickled objects.
47. Task arguments must be serializable.
48. Stick to basics: text or numbers.
49. Which serializer?
50. Keep it simple: JSON.
51. Avoid complex data times.
52. Specially dates.
54. Don’t need results? Ignore them.
56. Split tasks into queues.
59. You can run workers that consume all queues (they do that by default).
61. Or run workers for each queue as your needs grow.
63. Or group your queues into priority groups.
65. Task queues create a new type of race condition. Why?
66. Because message queues are fast!
67. How fast?
69. Broker Database Worker
70. Broker Database Worker
71. Broker Database Worker
72. Broker Database Worker
73. How fast?
74. Faster than databases.
75. Be mindful of commit or replication delays.
76. Add preventive delays to fast tasks that depend on database access.
77. Monitor your setup.
78. https://flower.readthedocs.org
80. But, don’t micro-manage your queues.
81. Such as frozen tasks consuming system resources...
82. Let your tasks die an honorable death. Use timeouts.
83. http://knowyourmeme.com/memes/commit-sudoku
85. Use a global hard timeout too.
87. Never worry about runaway frozen task retries filling up your queues.
88. Use a soft timeout to give your task a chance to cleanup.
90. These don’t work if your task is frozen.
91. Retry delays.
92. Use a static retry delay for tasks using local services.
93. Use increasing retry delays for task using external services.
94. Continually hitting Twitter’s API won’t make it answer faster.
95. https://www.carthrottle.com/post/if-you-press-the-pedestrian-crossing-button-youre-probably-wasting-your-time/
96. Disable rate limits, unless you REALLY needed them.
97. Hint: You don’t.
98. Hint #2: You can implement the rate limit on your task code only for what you need.
100. Use the best tool for the job.
101. Redis as a broker?
105. “But Redis is very fast because it is a memory database”
106. http://www.lifetime-weightloss.com/blog/2013/1/30/the-finishing-line-effect.html
107. https://imgflip.com
108. https://imgflip.com
109. http://www.minvolai.com/blog/2013/10/RabbitMQ-vs-Redis-as-Message-Brokers/rabbitmq-vs-redis-message-broker/
110. “RabbitMQ was created specifically for queueing, whereas Redis is an in-memory datastore: They were designed with different intents in mind. On my end, I am impressed that Redis even comes close in terms of performance.” - Björn Lindfors
111. RabbitMQ as a broker, Redis as the result backend.
112. librabbitmq
114. Not all tasks are vital. Use transient queues for those.
117. Task orchestration.
119. Resource locking.
122. Local locks are worthless.
123. We need distributed locks.
124. No shared memory.
125. Or many other resources for that matter.
134. “It is Easier to Ask for Forgiveness than Permission”
135. The same applies to locks.
136. Never check for a lock existence.
137. Task dies after acquiring a lock. Server is stopped after tasks acquire locks.
138. Infinite locks.
139. Stale locks.
141. Managed or cooperative?
142. Periodic lock purging.
143. Lock expiration.
148. CursorPatcher(error_producer=CountErrorProducer, kwargs={'exception': OperationalError, 'number': 500, 'reset': True}),
149. SQLite as DB SQLite as Lock Manager DEBUG=True runserver
151. Thank you!