Django tasty salad- DOs and DON'Ts using Celery

Django tasty salad- DOs and DON'Ts using Celery

Best practices and optimizations when using Celery for background tasks.

943620c3bc4056a40ce132690f1d9ac1?s=128

Roberto Rosario

April 16, 2016
Tweet

Transcript

  1. 3.
  2. 5.

    Celery is a simple, flexible and reliable distributed system to

    process vast amounts of messages, while providing operations with the tools required to maintain such a system. It’s a task queue with focus on real-time processing, while also supporting task scheduling.
  3. 7.
  4. 8.
  5. 9.
  6. 10.

    • For tasks outside the HTTP request/response cycle. • Long

    running process after an HTTP request. • Distribute a task to better use hardware. • Automatic (scheduled) tasks.
  7. 11.
  8. 17.
  9. 24.

    Message broker is an intermediary program module that translates a

    message from the formal messaging protocol of the sender to the formal messaging protocol of the receiver. Message brokers are elements in telecommunication networks where software applications communicate by exchanging formally-defined messages. Message brokers are a building block of Message oriented middleware. https://en.wikipedia.org/wiki/Message_broker
  10. 28.
  11. 30.
  12. 38.
  13. 53.
  14. 55.
  15. 57.
  16. 58.
  17. 60.
  18. 62.
  19. 64.
  20. 67.
  21. 68.
  22. 73.
  23. 79.
  24. 84.
  25. 86.
  26. 89.
  27. 98.

    Hint #2: You can implement the rate limit on your

    task code only for what you need.
  28. 99.
  29. 102.
  30. 103.
  31. 104.
  32. 110.

    “RabbitMQ was created specifically for queueing, whereas Redis is an

    in-memory datastore: They were designed with different intents in mind. On my end, I am impressed that Redis even comes close in terms of performance.” - Björn Lindfors
  33. 113.
  34. 115.
  35. 116.
  36. 118.
  37. 120.
  38. 121.
  39. 126.
  40. 127.
  41. 128.
  42. 129.
  43. 130.
  44. 131.
  45. 132.
  46. 133.
  47. 140.
  48. 144.
  49. 145.
  50. 146.
  51. 147.
  52. 150.
  53. 151.