Scale like a pro

Distributed task processing with Python and Celery

1. Scale like a pro Distributed computing with message queues and Python Roman Imankulov | Python Porto | December 2017
2. Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
3. browser ./webserver.py request POST /obj/<id> obj.update()
4. browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
5. browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send email “object updated” …
6. browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update business analytics send email “object updated” …
7. browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update business analytics send email “object updated” … response
9. browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py update search indexes send email “object updated” update business analytics
10. Message queues
13. queue
14. frontends queue
15. frontends queue workers
16. frontends queue job 1 workers
17. frontends queue job 1 workers job 2
18. frontends queue job 1 workers job 2 job 3
19. frontends queue put() job 1 workers job 2 job 3
20. frontends queue put() job 1 workers job 2 job 3
21. frontends queue job 1 workers job 2 job 3
22. frontends queue job 1 workers put() job 2 job 3
23. frontends queue job 1 workers put() job 2 job 3
24. frontends queue job 1 workers job 2 job 3
25. frontends queue job 1 workers put() job 2 job 3
26. frontends queue job 1 workers put() job 2 job 3
27. frontends queue job 1 workers job 2 job 3
28. frontends queue job 1 get() workers job 2 job 3
29. frontends queue get() workers job 2 job 3 job 1
30. frontends queue workers job 2 job 3 job 1
31. frontends queue workers get() job 2 job 3 job 1
32. frontends queue workers get() job 3 job 1 job 2
33. frontends queue workers job 3 job 1 job 2
34. frontends queue workers job 3 job 2
35. frontends queue workers job 3 job 2 get()
36. frontends queue workers job 2 job 3 get()
37. frontends queue workers job 2 job 3
38. frontends queue workers
39. Queue in python
40. Multiprocessing
48. Celery The queue out of the box
54. Celery Workflows chains, groups and chords
55. Task signatures
56. Task signatures
57. Task signatures
58. Chains a(…) b(…) c(…)
59. Chains a(…) b(…) c(…)
60. Chains a(…) b(…) c(…)
61. a(…) b(…) c(…) Groups
62. a(…) b(…) c(…) Groups
63. a(…) b(…) c(…) Groups
64. Chords a(…) b(…) c(…) d(…)
65. Chords a(…) b(…) c(…) d(…)
66. Chords a(…) b(…) c(…) d(…)
67. Celery Extras out of the box
69. • Different backends
70. • Different backends • Different serializers
71. • Different backends • Different serializers • Callbacks / errbacks
72. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks
73. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks
74. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results
75. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
76. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
77. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
78. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
79. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
80. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
81. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
82. • Different backends • Different serializers • Callbacks / errbacks • Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
84. facebook.com/pyporto