Background jobs at scale

Talk at RubyUnconfEU

Transcript

1. Kerstin Puschke @titanoboa42 Background jobs at scale
3. Scaling applications using background jobs keeping code simple
4. Outline
5. • Introduction to background jobs Outline
6. • Introduction to background jobs • Scaling applications Outline
7. • Introduction to background jobs • Scaling applications • Mastering challenges Outline
8. Outline
9. • Being RESTful Outline
10. • Being RESTful • Background jobs at scale Outline
11. • Being RESTful • Background jobs at scale • Summary Outline
12. Introduction to background jobs
13. Decoupling user facing request from time consuming task App Server Worker
14. Asynchronous communication App Server Message Queue Worker
15. Asynchronous communication App Server Message Queue Worker Task Queue
16. Asynchronous communication App Server Message Queue Worker Worker Worker Task Queue
17. Background job backend:
 task queue & broker App Server Task Queue Broker Worker Worker Worker
18. Scaling applications
19. Task Queue Spikeability App Server Worker
20. Task Queue Spikeability App Server Worker Worker Worker
21. Task Queue Parallelization App Server Worker Worker Worker
22. Task Queue Retries & Redundancy App Server Worker Worker Worker
23. Low Prio Queue Prioritization & Specialization App Server High Prio Queue
24. Low Prio Queue Prioritization & Specialization App Server Worker Worker High Prio Queue
25. Low Prio Queue Prioritization & Specialization App Server Worker Worker High Prio Queue
26. Low Prio Queue Prioritization & Specialization App Server Worker Worker Worker High Prio Queue Special Queue Worker
27. Mastering challenges
28. Data inconsistency
29. Out-of-order delivery
30. No exactly-once delivery
31. Processing time
32. Being RESTful
33. Don’t lie about resource creation
34. • 202 Accepted Don’t lie about resource creation
35. • 202 Accepted • Location: temporary resource Don’t lie about resource creation
36. • 202 Accepted • Location: temporary resource • 303 See other Don’t lie about resource creation
37. • 202 Accepted • Location: temporary resource • 303 See other • Location: does not represent target resource Don’t lie about resource creation
38. Callers can enforce (a)sync behaviour
39. • Expect header Callers can enforce (a)sync behaviour
40. • Expect header • 202-accepted Callers can enforce (a)sync behaviour
41. • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce (a)sync behaviour
42. • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation failed Callers can enforce (a)sync behaviour
43. Background jobs at scale
44. DelayedJob is easy to get started
45. • No additional infrastructure DelayedJob is easy to get started
46. • No additional infrastructure • ActiveRecord DelayedJob is easy to get started
47. ActiveJob makes swapping backends easy
48. DelayedJob has downsides at scale
49. • Overhead of relational database DelayedJob has downsides at scale
50. • Overhead of relational database • Workers monitored from outside DelayedJob has downsides at scale
51. • Overhead of relational database • Workers monitored from outside • Frequently needs workers to restart DelayedJob has downsides at scale
52. • Overhead of relational database • Workers monitored from outside • Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
53. Resque scales
54. • Redis Resque scales
55. • Redis • Parent-child forking for workers Resque scales
56. • Redis • Parent-child forking for workers • Rarely needs workers to restart Resque scales
57. • Redis • Parent-child forking for workers • Rarely needs workers to restart • Easy to keep track, since workers manage their own state Resque scales
58. • Redis • Parent-child forking for workers • Rarely needs workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
59. Sidekiq scales
60. • Resque compatible Sidekiq scales
61. • Resque compatible • Worker uses threads instead of child processes Sidekiq scales
62. • Resque compatible • Worker uses threads instead of child processes • Fast Sidekiq scales
63. • Resque compatible • Worker uses threads instead of child processes • Fast • Less memory hungry Sidekiq scales
64. • Resque compatible • Worker uses threads instead of child processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
65. Sharding
66. Database migrations
67. Backfills & Updates
68. Large collections
69. • Split job into Large collections
70. • Split job into • Collection Large collections
71. • Split job into • Collection • Task to be done Large collections
72. • Split job into • Collection • Task to be done • Checkpoint after iteration & requeue Large collections
73. Interruptible job with automatic resuming
74. • Allows for frequent deployments Interruptible job with automatic resuming
75. • Allows for frequent deployments • Disaster prevention Interruptible job with automatic resuming
76. • Allows for frequent deployments • Disaster prevention • Data integrity Interruptible job with automatic resuming
77. Controlling iterations
78. • Progress tracking Controlling iterations
79. • Progress tracking • Parallelization Controlling iterations
80. Simplicity
81. Background jobs
82. • Benefit apps of all sizes Background jobs
83. • Benefit apps of all sizes • Require trade-offs Background jobs
84. • Benefit apps of all sizes • Require trade-offs • Keep code simple at scale Background jobs
85. Thanks!
 Questions?
 @titanoboa42
 
 https://www.shopify.com/careers