Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale
Search
Kerstin Puschke
May 05, 2018
Programming
0
57
Background jobs at scale
Talk at RubyUnconfEU
Kerstin Puschke
May 05, 2018
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
450
Background jobs at scale (Montreal.rb)
titanoboa
0
370
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
130
Background jobs at scale
titanoboa
1
58
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
110
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
86
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
470
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
620
No RSVP required: Asynchronous Messaging
titanoboa
0
310
Other Decks in Programming
See All in Programming
20250326_生成AIによる_レビュー承認システムの実現.pdf
takahiromatsui
17
5.3k
フロントエンドテストの育て方
quramy
8
2.5k
MCP世界への招待: AIエンジニアが創る次世代エージェント連携の世界
gunta
2
560
SLI/SLOの設定を進めるその前に アラート品質の改善に取り組んだ話
tanden
2
710
アーキテクトと美学 / Architecture and Aesthetics
nrslib
12
3k
goにおける コネクションプールの仕組み を軽く掘って見た
aronokuyama
0
120
PHPer's Guide to Daemon Crafting Taming and Summoning
uzulla
2
970
体得しよう!RSA暗号の原理と解読
laysakura
3
520
PHPのガベージコレクションを深掘りしよう
rinchoku
0
240
AI時代のプログラミング教育 / programming education in ai era
kishida
22
20k
Go1.24 go vetとtestsアナライザ
kuro_kurorrr
2
450
英語文法から学ぶ、クリーンな設計の秘訣
newnomad
1
270
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
177
52k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
51
2.4k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
How to Ace a Technical Interview
jacobian
276
23k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
The Power of CSS Pseudo Elements
geoffreycrofte
75
5.7k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.1k
A Philosophy of Restraint
colly
203
16k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
Scaling applications using background jobs keeping code simple
Outline
• Introduction to background jobs Outline
• Introduction to background jobs • Scaling applications Outline
• Introduction to background jobs • Scaling applications • Mastering
challenges Outline
Outline
• Being RESTful Outline
• Being RESTful • Background jobs at scale Outline
• Being RESTful • Background jobs at scale • Summary
Outline
Introduction to background jobs
Decoupling user facing request from time consuming task App Server
Worker
Asynchronous communication App Server Message Queue Worker
Asynchronous communication App Server Message Queue Worker Task Queue
Asynchronous communication App Server Message Queue Worker Worker Worker Task
Queue
Background job backend: task queue & broker App Server Task
Queue Broker Worker Worker Worker
Scaling applications
Task Queue Spikeability App Server Worker
Task Queue Spikeability App Server Worker Worker Worker
Task Queue Parallelization App Server Worker Worker Worker
Task Queue Retries & Redundancy App Server Worker Worker Worker
Low Prio Queue Prioritization & Specialization App Server High Prio
Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
Worker High Prio Queue Special Queue Worker
Mastering challenges
Data inconsistency
Out-of-order delivery
No exactly-once delivery
Processing time
Being RESTful
Don’t lie about resource creation
• 202 Accepted Don’t lie about resource creation
• 202 Accepted • Location: temporary resource Don’t lie about
resource creation
• 202 Accepted • Location: temporary resource • 303 See
other Don’t lie about resource creation
• 202 Accepted • Location: temporary resource • 303 See
other • Location: does not represent target resource Don’t lie about resource creation
Callers can enforce (a)sync behaviour
• Expect header Callers can enforce (a)sync behaviour
• Expect header • 202-accepted Callers can enforce (a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce
(a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation
failed Callers can enforce (a)sync behaviour
Background jobs at scale
DelayedJob is easy to get started
• No additional infrastructure DelayedJob is easy to get started
• No additional infrastructure • ActiveRecord DelayedJob is easy to
get started
ActiveJob makes swapping backends easy
DelayedJob has downsides at scale
• Overhead of relational database DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
Resque scales
• Redis Resque scales
• Redis • Parent-child forking for workers Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
Sidekiq scales
• Resque compatible Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
Sharding
Database migrations
Backfills & Updates
Large collections
• Split job into Large collections
• Split job into • Collection Large collections
• Split job into • Collection • Task to be
done Large collections
• Split job into • Collection • Task to be
done • Checkpoint after iteration & requeue Large collections
Interruptible job with automatic resuming
• Allows for frequent deployments Interruptible job with automatic resuming
• Allows for frequent deployments • Disaster prevention Interruptible job
with automatic resuming
• Allows for frequent deployments • Disaster prevention • Data
integrity Interruptible job with automatic resuming
Controlling iterations
• Progress tracking Controlling iterations
• Progress tracking • Parallelization Controlling iterations
Simplicity
Background jobs
• Benefit apps of all sizes Background jobs
• Benefit apps of all sizes • Require trade-offs Background
jobs
• Benefit apps of all sizes • Require trade-offs •
Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers