Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale
Search
Kerstin Puschke
May 05, 2018
Programming
0
51
Background jobs at scale
Talk at RubyUnconfEU
Kerstin Puschke
May 05, 2018
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
360
Background jobs at scale (Montreal.rb)
titanoboa
0
270
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
110
Background jobs at scale
titanoboa
1
58
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
110
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
78
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
360
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
510
No RSVP required: Asynchronous Messaging
titanoboa
0
290
Other Decks in Programming
See All in Programming
今、知っておきたい! 生成AIエージェントの世界
elith
3
350
二郎系ラーメンのコールで学ぶ AST 解析
memory1994
PRO
7
1.7k
"config" ってなんだ? / What is "config"?
okashoi
0
240
VS Code をプロダクトにどう取り込むか
onomax
1
350
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
What We Can Learn From OSS
inouehi
0
420
⼤規模⾔語モデルの拡張(RAG)が 終わったかも知れない件について
nearme_tech
23
15k
TYPO3 v13 – The road to LTS: What's new and new APIs
luisasofie_xoxo
0
190
使ってみよう Azure AI Document Intelligence
kosmosebi
2
290
効率化に挑戦してみたらモバイル開発が少し快適になった話
ryunakayama
0
130
MetricKitで予期せぬ終了を検知する話 / Detect unexpected termination with MetricKit
nekowen
1
180
Micro Frontends for Java Microservices - Devnexus 2024
mraible
PRO
0
480
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
50
9.2k
Web Components: a chance to create the future
zenorocha
305
41k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
40
4.4k
Rails Girls Zürich Keynote
gr2m
91
13k
Embracing the Ebb and Flow
colly
80
4.1k
The Invisible Side of Design
smashingmag
294
49k
The Mythical Team-Month
searls
216
42k
10 Git Anti Patterns You Should be Aware of
lemiorhan
648
58k
For a Future-Friendly Web
brad_frost
172
9k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Fantastic passwords and where to find them - at NoRuKo
philnash
37
2.5k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
Scaling applications using background jobs keeping code simple
Outline
• Introduction to background jobs Outline
• Introduction to background jobs • Scaling applications Outline
• Introduction to background jobs • Scaling applications • Mastering
challenges Outline
Outline
• Being RESTful Outline
• Being RESTful • Background jobs at scale Outline
• Being RESTful • Background jobs at scale • Summary
Outline
Introduction to background jobs
Decoupling user facing request from time consuming task App Server
Worker
Asynchronous communication App Server Message Queue Worker
Asynchronous communication App Server Message Queue Worker Task Queue
Asynchronous communication App Server Message Queue Worker Worker Worker Task
Queue
Background job backend: task queue & broker App Server Task
Queue Broker Worker Worker Worker
Scaling applications
Task Queue Spikeability App Server Worker
Task Queue Spikeability App Server Worker Worker Worker
Task Queue Parallelization App Server Worker Worker Worker
Task Queue Retries & Redundancy App Server Worker Worker Worker
Low Prio Queue Prioritization & Specialization App Server High Prio
Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
Worker High Prio Queue Special Queue Worker
Mastering challenges
Data inconsistency
Out-of-order delivery
No exactly-once delivery
Processing time
Being RESTful
Don’t lie about resource creation
• 202 Accepted Don’t lie about resource creation
• 202 Accepted • Location: temporary resource Don’t lie about
resource creation
• 202 Accepted • Location: temporary resource • 303 See
other Don’t lie about resource creation
• 202 Accepted • Location: temporary resource • 303 See
other • Location: does not represent target resource Don’t lie about resource creation
Callers can enforce (a)sync behaviour
• Expect header Callers can enforce (a)sync behaviour
• Expect header • 202-accepted Callers can enforce (a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce
(a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation
failed Callers can enforce (a)sync behaviour
Background jobs at scale
DelayedJob is easy to get started
• No additional infrastructure DelayedJob is easy to get started
• No additional infrastructure • ActiveRecord DelayedJob is easy to
get started
ActiveJob makes swapping backends easy
DelayedJob has downsides at scale
• Overhead of relational database DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
Resque scales
• Redis Resque scales
• Redis • Parent-child forking for workers Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
Sidekiq scales
• Resque compatible Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
Sharding
Database migrations
Backfills & Updates
Large collections
• Split job into Large collections
• Split job into • Collection Large collections
• Split job into • Collection • Task to be
done Large collections
• Split job into • Collection • Task to be
done • Checkpoint after iteration & requeue Large collections
Interruptible job with automatic resuming
• Allows for frequent deployments Interruptible job with automatic resuming
• Allows for frequent deployments • Disaster prevention Interruptible job
with automatic resuming
• Allows for frequent deployments • Disaster prevention • Data
integrity Interruptible job with automatic resuming
Controlling iterations
• Progress tracking Controlling iterations
• Progress tracking • Parallelization Controlling iterations
Simplicity
Background jobs
• Benefit apps of all sizes Background jobs
• Benefit apps of all sizes • Require trade-offs Background
jobs
• Benefit apps of all sizes • Require trade-offs •
Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers