Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale
Search
Kerstin Puschke
May 05, 2018
Programming
0
53
Background jobs at scale
Talk at RubyUnconfEU
Kerstin Puschke
May 05, 2018
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
390
Background jobs at scale (Montreal.rb)
titanoboa
0
310
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
110
Background jobs at scale
titanoboa
1
58
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
110
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
80
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
400
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
550
No RSVP required: Asynchronous Messaging
titanoboa
0
300
Other Decks in Programming
See All in Programming
データカタログ運用物語 〜令和6年夏の理想と現実〜
kuro_kurorrr
0
110
Product Management LT会_クアンド新家
shinshin
0
260
継続的な活動で築く地方エンジニアの道
myamashii
2
360
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
企業向け生成AIアプリの 開発から得られた知見
takaakikakei
0
310
Webエディタライブラリ 「CodeMirror」から学ぶ Webアプリ開発のテクニック
ryosukeigarashi
0
250
しくじり先生 Image Matching Challenge 2024 編
goosehaaan
0
810
CSC307 Lecture 10
javiergs
PRO
0
310
Jetpack for KMP
fornewid
1
290
生成AIをkintoneに連携してみた
hideg
0
230
Microservices rules (July 2024) : what good looks like
cer
PRO
0
1.6k
小さな開発会社を作った理由
polidog
0
1.9k
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
13
430
For a Future-Friendly Web
brad_frost
173
9.2k
Robots, Beer and Maslow
schacon
PRO
157
8.1k
Visualization
eitanlees
139
14k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
20
7.2k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
23
1.9k
A Tale of Four Properties
chriscoyier
155
22k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
105
6.8k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
26
1.6k
Scaling GitHub
holman
458
140k
Fashionably flexible responsive web design (full day workshop)
malarkey
399
65k
[RailsConf 2023] Rails as a piece of cake
palkan
35
4.4k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
Scaling applications using background jobs keeping code simple
Outline
• Introduction to background jobs Outline
• Introduction to background jobs • Scaling applications Outline
• Introduction to background jobs • Scaling applications • Mastering
challenges Outline
Outline
• Being RESTful Outline
• Being RESTful • Background jobs at scale Outline
• Being RESTful • Background jobs at scale • Summary
Outline
Introduction to background jobs
Decoupling user facing request from time consuming task App Server
Worker
Asynchronous communication App Server Message Queue Worker
Asynchronous communication App Server Message Queue Worker Task Queue
Asynchronous communication App Server Message Queue Worker Worker Worker Task
Queue
Background job backend: task queue & broker App Server Task
Queue Broker Worker Worker Worker
Scaling applications
Task Queue Spikeability App Server Worker
Task Queue Spikeability App Server Worker Worker Worker
Task Queue Parallelization App Server Worker Worker Worker
Task Queue Retries & Redundancy App Server Worker Worker Worker
Low Prio Queue Prioritization & Specialization App Server High Prio
Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
Worker High Prio Queue Special Queue Worker
Mastering challenges
Data inconsistency
Out-of-order delivery
No exactly-once delivery
Processing time
Being RESTful
Don’t lie about resource creation
• 202 Accepted Don’t lie about resource creation
• 202 Accepted • Location: temporary resource Don’t lie about
resource creation
• 202 Accepted • Location: temporary resource • 303 See
other Don’t lie about resource creation
• 202 Accepted • Location: temporary resource • 303 See
other • Location: does not represent target resource Don’t lie about resource creation
Callers can enforce (a)sync behaviour
• Expect header Callers can enforce (a)sync behaviour
• Expect header • 202-accepted Callers can enforce (a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce
(a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation
failed Callers can enforce (a)sync behaviour
Background jobs at scale
DelayedJob is easy to get started
• No additional infrastructure DelayedJob is easy to get started
• No additional infrastructure • ActiveRecord DelayedJob is easy to
get started
ActiveJob makes swapping backends easy
DelayedJob has downsides at scale
• Overhead of relational database DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
Resque scales
• Redis Resque scales
• Redis • Parent-child forking for workers Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
Sidekiq scales
• Resque compatible Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
Sharding
Database migrations
Backfills & Updates
Large collections
• Split job into Large collections
• Split job into • Collection Large collections
• Split job into • Collection • Task to be
done Large collections
• Split job into • Collection • Task to be
done • Checkpoint after iteration & requeue Large collections
Interruptible job with automatic resuming
• Allows for frequent deployments Interruptible job with automatic resuming
• Allows for frequent deployments • Disaster prevention Interruptible job
with automatic resuming
• Allows for frequent deployments • Disaster prevention • Data
integrity Interruptible job with automatic resuming
Controlling iterations
• Progress tracking Controlling iterations
• Progress tracking • Parallelization Controlling iterations
Simplicity
Background jobs
• Benefit apps of all sizes Background jobs
• Benefit apps of all sizes • Require trade-offs Background
jobs
• Benefit apps of all sizes • Require trade-offs •
Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers