Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale
Search
Kerstin Puschke
May 05, 2018
Programming
0
64
Background jobs at scale
Talk at RubyUnconfEU
Kerstin Puschke
May 05, 2018
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
460
Background jobs at scale (Montreal.rb)
titanoboa
0
380
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
140
Background jobs at scale
titanoboa
1
58
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
120
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
88
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
490
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
630
No RSVP required: Asynchronous Messaging
titanoboa
0
310
Other Decks in Programming
See All in Programming
LLMOpsのパフォーマンスを支える技術と現場で実践した改善
po3rin
7
670
オホーツクでコミュニティを立ち上げた理由―地方出身プログラマの挑戦 / TechRAMEN 2025 Conference
lemonade_37
2
460
画像コンペでのベースラインモデルの育て方
tattaka
3
1.5k
なぜ今、Terraformの本を書いたのか? - 著者陣に聞く!『Terraformではじめる実践IaC』登壇資料
fufuhu
4
570
대규모 트래픽을 처리하는 프론트 개발자의 전략
maryang
0
120
WebAssemblyインタプリタを書く ~Component Modelを添えて~
ruccho
1
750
技術的負債で信頼性が限界だったWordPress運用をShifterで完全復活させた話
rvirus0817
1
1.4k
Terraform やるなら公式スタイルガイドを読もう 〜重要項目 10選〜
hiyanger
12
3k
11年かかって やっとVibe Codingに 時代が追いつきましたね
yimajo
1
260
構文解析器入門
ydah
7
2.1k
あまり知られていない MCP 仕様たち / MCP specifications that aren’t widely known
ktr_0731
0
240
バイブコーディング × 設計思考
nogu66
0
110
Featured
See All Featured
Docker and Python
trallard
45
3.5k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Raft: Consensus for Rubyists
vanstee
140
7.1k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.3k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
The Invisible Side of Design
smashingmag
301
51k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
283
13k
Balancing Empowerment & Direction
lara
1
540
The World Runs on Bad Software
bkeepers
PRO
70
11k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
Scaling applications using background jobs keeping code simple
Outline
• Introduction to background jobs Outline
• Introduction to background jobs • Scaling applications Outline
• Introduction to background jobs • Scaling applications • Mastering
challenges Outline
Outline
• Being RESTful Outline
• Being RESTful • Background jobs at scale Outline
• Being RESTful • Background jobs at scale • Summary
Outline
Introduction to background jobs
Decoupling user facing request from time consuming task App Server
Worker
Asynchronous communication App Server Message Queue Worker
Asynchronous communication App Server Message Queue Worker Task Queue
Asynchronous communication App Server Message Queue Worker Worker Worker Task
Queue
Background job backend: task queue & broker App Server Task
Queue Broker Worker Worker Worker
Scaling applications
Task Queue Spikeability App Server Worker
Task Queue Spikeability App Server Worker Worker Worker
Task Queue Parallelization App Server Worker Worker Worker
Task Queue Retries & Redundancy App Server Worker Worker Worker
Low Prio Queue Prioritization & Specialization App Server High Prio
Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
Worker High Prio Queue Special Queue Worker
Mastering challenges
Data inconsistency
Out-of-order delivery
No exactly-once delivery
Processing time
Being RESTful
Don’t lie about resource creation
• 202 Accepted Don’t lie about resource creation
• 202 Accepted • Location: temporary resource Don’t lie about
resource creation
• 202 Accepted • Location: temporary resource • 303 See
other Don’t lie about resource creation
• 202 Accepted • Location: temporary resource • 303 See
other • Location: does not represent target resource Don’t lie about resource creation
Callers can enforce (a)sync behaviour
• Expect header Callers can enforce (a)sync behaviour
• Expect header • 202-accepted Callers can enforce (a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce
(a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation
failed Callers can enforce (a)sync behaviour
Background jobs at scale
DelayedJob is easy to get started
• No additional infrastructure DelayedJob is easy to get started
• No additional infrastructure • ActiveRecord DelayedJob is easy to
get started
ActiveJob makes swapping backends easy
DelayedJob has downsides at scale
• Overhead of relational database DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
Resque scales
• Redis Resque scales
• Redis • Parent-child forking for workers Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
Sidekiq scales
• Resque compatible Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
Sharding
Database migrations
Backfills & Updates
Large collections
• Split job into Large collections
• Split job into • Collection Large collections
• Split job into • Collection • Task to be
done Large collections
• Split job into • Collection • Task to be
done • Checkpoint after iteration & requeue Large collections
Interruptible job with automatic resuming
• Allows for frequent deployments Interruptible job with automatic resuming
• Allows for frequent deployments • Disaster prevention Interruptible job
with automatic resuming
• Allows for frequent deployments • Disaster prevention • Data
integrity Interruptible job with automatic resuming
Controlling iterations
• Progress tracking Controlling iterations
• Progress tracking • Parallelization Controlling iterations
Simplicity
Background jobs
• Benefit apps of all sizes Background jobs
• Benefit apps of all sizes • Require trade-offs Background
jobs
• Benefit apps of all sizes • Require trade-offs •
Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers