Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Background jobs at scale
Search
Kerstin Puschke
May 05, 2018
Technology
60
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Background jobs at scale
Talk at RubyUnconfEU 2018
Kerstin Puschke
May 05, 2018
Transcript
85 slides
Kerstin Puschke @titanoboa42 Background jobs at scale
None
Scaling applications using background jobs keeping code simple
Outline
• Introduction to background jobs Outline
• Introduction to background jobs • Scaling applications Outline
• Introduction to background jobs • Scaling applications • Mastering
challenges Outline
Outline
• Being RESTful Outline
• Being RESTful • Background jobs at scale Outline
• Being RESTful • Background jobs at scale • Summary
Outline
Introduction to background jobs
Decoupling user facing request from time consuming task App Server
Worker
Asynchronous communication App Server Message Queue Worker
Asynchronous communication App Server Message Queue Worker Task Queue
Asynchronous communication App Server Message Queue Worker Worker Worker Task
Queue
Background job backend: task queue & broker App Server Task
Queue Broker Worker Worker Worker
Scaling applications
Task Queue Spikeability App Server Worker
Task Queue Spikeability App Server Worker Worker Worker
Task Queue Parallelization App Server Worker Worker Worker
Task Queue Retries & Redundancy App Server Worker Worker Worker
Low Prio Queue Prioritization & Specialization App Server High Prio
Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
High Prio Queue
Low Prio Queue Prioritization & Specialization App Server Worker Worker
Worker High Prio Queue Special Queue Worker
Mastering challenges
Data inconsistency
Out-of-order delivery
No exactly-once delivery
Processing time
Being RESTful
Don’t lie about resource creation
• 202 Accepted Don’t lie about resource creation
• 202 Accepted • Location: temporary resource Don’t lie about
resource creation
• 202 Accepted • Location: temporary resource • 303 See
other Don’t lie about resource creation
• 202 Accepted • Location: temporary resource • 303 See
other • Location: does not represent target resource Don’t lie about resource creation
Callers can enforce (a)sync behaviour
• Expect header Callers can enforce (a)sync behaviour
• Expect header • 202-accepted Callers can enforce (a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can enforce
(a)sync behaviour
• Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417 Expectation
failed Callers can enforce (a)sync behaviour
Background jobs at scale
DelayedJob is easy to get started
• No additional infrastructure DelayedJob is easy to get started
• No additional infrastructure • ActiveRecord DelayedJob is easy to
get started
ActiveJob makes swapping backends easy
DelayedJob has downsides at scale
• Overhead of relational database DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart DelayedJob has downsides at scale
• Overhead of relational database • Workers monitored from outside
• Frequently needs workers to restart • Hard to keep track DelayedJob has downsides at scale
Resque scales
• Redis Resque scales
• Redis • Parent-child forking for workers Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state Resque scales
• Redis • Parent-child forking for workers • Rarely needs
workers to restart • Easy to keep track, since workers manage their own state • Memory hungry Resque scales
Sidekiq scales
• Resque compatible Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry Sidekiq scales
• Resque compatible • Worker uses threads instead of child
processes • Fast • Less memory hungry • Requires thread safe code Sidekiq scales
Sharding
Database migrations
Backfills & Updates
Large collections
• Split job into Large collections
• Split job into • Collection Large collections
• Split job into • Collection • Task to be
done Large collections
• Split job into • Collection • Task to be
done • Checkpoint after iteration & requeue Large collections
Interruptible job with automatic resuming
• Allows for frequent deployments Interruptible job with automatic resuming
• Allows for frequent deployments • Disaster prevention Interruptible job
with automatic resuming
• Allows for frequent deployments • Disaster prevention • Data
integrity Interruptible job with automatic resuming
Controlling iterations
• Progress tracking Controlling iterations
• Progress tracking • Parallelization Controlling iterations
Simplicity
Background jobs
• Benefit apps of all sizes Background jobs
• Benefit apps of all sizes • Require trade-offs Background
jobs
• Benefit apps of all sizes • Require trade-offs •
Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
520
Background jobs at scale (Montreal.rb)
titanoboa
0
410
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
190
Background jobs at scale
titanoboa
0
78
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
150
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
120
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
550
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
720
No RSVP required: Asynchronous Messaging
titanoboa
0
340
Other Decks in Technology
See All in Technology
AIで実装が加速したのに、 なぜ私は仕事の大半で人と話し続けていたのか?
say_no_w
0
190
Gov-JAWS#9 統合監視機能のマネージドサービス化+AIによる障害時自動調査
fj_yoshimura_kosuke
0
200
目を覚ませ!僕らの認証情報が何者かに侵略されてるぞ
tkito
2
450
マネージャーをエスパーにしない マネジメントされる側から始める、働きやすい現場づくり
comi190327
1
750
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
580
Biohub - Cell Tracking During Development 5th place solution
rist
0
170
人の努力に頼らないセキュリティをエンジニアリングする在り方 / An approach to engineering security that does not rely on human effort
carta_engineering
0
210
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
10
5.4k
Datadog の権限統制 〜Org 統合に向けて調べた注意点 / JDDUG-Meetup_24
visional_engineering_and_design
1
180
コピーオンライト型ファイルシステムと ランダム書き込み
sat
PRO
2
120
Linuxネットワークスタック進化の軌跡: 過去10年の歩みと最新トレンド
orimanabu
4
750
SSFの現在地 ― CAEP/RISCからAI Identityへ
oidfj
0
470
Featured
See All Featured
Heart Work Chapter 1 - Part 1
lfama
PRO
11
37k
Utilizing Notion as your number one productivity tool
mfonobong
4
640
Faster Mobile Websites
deanohume
310
32k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
konakalab
0
660
Visualization
eitanlees
153
17k
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
Agile Actions for Facilitating Distributed Teams - ADO2019
mkilby
0
340
Bioeconomy Workshop: Dr. Julius Ecuru, Opportunities for a Bioeconomy in West Africa
akademiya2063
PRO
1
460
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
szymonslowik
1
1.4k
SEO in 2025: How to Prepare for the Future of Search
ipullrank
3
3.9k
Speed Design
sergeychernyshev
33
2.2k
Fireside Chat
paigeccino
43
4.1k