Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale (Montreal.rb)
Search
Kerstin Puschke
February 26, 2019
Programming
0
310
Background jobs at scale (Montreal.rb)
Talks at Montreal.rb
Kerstin Puschke
February 26, 2019
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
390
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
110
Background jobs at scale
titanoboa
1
58
Background jobs at scale
titanoboa
0
53
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
110
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
80
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
400
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
550
No RSVP required: Asynchronous Messaging
titanoboa
0
300
Other Decks in Programming
See All in Programming
유연한 Composable 설계
l2hyunwoo
0
380
AWS CDKにおける「再利用性」を考える / aws-cdk-reusability
gotok365
6
1.3k
【Go言語】ジェネリクス
tomo1227
0
170
スクラムマスターって孤独じゃないですか?
yoshitaroyoyo
1
140
DMMプラットフォームにおけるTiDBの導入から運用まで
pospome
7
3k
Advanced App Shrinking Techniques
cbeyls
2
150
Rust.Nagoya #1
codemountains
0
170
リハビリmruby
kishima
1
160
継続的な活動で築く地方エンジニアの道
myamashii
2
350
Google's Recipe for Scaling (Web) Security – LocoMocoSec 2024
lweichselbaum
0
170
「2024年版 Kotlin サーバーサイドプログラミング実践開発」の補講 〜O/Rマッパー編〜
n_takehata
2
260
Temporalを取り巻く仕様を整理する
sajikix
0
110
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.3k
Building Effective Engineering Teams - LeadDev
addyosmani
47
2.2k
Debugging Ruby Performance
tmm1
71
11k
Visualization
eitanlees
139
14k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
18
1.2k
KATA
mclloyd
20
13k
How GitHub Uses GitHub to Build GitHub
holman
471
290k
Scaling GitHub
holman
458
140k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
353
29k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
245
1.2M
Leading Effective Engineering Teams 2024
addyosmani
3
300
Raft: Consensus for Rubyists
vanstee
134
6.5k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
@titanoboa42 Scaling applications using background jobs keeping code simple
@titanoboa42 Outline
@titanoboa42 • Introduction to background jobs Outline
@titanoboa42 • Introduction to background jobs • Features Outline
@titanoboa42 • Introduction to background jobs • Features • Mastering
challenges Outline
@titanoboa42 Outline
@titanoboa42 • Being RESTful Outline
@titanoboa42 • Being RESTful • Background jobs at scale Outline
@titanoboa42 • Being RESTful • Background jobs at scale •
Summary Outline
@titanoboa42 Introduction to background jobs
@titanoboa42 Background job: Work to be done later App Server
Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue
@titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker
Task Queue
@titanoboa42 Background job backend: task queue & broker
@titanoboa42 Encapsulating async communication
@titanoboa42 Features
@titanoboa42 Task Queue Response times App Server Worker
@titanoboa42 Task Queue Spikeability App Server Worker
@titanoboa42 Task Queue Parallelization App Server Worker Worker Worker
@titanoboa42 Task Queue Retries App Server Worker Worker Worker
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Mastering challenges
@titanoboa42 No exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent jobs & at least once delivery No exactly once delivery
@titanoboa42 Out of order delivery
@titanoboa42 • If order matters, queue sequentially Out of order
delivery
@titanoboa42 • If order matters, queue sequentially • First job
queues follow up jobs Out of order delivery
@titanoboa42 Job queued and processed by different versions
@titanoboa42 • No breaking changes to job parameters Job queued
and processed by different versions
@titanoboa42 • No breaking changes to job parameters • Changes
need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
@titanoboa42 Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency
guarantees vs. benefits of background jobs Eventual consistency (at best)
@titanoboa42 Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction Non-transactional
queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback • Commit before queuing or stage transactionally Non-transactional queuing
@titanoboa42 Being RESTful
@titanoboa42 Don’t lie about resource creation
@titanoboa42 • 202 Accepted Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource Don’t lie
about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other • Location: does not represent target resource Don’t lie about resource creation
@titanoboa42 Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted Callers can enforce (a)sync
behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can
enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417
Expectation failed Callers can enforce (a)sync behaviour
@titanoboa42 Background jobs at scale
@titanoboa42 DelayedJob is easy to get started
@titanoboa42 • No additional infrastructure DelayedJob is easy to get
started
@titanoboa42 • No additional infrastructure • ActiveRecord DelayedJob is easy
to get started
@titanoboa42 ActiveJob makes swapping backends easy
@titanoboa42 DelayedJob issues
@titanoboa42 • Overhead of relational database DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside • Frequently needs workers to restart DelayedJob issues
@titanoboa42 Resque scales
@titanoboa42 • Redis - no relational db Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
@titanoboa42 Resque issues
@titanoboa42 • Child processes Resque issues
@titanoboa42 • Child processes • Memory hungry and slow Resque
issues
@titanoboa42 Sidekiq scales
@titanoboa42 • Redis - no relational db Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes • Fast and less memory hungry Sidekiq scales
@titanoboa42 Sidekiq issues
@titanoboa42 • Requires thread safe code Sidekiq issues
@titanoboa42 Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments Long running
jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments • Not
cloud-friendly Long running jobs - Resque
@titanoboa42 • Aborted and requeued on shutdown Long running jobs
- Sidekiq
@titanoboa42 • Aborted and requeued on shutdown • Job may
not finish before being aborted again Long running jobs - Sidekiq
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Large collections
@titanoboa42 • Split job into collection and task to be
done Large collections
@titanoboa42 • Split job into collection and task to be
done • Checkpoint after iteration & requeue Large collections
@titanoboa42 Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job
with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention • Data
integrity Interruptible job with automatic resuming
@titanoboa42 Abstracting scaling issues simplifies concrete background jobs
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Background jobs
@titanoboa42 • Benefit apps of all sizes Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
• Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers