Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Background jobs at scale (Montreal.rb)
Search
Kerstin Puschke
February 26, 2019
Programming
0
260
Background jobs at scale (Montreal.rb)
Talks at Montreal.rb
Kerstin Puschke
February 26, 2019
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
350
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
110
Background jobs at scale
titanoboa
1
58
Background jobs at scale
titanoboa
0
51
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
110
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
78
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
350
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
500
No RSVP required: Asynchronous Messaging
titanoboa
0
290
Other Decks in Programming
See All in Programming
IntelliJ IDEA を知らなかった 自分に教えたい小ネタ集 / IntelliJ IDEA Hints for My Past Self
mackey0225
3
170
document.write再考
brn
5
2.5k
MySQL のインデックスの種類をおさらいしよう! / overviewing indexes in MySQL
okashoi
0
170
LPIXEL×CADDi_kaerururu
kaerururu
3
300
LLMチャットボットのアプリケーション設計Tips
os1ma
4
660
生成 AI の中身を覗いてみよう〜基礎から医療現場での応用まで〜
soh9834
2
760
ファイル先頭の use の意味、説明できますか? 〜PHP の namespace と autoloading の関係を正しく理解しよう〜 / namespace and autoloading in php
okashoi
2
470
導入から5年が経って見えた Datadog APM 運用の課題
bgpat
2
540
今の SmartHR にエンジニアで入社するとどうなるの?
daisukeshinoku
3
950
孤独のCTOグルメという やや奇抜な企画をやった目的と効果
shoheimitani
3
1k
phpunit/php-code-coverageって何をしてるんだ #phperkaigi
o0h
PRO
2
220
Laravel標準バリデーションでできること
hmb_ok
2
360
Featured
See All Featured
Designing for Performance
lara
601
67k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
67
38k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
28
5.9k
Optimizing for Happiness
mojombo
369
69k
Creatively Recalculating Your Daily Design Routine
revolveconf
209
11k
Product Roadmaps are Hard
iamctodd
43
9.6k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
12
1.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
19
1.9k
Rails Girls Zürich Keynote
gr2m
91
13k
Thoughts on Productivity
jonyablonski
57
3.8k
Practical Orchestrator
shlominoach
180
9.7k
We Have a Design System, Now What?
morganepeng
42
6.7k
Transcript
Kerstin Puschke @titanoboa42 Background jobs at scale
None
@titanoboa42 Scaling applications using background jobs keeping code simple
@titanoboa42 Outline
@titanoboa42 • Introduction to background jobs Outline
@titanoboa42 • Introduction to background jobs • Features Outline
@titanoboa42 • Introduction to background jobs • Features • Mastering
challenges Outline
@titanoboa42 Outline
@titanoboa42 • Being RESTful Outline
@titanoboa42 • Being RESTful • Background jobs at scale Outline
@titanoboa42 • Being RESTful • Background jobs at scale •
Summary Outline
@titanoboa42 Introduction to background jobs
@titanoboa42 Background job: Work to be done later App Server
Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue
@titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker
Task Queue
@titanoboa42 Background job backend: task queue & broker
@titanoboa42 Encapsulating async communication
@titanoboa42 Features
@titanoboa42 Task Queue Response times App Server Worker
@titanoboa42 Task Queue Spikeability App Server Worker
@titanoboa42 Task Queue Parallelization App Server Worker Worker Worker
@titanoboa42 Task Queue Retries App Server Worker Worker Worker
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Mastering challenges
@titanoboa42 No exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent jobs & at least once delivery No exactly once delivery
@titanoboa42 Out of order delivery
@titanoboa42 • If order matters, queue sequentially Out of order
delivery
@titanoboa42 • If order matters, queue sequentially • First job
queues follow up jobs Out of order delivery
@titanoboa42 Job queued and processed by different versions
@titanoboa42 • No breaking changes to job parameters Job queued
and processed by different versions
@titanoboa42 • No breaking changes to job parameters • Changes
need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
@titanoboa42 Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency
guarantees vs. benefits of background jobs Eventual consistency (at best)
@titanoboa42 Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction Non-transactional
queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback • Commit before queuing or stage transactionally Non-transactional queuing
@titanoboa42 Being RESTful
@titanoboa42 Don’t lie about resource creation
@titanoboa42 • 202 Accepted Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource Don’t lie
about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other • Location: does not represent target resource Don’t lie about resource creation
@titanoboa42 Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted Callers can enforce (a)sync
behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can
enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417
Expectation failed Callers can enforce (a)sync behaviour
@titanoboa42 Background jobs at scale
@titanoboa42 DelayedJob is easy to get started
@titanoboa42 • No additional infrastructure DelayedJob is easy to get
started
@titanoboa42 • No additional infrastructure • ActiveRecord DelayedJob is easy
to get started
@titanoboa42 ActiveJob makes swapping backends easy
@titanoboa42 DelayedJob issues
@titanoboa42 • Overhead of relational database DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside • Frequently needs workers to restart DelayedJob issues
@titanoboa42 Resque scales
@titanoboa42 • Redis - no relational db Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
@titanoboa42 Resque issues
@titanoboa42 • Child processes Resque issues
@titanoboa42 • Child processes • Memory hungry and slow Resque
issues
@titanoboa42 Sidekiq scales
@titanoboa42 • Redis - no relational db Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes • Fast and less memory hungry Sidekiq scales
@titanoboa42 Sidekiq issues
@titanoboa42 • Requires thread safe code Sidekiq issues
@titanoboa42 Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments Long running
jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments • Not
cloud-friendly Long running jobs - Resque
@titanoboa42 • Aborted and requeued on shutdown Long running jobs
- Sidekiq
@titanoboa42 • Aborted and requeued on shutdown • Job may
not finish before being aborted again Long running jobs - Sidekiq
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Large collections
@titanoboa42 • Split job into collection and task to be
done Large collections
@titanoboa42 • Split job into collection and task to be
done • Checkpoint after iteration & requeue Large collections
@titanoboa42 Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job
with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention • Data
integrity Interruptible job with automatic resuming
@titanoboa42 Abstracting scaling issues simplifies concrete background jobs
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Background jobs
@titanoboa42 • Benefit apps of all sizes Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
• Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers