Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Background jobs at scale (Montreal.rb)
Search
Kerstin Puschke
February 26, 2019
Programming
410
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Background jobs at scale (Montreal.rb)
Talks at Montreal.rb
Kerstin Puschke
February 26, 2019
Transcript
98 slides
Kerstin Puschke @titanoboa42 Background jobs at scale
None
@titanoboa42 Scaling applications using background jobs keeping code simple
@titanoboa42 Outline
@titanoboa42 • Introduction to background jobs Outline
@titanoboa42 • Introduction to background jobs • Features Outline
@titanoboa42 • Introduction to background jobs • Features • Mastering
challenges Outline
@titanoboa42 Outline
@titanoboa42 • Being RESTful Outline
@titanoboa42 • Being RESTful • Background jobs at scale Outline
@titanoboa42 • Being RESTful • Background jobs at scale •
Summary Outline
@titanoboa42 Introduction to background jobs
@titanoboa42 Background job: Work to be done later App Server
Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue
@titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker
Task Queue
@titanoboa42 Background job backend: task queue & broker
@titanoboa42 Encapsulating async communication
@titanoboa42 Features
@titanoboa42 Task Queue Response times App Server Worker
@titanoboa42 Task Queue Spikeability App Server Worker
@titanoboa42 Task Queue Parallelization App Server Worker Worker Worker
@titanoboa42 Task Queue Retries App Server Worker Worker Worker
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Mastering challenges
@titanoboa42 No exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent jobs & at least once delivery No exactly once delivery
@titanoboa42 Out of order delivery
@titanoboa42 • If order matters, queue sequentially Out of order
delivery
@titanoboa42 • If order matters, queue sequentially • First job
queues follow up jobs Out of order delivery
@titanoboa42 Job queued and processed by different versions
@titanoboa42 • No breaking changes to job parameters Job queued
and processed by different versions
@titanoboa42 • No breaking changes to job parameters • Changes
need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
@titanoboa42 Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency Eventual consistency (at best)
@titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency
guarantees vs. benefits of background jobs Eventual consistency (at best)
@titanoboa42 Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction Non-transactional
queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or if rollback • Commit before queuing or stage transactionally Non-transactional queuing
@titanoboa42 Being RESTful
@titanoboa42 Don’t lie about resource creation
@titanoboa42 • 202 Accepted Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource Don’t lie
about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other Don’t lie about resource creation
@titanoboa42 • 202 Accepted • Location: temporary resource • 303
See other • Location: does not represent target resource Don’t lie about resource creation
@titanoboa42 Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header Callers can enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted Callers can enforce (a)sync
behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can
enforce (a)sync behaviour
@titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417
Expectation failed Callers can enforce (a)sync behaviour
@titanoboa42 Background jobs at scale
@titanoboa42 DelayedJob is easy to get started
@titanoboa42 • No additional infrastructure DelayedJob is easy to get
started
@titanoboa42 • No additional infrastructure • ActiveRecord DelayedJob is easy
to get started
@titanoboa42 ActiveJob makes swapping backends easy
@titanoboa42 DelayedJob issues
@titanoboa42 • Overhead of relational database DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside DelayedJob issues
@titanoboa42 • Overhead of relational database • Workers monitored from
outside • Frequently needs workers to restart DelayedJob issues
@titanoboa42 Resque scales
@titanoboa42 • Redis - no relational db Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart Resque scales
@titanoboa42 • Redis - no relational db • Parent-child forking
for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
@titanoboa42 Resque issues
@titanoboa42 • Child processes Resque issues
@titanoboa42 • Child processes • Memory hungry and slow Resque
issues
@titanoboa42 Sidekiq scales
@titanoboa42 • Redis - no relational db Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes Sidekiq scales
@titanoboa42 • Redis - no relational db • Threads instead
of child processes • Fast and less memory hungry Sidekiq scales
@titanoboa42 Sidekiq issues
@titanoboa42 • Requires thread safe code Sidekiq issues
@titanoboa42 Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments Long running
jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments • Not
cloud-friendly Long running jobs - Resque
@titanoboa42 • Aborted and requeued on shutdown Long running jobs
- Sidekiq
@titanoboa42 • Aborted and requeued on shutdown • Job may
not finish before being aborted again Long running jobs - Sidekiq
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Large collections
@titanoboa42 • Split job into collection and task to be
done Large collections
@titanoboa42 • Split job into collection and task to be
done • Checkpoint after iteration & requeue Large collections
@titanoboa42 Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job
with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention • Data
integrity Interruptible job with automatic resuming
@titanoboa42 Abstracting scaling issues simplifies concrete background jobs
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Background jobs
@titanoboa42 • Benefit apps of all sizes Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
Background jobs
@titanoboa42 • Benefit apps of all sizes • Require trade-offs
• Keep code simple at scale Background jobs
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
520
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
190
Background jobs at scale
titanoboa
1
60
Background jobs at scale
titanoboa
0
78
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
150
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
120
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
550
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
720
No RSVP required: Asynchronous Messaging
titanoboa
0
340
Other Decks in Programming
See All in Programming
TS 7.0 で baseUrl が使えなくなった今、パスエイリアスをどうするか
bicstone
2
700
動画プレイヤーはなぜカクつくのか 〜Flutterのレンダリングとネイティブの境界を解剖する〜
naruogram
0
340
AI時代のTerminal環境を UNIX哲学で設計・運用する
yasunori0418
1
360
Kotlin JVM用に作られたライブラリのKMPを前提とした再設計
jsoizo
0
140
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
leegeunhyeok
0
180
Augmenting AI with the Power of Jakarta EE
ivargrimstad
0
360
우리의 Nitro Modules는 당신의 Turbo Modules보다 아름답다
l2hyunwoo
0
270
Claude Codeのおかげでやりたいことがやれるようになったんだよォー!
sms_tech
0
120
あなたの await、OS まで届いてる? strace で覗く JavaScript の非同期
hasutosasaki
1
2.4k
学びを記事・スライド・Xへ高速展開する― AI-assisted Learning Loopで回すコンテンツ生成術
ochtum
0
210
graphql-ruby を Spinel で速くしようとした話
mtsmfm
0
150
LLM 101: Exploring LLMs, Starting with Tokens.
dalinaum
0
110
Featured
See All Featured
Scaling GitHub
holman
464
140k
Facilitating Awesome Meetings
lara
57
7.2k
Being A Developer After 40
akosma
PRO
92
590k
Digital Ethics as a Driver of Design Innovation
axbom
PRO
2
530
How Software Deployment tools have changed in the past 20 years
geshan
3
35k
Data-driven link building: lessons from a $708K investment (BrightonSEO talk)
szymonslowik
1
1.4k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
31
3.4k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
46
3.1k
Why Our Code Smells
bkeepers
PRO
340
58k
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
aleyda
2
12k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Marketing to machines
jonoalderson
1
6k