Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
High availability by offloading work - backgrou...
Search
Kerstin Puschke
September 20, 2019
Technology
0
460
High availability by offloading work - background jobs, message queues, or Kafka
Talk at RubyConf Colombia 2019
Kerstin Puschke
September 20, 2019
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
Background jobs at scale (Montreal.rb)
titanoboa
0
380
EuRuKo 2018: Scaling a monolith isn't scaling microservices
titanoboa
2
130
Background jobs at scale
titanoboa
1
58
Background jobs at scale
titanoboa
0
64
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
120
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
88
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
490
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
630
No RSVP required: Asynchronous Messaging
titanoboa
0
310
Other Decks in Technology
See All in Technology
ゼロからはじめる採用広報
yutadayo
3
950
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
7
5.3k
第4回Snowflake 金融ユーザー会 Snowflake summit recap
tamaoki
1
280
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
1.9k
American airlines ®️ USA Contact Numbers: Complete 2025 Support Guide
airhelpsupport
0
380
20250707-AI活用の個人差を埋めるチームづくり
shnjtk
4
3.9k
開発生産性を測る前にやるべきこと - 組織改善の実践 / Before Measuring Dev Productivity
kaonavi
10
4.6k
ビズリーチにおけるリアーキテクティング実践事例 / JJUG CCC 2025 Spring
visional_engineering_and_design
1
120
SmartNewsにおける 1000+ノード規模 K8s基盤 でのコスト最適化 – Spot・Gravitonの大規模導入への挑戦
vsanna2
0
130
MUITにおける開発プロセスモダナイズの取り組みと開発生産性可視化の取り組みについて / Modernize the Development Process and Visualize Development Productivity at MUIT
muit
1
16k
KubeCon + CloudNativeCon Japan 2025 Recap Opening & Choose Your Own Adventureシリーズまとめ
mmmatsuda
0
280
敢えて生成AIを使わないマネジメント業務
kzkmaeda
2
450
Featured
See All Featured
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Bash Introduction
62gerente
613
210k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Documentation Writing (for coders)
carmenintech
72
4.9k
Six Lessons from altMBA
skipperchong
28
3.9k
Faster Mobile Websites
deanohume
307
31k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
970
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
107
19k
4 Signs Your Business is Dying
shpigford
184
22k
Practical Orchestrator
shlominoach
189
11k
Unsuck your backbone
ammeep
671
58k
Transcript
Kerstin Puschke @titanoboa42 High availability by offloading work - background
jobs, message queues, or Kafka
@titanoboa42
@titanoboa42 Different approaches to offload work to ensure high availability
@titanoboa42 • High availability & offloading work Outline
@titanoboa42 • High availability & offloading work • Background jobs
Outline
@titanoboa42 • High availability & offloading work • Background jobs
• Message oriented middleware Outline
@titanoboa42 • Event log Outline
@titanoboa42 • Event log • Summary Outline
@titanoboa42 High availability & offloading work
@titanoboa42 High Availability
@titanoboa42 users can interact with the system High Availability
@titanoboa42 users can interact meaningfully with the system High
Availability
@titanoboa42 community of users can interact meaningfully with the system
High Availability
@titanoboa42 community of users can interact meaningfully with the system
whenever needed High Availability
@titanoboa42 community of users can interact meaningfully with the system
whenever needed High Availability community of users can interact meaningfully with the system whenever needed
@titanoboa42 Background jobs
@titanoboa42 • Resque Background job backends
@titanoboa42 • Resque • Sidekiq Background job backends
@titanoboa42 • Resque • Sidekiq • … Background job backends
@titanoboa42 Background job: Unit of work to be done
later App Server Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue
@titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker
Task Queue
@titanoboa42 Background job backend: task queue & broker
@titanoboa42 Encapsulating async communication
@titanoboa42 Features
@titanoboa42 Task Queue Response times App Server Worker
@titanoboa42 Task Queue Spikeability App Server Worker
@titanoboa42 Task Queue Parallelization App Server Worker Worker Worker
@titanoboa42 Task Queue Retries App Server Worker Worker Worker
@titanoboa42 Mastering challenges
@titanoboa42 Job queued and processed by different versions
@titanoboa42 • No breaking changes to job parameters Job queued
and processed by different versions
@titanoboa42 • No breaking changes to job parameters • Changes
need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
@titanoboa42 No exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent jobs & at least once delivery No exactly once delivery
@titanoboa42 Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction Non-transactional
queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or in case of rollback Non-transactional queuing
@titanoboa42 • Don’t queue from within a db transaction •
Job runs before commit, or in case of rollback • Commit first: Job not guaranteed to be queued Non-transactional queuing
@titanoboa42 Non-transactional queuing
@titanoboa42 • Stage transactionally Non-transactional queuing
@titanoboa42 • Stage transactionally • Scheduler queues job, updates staging
data Non-transactional queuing
@titanoboa42 Local transactions
@titanoboa42 • Eventual consistency at best Local transactions
@titanoboa42 • Eventual consistency at best • SAGA command/orchestration Local
transactions
@titanoboa42 Out of order delivery
@titanoboa42 • SAGA events/choreography: jobs queue jobs Out of order
delivery
@titanoboa42 • SAGA events/choreography: jobs queue jobs • easy to
build, hard to maintain or debug Out of order delivery
@titanoboa42 • SAGA events/choreography: jobs queue jobs • easy to
build, hard to maintain or debug • SAGA command/orchestrator Out of order delivery
@titanoboa42 Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments Long running
jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments • Not
cloud-friendly Long running jobs - Resque
@titanoboa42 • Aborted and requeued Long running jobs - Sidekiq
@titanoboa42 • Aborted and requeued • Job may not finish
before being aborted again Long running jobs - Sidekiq
@titanoboa42 Large collections
@titanoboa42 • Split job into collection and task to be
done Large collections
@titanoboa42 • Split job into collection and task to be
done • Checkpoint after iteration & requeue Large collections
@titanoboa42 Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job
with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention • Data
integrity Interruptible job with automatic resuming
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 Abstracting scaling issues simplifies concrete background jobs
@titanoboa42 Task Queue Background jobs are ruby objects App Server
Worker
@titanoboa42 Task Queue Background jobs are ruby objects App Server
Worker Broker Broker
@titanoboa42 Offloading work to a worker running the same code
base
@titanoboa42 Background jobs Summary
@titanoboa42 • Based on task queues Background jobs
@titanoboa42 • Based on task queues • Complex overall system,
simple concrete jobs Background jobs
@titanoboa42 • Based on task queues • Complex overall system,
simple concrete jobs • Great for monolith Background jobs
@titanoboa42 Message oriented middleware
@titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… Message oriented middleware
@titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… • Protocols: AMQP, MQTT, Stomp,…
Message oriented middleware
@titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface
Worker Consumer
@titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface
Worker Consumer Broker
@titanoboa42 Features
@titanoboa42 Commands & Events
@titanoboa42 Propagating updates Business Partners Support Contracts Orders
@titanoboa42 Propagating updates Business Partners Support Contracts Orders
@titanoboa42 Messaging Middleware Resiliency Business Partners Orders
@titanoboa42 Messaging Middleware Resiliency Business Partners
@titanoboa42 Messaging Middleware Resiliency Business Partners Orders
@titanoboa42 Topic with queues provides advanced routing App Server Business
Partners Support Contracts Orders Messaging Middleware
@titanoboa42 Topic with queues provides advanced routing App Server Message
Queue Business Partners Support Contracts Orders Message Queue Messaging Middleware
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners
Support Contracts Orders
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners
Invoices Support Contracts Orders
@titanoboa42 Messaging Middleware Anonymity for producer and consumer FraudScore Orders
Support Contracts
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Invoices FraudScore
Orders Support Contracts
@titanoboa42 Mastering challenges
@titanoboa42 Keep breaking changes manageable
@titanoboa42 • Avoid n:m routing Keep breaking changes manageable
@titanoboa42 • Avoid n:m routing • Better representation of domain:
multiple messages routed 1:n, n:1 instead Keep breaking changes manageable
@titanoboa42 • No exactly once delivery Lack of guarantees
@titanoboa42 • No exactly once delivery • No strong consistency
Lack of guarantees
@titanoboa42 • No exactly once delivery • No strong consistency
• Out of order delivery Lack of guarantees
@titanoboa42 No single source of truth
@titanoboa42 • Messages removed after processing No single source of
truth
@titanoboa42 • Messages removed after processing • No replayability No
single source of truth
@titanoboa42 Offloading work to decoupled services with no notion of
system wide state
@titanoboa42 Message oriented middleware Summary
@titanoboa42 • Based on queues and topics Message oriented middleware
@titanoboa42 • Based on queues and topics • Complex overall
system Message oriented middleware
@titanoboa42 • Based on queues and topics • Complex overall
system • Simple message consumers Message oriented middleware
@titanoboa42 • Great for decoupled microservices Message oriented middleware
@titanoboa42 • Great for decoupled microservices • No system wide
state Message oriented middleware
@titanoboa42 Event log
@titanoboa42 • Kafka Event logs
@titanoboa42 • Kafka • … Event logs
@titanoboa42 • Events persisted into append-only log Event log
@titanoboa42 • Events persisted into append-only log • Consumers read
shared log Event log
@titanoboa42 • Events persisted into append-only log • Consumers read
shared log • Stateless broker (no queues) Event log
@titanoboa42 High throughput
@titanoboa42 Single source of truth
@titanoboa42 • Events are not removed after processing Single source
of truth
@titanoboa42 • Events are not removed after processing • Replayability
Single source of truth
@titanoboa42 Offloading work to services keeping the notion of system
wide state
@titanoboa42 Event log Summary
@titanoboa42 • Based on shared log, no queues Event logs
@titanoboa42 • Based on shared log, no queues • Complex
overall system Event logs
@titanoboa42 Event logs
@titanoboa42 • Single source of truth (e.g. for event sourcing)
Event logs
@titanoboa42 • Single source of truth (e.g. for event sourcing)
• High throughput applications Event logs
@titanoboa42 Summary
@titanoboa42 • Queues Background jobs
@titanoboa42 • Queues • For monolithic code base Background jobs
@titanoboa42 • Topics and Queues Message oriented middleware
@titanoboa42 • Topics and Queues • For decoupled microservices Message
oriented middleware
@titanoboa42 • Shared log, no queues Event logs
@titanoboa42 • Shared log, no queues • For event sourcing
& high throughput applications Event logs
@titanoboa42 BFCM video
@titanoboa42 BFCM video
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers