Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
EuRuKo 2018: Scaling a monolith isn't scaling m...
Search
Kerstin Puschke
August 25, 2018
Programming
2
140
EuRuKo 2018: Scaling a monolith isn't scaling microservices
talk at EuRuKo 2018 about background jobs and message oriented middleware
Kerstin Puschke
August 25, 2018
Tweet
Share
More Decks by Kerstin Puschke
See All by Kerstin Puschke
High availability by offloading work - background jobs, message queues, or Kafka
titanoboa
0
460
Background jobs at scale (Montreal.rb)
titanoboa
0
390
Background jobs at scale
titanoboa
1
58
Background jobs at scale
titanoboa
0
64
Decouple all the things: Asynchronous messaging keeps it simple
titanoboa
0
120
Decouple all the things - Asynchronous messaging keeps it simple
titanoboa
0
88
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
500
Decouple all the things - asynchronous messaging keeps it simple
titanoboa
0
640
No RSVP required: Asynchronous Messaging
titanoboa
0
320
Other Decks in Programming
See All in Programming
Serena MCPのすすめ
wadakatu
4
900
CSC305 Lecture 04
javiergs
PRO
0
250
ABEMAモバイルアプリが Kotlin Multiplatformと歩んだ5年 ─ 導入と運用、成功と課題 / iOSDC 2025
akkyie
0
320
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
370
LLMとPlaywright/reg-suitを活用した jQueryリファクタリングの実際
kinocoboy2
4
670
止められない医療アプリ、そっと Swift 6 へ
medley
1
120
私達はmodernize packageに夢を見るか feat. go/analysis, go/ast / Go Conference 2025
kaorumuta
2
490
After go func(): Goroutines Through a Beginner’s Eye
97vaibhav
0
230
デミカツ切り抜きで面倒くさいことはPythonにやらせよう
aokswork3
0
190
SpecKitでどこまでできる? コストはどれくらい?
leveragestech
0
520
GraphQL×Railsアプリのデータベース負荷分散 - 月間3,000万人利用サービスを無停止で
koxya
1
1.1k
育てるアーキテクチャ:戦い抜くPythonマイクロサービスの設計と進化戦略
fujidomoe
1
150
Featured
See All Featured
Scaling GitHub
holman
463
140k
Gamification - CAS2011
davidbonilla
81
5.5k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
30
2.9k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
45
2.5k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
Music & Morning Musume
bryan
46
6.8k
The Power of CSS Pseudo Elements
geoffreycrofte
79
6k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
32
2.2k
Code Review Best Practice
trishagee
72
19k
Designing Experiences People Love
moore
142
24k
Being A Developer After 40
akosma
91
590k
It's Worth the Effort
3n
187
28k
Transcript
Kerstin Puschke @titanoboa42 Scaling a monolith isn’t scaling microservices
None
@titanoboa42 Workload distribution in different architectures
@titanoboa42 • Background jobs Outline
@titanoboa42 • Background jobs • Features & Challenges Outline
@titanoboa42 • Background jobs • Features & Challenges • When
to choose Outline
@titanoboa42 • Message oriented middleware Outline
@titanoboa42 • Message oriented middleware • Features & Challenges Outline
@titanoboa42 • Message oriented middleware • Features & Challenges •
When to choose Outline
@titanoboa42 • Event log Outline
@titanoboa42 • Event log • Summary Outline
@titanoboa42 Background jobs
@titanoboa42 • Resque Background job backends
@titanoboa42 • Resque • Sidekiq Background job backends
@titanoboa42 • Resque • Sidekiq • … Background job backends
@titanoboa42 Background job: Unit of work to be done
later App Server Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker
@titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue
@titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker
Task Queue
@titanoboa42 Background job backend: task queue & broker
@titanoboa42 Encapsulating async communication
@titanoboa42 Features
@titanoboa42 Task Queue Response times App Server Worker
@titanoboa42 Task Queue Spikeability App Server Worker
@titanoboa42 Task Queue Parallelization App Server Worker Worker Worker
@titanoboa42 Task Queue Retries App Server Worker Worker Worker
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low
Prio Queue
@titanoboa42 Mastering challenges
@titanoboa42 No exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent jobs & at least once delivery No exactly once delivery
@titanoboa42 Out of order delivery
@titanoboa42 • If order matters, queue sequentially Out of order
delivery
@titanoboa42 • If order matters, queue sequentially • First job
queues follow up jobs Out of order delivery
@titanoboa42 Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown Long running jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments Long running
jobs - Resque
@titanoboa42 • Prevent worker shutdown • No deployments • Not
cloud-friendly Long running jobs - Resque
@titanoboa42 • Aborted and requeued Long running jobs - Sidekiq
@titanoboa42 • Aborted and requeued • Job may not finish
before being aborted again Long running jobs - Sidekiq
@titanoboa42 Large collections
@titanoboa42 • Split job into collection and task to be
done Large collections
@titanoboa42 • Split job into collection and task to be
done • Checkpoint after iteration & requeue Large collections
@titanoboa42 Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job
with automatic resuming
@titanoboa42 • Shutdown workers anytime • Disaster prevention • Data
integrity Interruptible job with automatic resuming
@titanoboa42 Abstracting scaling issues simplifies concrete background jobs
@titanoboa42 github.com /Shopify/job-iteration
@titanoboa42 When to choose background jobs
@titanoboa42 Task Queue Background jobs are ruby objects App Server
Worker
@titanoboa42 Task Queue Background jobs are ruby objects App Server
Worker Broker Broker
@titanoboa42 Great fit for monolithic architecture
@titanoboa42 Background jobs Summary
@titanoboa42 • Great features based on task queues Background jobs
@titanoboa42 • Great features based on task queues • Abstracting
the complexity of jobs running in the future and / or in parallel Background jobs
@titanoboa42 • Simple concrete jobs Background jobs
@titanoboa42 • Simple concrete jobs • Complex overall system Background
jobs
@titanoboa42 • Simple concrete jobs • Complex overall system •
Great for monolithic architecture Background jobs
@titanoboa42 Message oriented middleware
@titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… Message oriented middleware
@titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… • Protocols: AMQP, MQTT, Stomp,…
Message oriented middleware
@titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface
Worker Consumer
@titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface
Worker Consumer Broker
@titanoboa42 Features
@titanoboa42 Decoupling
@titanoboa42 Interoperability
@titanoboa42 Command messages
@titanoboa42 Event messages
@titanoboa42 Propagating updates Business Partners Support Contracts Orders
@titanoboa42 Propagating updates Business Partners Support Contracts Orders
@titanoboa42 REST/webhooks has disadvantages Business Partners Support Contracts Orders
@titanoboa42 REST/webhooks has disadvantages Business Partners Orders
@titanoboa42 REST/webhooks has disadvantages Business Partners Support Contracts Orders
@titanoboa42 REST/webhooks has disadvantages Business Partners Support Contracts Orders Invoices
@titanoboa42 Messaging Middleware Resiliency Business Partners Invoices
@titanoboa42 Messaging Middleware Resiliency Business Partners
@titanoboa42 Messaging Middleware Resiliency Business Partners Invoices
@titanoboa42 Topic with queues provides advanced routing App Server Business
Partners Support Contracts Orders Messaging Middleware
@titanoboa42 Topic with queues provides advanced routing App Server Message
Queue Business Partners Support Contracts Orders Message Queue Messaging Middleware
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners
Support Contracts Orders
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners
Invoices Support Contracts Orders
@titanoboa42 Messaging Middleware Anonymity for producer and consumer FraudScore Orders
Support Contracts
@titanoboa42 Messaging Middleware Anonymity for producer and consumer Invoices FraudScore
Orders Support Contracts
@titanoboa42 Mastering challenges
@titanoboa42 Keep breaking changes manageable
@titanoboa42 • Avoid n:m routing Keep breaking changes manageable
@titanoboa42 • Avoid n:m routing • Better representation of domain:
multiple messages routed 1:n, n:1 instead Keep breaking changes manageable
@titanoboa42 • Messages are kept if not consumed Remove queues
if consumer is gone
@titanoboa42 • Messages are kept if not consumed • Growing
queue becomes problematic Remove queues if consumer is gone
@titanoboa42 • “At least” vs. “at most” once delivery No
exactly once delivery
@titanoboa42 • “At least” vs. “at most” once delivery •
Idempotent consumer & at least once delivery No exactly once delivery
@titanoboa42 • If order matters, queue sequentially Out of order
delivery
@titanoboa42 • If order matters, queue sequentially • First consumer
queues second message Out of order delivery
@titanoboa42 Event-carried state transfer requires order information
@titanoboa42 • Payload contains updated data Event-carried state transfer requires
order information
@titanoboa42 • Payload contains updated data • Avoids requests to
service that owns data Event-carried state transfer requires order information
@titanoboa42 • Payload contains updated data • Avoids requests to
service that owns data • Provide order information (e.g. updated_at) Event-carried state transfer requires order information
@titanoboa42 Event notifications are commutative
@titanoboa42 • Payload has id only Event notifications are commutative
@titanoboa42 • Payload has id only • Consumer requests data
separately Event notifications are commutative
@titanoboa42 • Payload has id only • Consumer requests data
separately • Consumer doesn’t learn every state Event notifications are commutative
@titanoboa42 No replayability
@titanoboa42 • Messages removed after processing No replayability
@titanoboa42 • Messages removed after processing • No single source
of truth for event sourcing No replayability
@titanoboa42 When to choose message oriented middleware
@titanoboa42 Great fit for stateful, distributed architecture
@titanoboa42 Message oriented middleware Summary
@titanoboa42 • Great features based on queues and topics Message
oriented middleware
@titanoboa42 • Great features based on queues and topics •
Abstracting the complexity of pub sub Message oriented middleware
@titanoboa42 • Abstracting the complexity of messages being consumed in
the future / in parallel Message oriented middleware
@titanoboa42 • Abstracting the complexity of messages being consumed in
the future / in parallel • Simple concrete message consumers Message oriented middleware
@titanoboa42 • Complex overall system Message oriented middleware
@titanoboa42 • Complex overall system • Great for stateful, distributed
architecture Message oriented middleware
@titanoboa42 Event log
@titanoboa42 • Kafka Event logs
@titanoboa42 • Kafka • … Event logs
@titanoboa42 • Events persisted into append-only log Event log
@titanoboa42 • Events persisted into append-only log • Consumers read
shared log Event log
@titanoboa42 • Events persisted into append-only log • Consumers read
shared log • Stateless broker (no queues) Event log
@titanoboa42 Consumers easy to remove
@titanoboa42 High throughput data streaming
@titanoboa42 Replayability
@titanoboa42 • Single source of truth for event sourcing Replayability
@titanoboa42 Great fit for event sourcing and real time
applications
@titanoboa42 Summary
@titanoboa42 • Queues Background jobs
@titanoboa42 • Queues • Simple concrete jobs Background jobs
@titanoboa42 • Queues • Simple concrete jobs • For monolithic
architecture Background jobs
@titanoboa42 • Topics and Queues Message oriented middleware
@titanoboa42 • Topics and Queues • Simple, stateless message consumers
Message oriented middleware
@titanoboa42 • Topics and Queues • Simple, stateless message consumers
• For distributed, stateful architecture Message oriented middleware
@titanoboa42 • Shared log, no queues Event logs
@titanoboa42 • Shared log, no queues • Stateful consumers Event
logs
@titanoboa42 • Shared log, no queues • Stateful consumers •
For event sourcing & real time applications Event logs
@titanoboa42 BFCM video
@titanoboa42 BFCM video
Thanks! Questions? @titanoboa42 https://www.shopify.com/careers