Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Asynchronous Processing for Fun and Profit
Search
Mike Perham
November 02, 2012
Technology
11
880
Asynchronous Processing for Fun and Profit
Some pro tips and an overview of Sidekiq for great async victory!
Mike Perham
November 02, 2012
Tweet
Share
More Decks by Mike Perham
See All by Mike Perham
Computer Science vs Software Engineering
mperham
0
330
A Tribute to Ezra Zygmuntowicz
mperham
1
1k
Other Decks in Technology
See All in Technology
Tech-Verse 2025 Global CTO Session
lycorptech_jp
PRO
0
1.2k
事業成長の裏側:エンジニア組織と開発生産性の進化 / 20250703 Rinto Ikenoue
shift_evolve
PRO
1
130
Oracle Cloud Infrastructure:2025年6月度サービス・アップデート
oracle4engineer
PRO
2
310
Zephyr RTOSを使った開発コンペに参加した件
iotengineer22
0
130
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
2
240
Model Mondays S2E03: SLMs & Reasoning
nitya
0
240
AI導入の理想と現実~コストと浸透〜
oprstchn
0
150
OpenHands🤲にContributeしてみた
kotauchisunsun
1
500
整頓のジレンマとの戦い〜Tidy First?で振り返る事業とキャリアの歩み〜/Fighting the tidiness dilemma〜Business and Career Milestones Reflected on in Tidy First?〜
bitkey
0
220
生成AI活用の組織格差を解消する 〜ビジネス職のCursor導入が開発効率に与えた好循環〜 / Closing the Organizational Gap in AI Adoption
upamune
5
4.5k
fukabori.fm 出張版: 売上高617億円と高稼働率を陰で支えた社内ツール開発のあれこれ話 / 20250704 Yoshimasa Iwase & Tomoo Morikawa
shift_evolve
PRO
1
110
使いたいMCPサーバーはWeb APIをラップして自分で作る #QiitaBash
bengo4com
0
1.2k
Featured
See All Featured
Rails Girls Zürich Keynote
gr2m
94
14k
It's Worth the Effort
3n
185
28k
Agile that works and the tools we love
rasmusluckow
329
21k
Six Lessons from altMBA
skipperchong
28
3.9k
How STYLIGHT went responsive
nonsquared
100
5.6k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.8k
Mobile First: as difficult as doing things right
swwweet
223
9.7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Embracing the Ebb and Flow
colly
86
4.7k
The World Runs on Bad Software
bkeepers
PRO
69
11k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
680
Transcript
Async Processing for Fun and Profit Mike Perham @mperham
Me Director of Engineering, TheClymb.com
Agenda • Basics • Protips • Sidekiq
Why? • User-Perceived Performance • I/O is SLOOOOOOOOOOOW • I/O
is unreliable
What? • Optional work • Anything not required to build
the HTTP response
How? • Client puts message on a queue • Server
pulls message off queue • Worker executes code based on message
How? async do # perform some work end
How? • Marshalling a Proc • Need to serialize closure
How? async(instance, :method, args)
How? • Marshal instance • Marshal args
How? async(Class, :method, args)
How? • Serialize just the class name • Marshal the
arguments
Congratulations! This is exactly how Resque and Sidekiq work.
How?
How?
How?
Tip #1 Small, Stateless Messages
Stateless • Database holds objects (nouns) • Queue holds actions
(verbs) • “Perform X on Object 123”
Avoid State • Bad • @user.delay.sync_images • Good • User.delay.sync_images(@user.id)
Simple Types • Small & Easy to read • Cross-platform
• Sidekiq / Resque use JSON
Debugging
Tip #2 Idempotent, Transactional Units of Work
Idempotent • Fancy computer science term • “Work can be
applied multiple times without changing the result beyond the initial application”
Idempotent • Canceling an order • Updating user’s email address
Not Idempotent! • Charging credit card • Sending email
Idempotent • Your code has bugs! • Sidekiq will retry
jobs that raise • Design jobs to be retried • e.g. verify you need to perform action before performing it
Transactional • Infinite credit?
Tip #3 Embrace Concurrency
Concurrency • Resque/DJ = 4 workers • Sidekiq = 100
workers • Mike, what is best in life?
“To crush their servers, see them smoking before you and
hear the lamentations of their admins.”
Concurrency • Use connection_pool gem to limit client connections •
Split work into small batches • 100 items => 10 jobs of 10 items
Concurrency • Thread safety rarely an issue • Most gem
maintainers very responsive • Recently fixed: • cocaine, typheuos
Theory, meet Practice
Sidekiq • Simple, efficient message processing • Like Resque, but
10x faster
MODERN RUBY IS NOT SLOW SINGLE THREADING IS SLOW
Concurrency • To scale single-threaded, create lots of processes. •
HORRIBLY RAM INEFFICIENT
Threads vs Processes • Example: 400MB single-threaded process • 25
processes = 10GB RAM = EC2 xlarge • 25 threads = 1GB RAM = EC2 small • $60/month vs $480/month • 160 dynos => 10 dynos • 150 dynos * $35/month =~ $5000/month
Quick Rant • GIL + poor GC • C extension
API must DIE • What’s larger: 50% or 800%?
Client Your App Client Middleware Redis Sidekiq Client API Rails
Process
Server Processor Server Middleware Redis Worker Sidekiq Process Processor Server
Middleware Worker Processor Server Middleware Worker Processor Server Middleware Worker Fetcher Manager
Versions • Sidekiq - Free, LGPLv3 • Sidekiq Pro -
more features, support, $ • motivation!
Features Sidekiq Resque DJ Concurrency Store Hooks Web UI Scheduler
Retry Delay Batches Threads Processes Processes Redis Redis DB middleware callbacks callbacks ✓ ✓ ? ✓ ? ✓ ✓ ? ✓ ✓ ? ✓ Pro ? - ? - optional
Future • Nicer, more functional Web UI • APIs for
managing queues / retries • Rails 4 Queue API
Pro Future • Enterprise-y features • Workflow • Notifications
Conclusion • small, stateless messages • idempotent / transactional •
concurrency • sidekiq
Questions? @mperham
[email protected]