Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Processes & Threads - Resque vs. Sidekiq
Search
James Dabbs
April 21, 2015
Technology
1
7.2k
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Tweet
Share
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
360
Other Decks in Technology
See All in Technology
CursorによるPMO業務の代替 / Automating PMO Tasks with Cursor
motoyoshi_kakaku
2
840
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
3
940
「良さそう」と「とても良い」の間には 「良さそうだがホンマか」がたくさんある / 2025.07.01 LLM品質Night
smiyawaki0820
1
460
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
26k
OPENLOGI Company Profile
hr01
0
67k
Liquid Glass革新とSwiftUI/UIKit進化
fumiyasac0921
0
310
KiCadでPad on Viaの基板作ってみた
iotengineer22
0
240
ハッカソン by 生成AIハッカソンvol.05
1ftseabass
PRO
0
160
タイミーのデータモデリング事例と今後のチャレンジ
ttccddtoki
6
2.1k
自律的なスケーリング手法FASTにおけるVPoEとしてのアカウンタビリティ / dev-productivity-con-2025
yoshikiiida
1
7k
怖くない!はじめてのClaude Code
shinya337
0
320
WordPressから ヘッドレスCMSへ! Storyblokへの移行プロセス
nyata
0
360
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Scaling GitHub
holman
459
140k
Typedesign – Prime Four
hannesfritz
42
2.7k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Gamification - CAS2011
davidbonilla
81
5.3k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Docker and Python
trallard
44
3.5k
What's in a price? How to price your products and services
michaelherold
246
12k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Optimizing for Happiness
mojombo
379
70k
Writing Fast Ruby
sferik
628
62k
Rails Girls Zürich Keynote
gr2m
94
14k
Transcript
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]