Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Processes & Threads - Resque vs. Sidekiq
Search
James Dabbs
April 21, 2015
Technology
1
7.2k
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Tweet
Share
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
380
Other Decks in Technology
See All in Technology
Shirankedo NOCで見えてきたeduroam/OpenRoaming運用ノウハウと課題 - BAKUCHIKU BANBAN #2
marokiki
0
170
AWS Top Engineer、浮いてませんか? / As an AWS Top Engineer, Are You Out of Place?
yuj1osm
2
200
社内報はAIにやらせよう / Let AI handle the company newsletter
saka2jp
8
1.3k
フルカイテン株式会社 エンジニア向け採用資料
fullkaiten
0
9.1k
自動テストのコストと向き合ってみた
qa
0
210
【Kaigi on Rails 事後勉強会LT】MeはどうしてGirlsに? 私とRubyを繋いだRail(s)
joyfrommasara
0
210
10年の共創が示す、これからの開発者と企業の関係 ~ Crossroad
soracom
PRO
1
690
生成AIとM5Stack / M5 Japan Tour 2025 Autumn 東京
you
PRO
0
240
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
3
5.5k
業務効率化をさらに加速させる、ノーコードツールとStep Functionsのハイブリッド化
smt7174
2
110
Escaping_the_Kraken_-_October_2025.pdf
mdalmijn
0
150
AIツールでどこまでデザインを忠実に実装できるのか
oikon48
6
3.1k
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Done Done
chrislema
185
16k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Art, The Web, and Tiny UX
lynnandtonic
303
21k
The Straight Up "How To Draw Better" Workshop
denniskardys
237
140k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Unsuck your backbone
ammeep
671
58k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
54
3k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
Transcript
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]