Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Processes & Threads - Resque vs. Sidekiq
Search
James Dabbs
April 21, 2015
Technology
1
7.2k
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Tweet
Share
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
370
Other Decks in Technology
See All in Technology
見てわかるテスト駆動開発
recruitengineers
PRO
3
230
ABEMAにおける 生成AI活用の現在地 / The Current Status of Generative AI at ABEMA
dekatotoro
0
640
KiroでGameDay開催してみよう(準備編)
yuuuuuuu168
1
120
Yahoo!広告ビジネス基盤におけるバックエンド開発
lycorptech_jp
PRO
1
270
アジャイルテストで高品質のスプリントレビューを
takesection
0
110
ウォンテッドリーのアラート設計と Datadog 移行での知見
donkomura
0
310
実践アプリケーション設計 ①データモデルとドメインモデル
recruitengineers
PRO
2
180
自治体職員がガバクラの AWS 閉域ネットワークを理解するのにやって良かった個人検証環境
takeda_h
2
380
会社にデータエンジニアがいることでできるようになること
10xinc
9
1.5k
モダンフロントエンド 開発研修
recruitengineers
PRO
2
220
Browser
recruitengineers
PRO
3
220
LLM時代の検索とコンテキストエンジニアリング
shibuiwilliam
2
1.1k
Featured
See All Featured
RailsConf 2023
tenderlove
30
1.2k
GitHub's CSS Performance
jonrohan
1031
460k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
1k
Gamification - CAS2011
davidbonilla
81
5.4k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
For a Future-Friendly Web
brad_frost
179
9.9k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
131
19k
The Language of Interfaces
destraynor
160
25k
A designer walks into a library…
pauljervisheath
207
24k
Site-Speed That Sticks
csswizardry
10
780
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Transcript
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]