Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Processes & Threads - Resque vs. Sidekiq
Search
James Dabbs
April 21, 2015
Technology
1
7.2k
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Tweet
Share
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
380
Other Decks in Technology
See All in Technology
20201008_ファインディ_品質意識を育てる役目は人かAIか___2_.pdf
findy_eventslides
1
520
Optuna DashboardにおけるPLaMo2連携機能の紹介 / PFN LLM セミナー
pfn
PRO
2
900
VCC 2025 Write-up
bata_24
0
180
フルカイテン株式会社 エンジニア向け採用資料
fullkaiten
0
9.1k
E2Eテスト設計_自動化のリアル___Playwrightでの実践とMCPの試み__AIによるテスト観点作成_.pdf
findy_eventslides
1
490
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
11
78k
データエンジニアがこの先生きのこるには...?
10xinc
0
450
いまさら聞けない ABテスト入門
skmr2348
1
210
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
OCI Network Firewall 概要
oracle4engineer
PRO
1
7.8k
ACA でMAGI システムを社内で展開しようとした話
mappie_kochi
1
290
extension 現場で使えるXcodeショートカット一覧
ktombow
0
220
Featured
See All Featured
Gamification - CAS2011
davidbonilla
81
5.5k
KATA
mclloyd
32
15k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Docker and Python
trallard
46
3.6k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.2k
The Invisible Side of Design
smashingmag
301
51k
The Cost Of JavaScript in 2023
addyosmani
53
9k
Faster Mobile Websites
deanohume
310
31k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
189
55k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Transcript
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]