Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Processes & Threads - Resque vs. Sidekiq
Search
James Dabbs
April 21, 2015
Technology
1
6.7k
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Tweet
Share
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
270
Other Decks in Technology
See All in Technology
LangSmith入門―トレース/評価/プロンプト管理などを担うLLMアプリ開発プラットフォーム
os1ma
3
120
MapLibreとAmazon Location Service
dayjournal
1
150
私が trocco を推す理由
__allllllllez__
1
220
アクセシビリティを考慮したUI/CSSフレームワーク・ライブラリ選定
yajihum
2
1k
レガシーをぶっ壊せ。AEONで始めるDevRelの話 / Qiita Night 2024-2-22
aeonpeople
3
1.3k
GraphQL 成熟度モデルの紹介と、プロダクトに当てはめた事例 / GraphQL maturity model
mh4gf
7
1.3k
障害対応をちょっとずつよくしていくための 演習の作りかた
heleeen
0
190
元インフラエンジニアに成る / Human Resources to Human Relations
bobtani
4
920
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
2
470
Tellus の衛星データを見てみよう #mf_fukuoka
kongmingstrap
0
190
推しは推せるときに推せ! プロダクトにフィードバックしていこう
nakasho
0
310
ExaDB-D dbaascli で出来ること
oracle4engineer
PRO
0
2.1k
Featured
See All Featured
Building an army of robots
kneath
300
41k
Git: the NoSQL Database
bkeepers
PRO
422
63k
Adopting Sorbet at Scale
ufuk
68
8.6k
What's new in Ruby 2.0
geeforr
337
31k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
BBQ
matthewcrist
80
8.8k
Agile that works and the tools we love
rasmusluckow
325
20k
A Philosophy of Restraint
colly
197
16k
Faster Mobile Websites
deanohume
299
30k
The Art of Programming - Codeland 2020
erikaheidi
42
12k
How GitHub Uses GitHub to Build GitHub
holman
468
290k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3k
Transcript
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]