Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Processes & Threads - Resque vs. Sidekiq
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
James Dabbs
April 21, 2015
Technology
7.5k
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Processes & Threads - Resque vs. Sidekiq
From RailsConf 2015
James Dabbs
April 21, 2015
Transcript
32 slides
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
Read, read, read. Read everything – trash, classics, good and
bad, and see how they do it. — William Faulkner, on writing
Background Workers
Resque Developed at Github, ca. 2009 » Let Redis handle
the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
Resque — How?
fork(2)
Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1
if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
Forking Produces (modulo different pids) 57388: Waiting on child 57416:
Child doing work 57416: a=2 57388: Done with fork. a=1
Forking The child process » springs into existence » performs
its job » dies
Resque — How?
Resque, distilled start # Initialize, register signal handlers, &c. loop
do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
Resque — Why?
“Assume chaos” — Resque
Forking » mitigates the risk of leaking memory » avoids
the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
Sidekiq — Why?
Sidekiq By @mperham, ca. 2012 » "What if one process
could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
Sidekiq — How?
Thread1 1 actually, pthread_create(3)
Threads Pros: shared memory Cons: shared memory
Race Conditions @wallet = 100 Thread.new { @wallet += 10
} Thread.new { @wallet -= 10 }
Race Conditions @wallet = 100 Thread.new do tmp = @wallet
sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
Actors & Celluloid
class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =
amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The
Celluloid Ecosystem"
Sidekiq — How?
Takeaways If you're: » memory-constrained - use Sidekiq » not
thread-safe - use Resque » both - ¯\_()_/¯
Takeaways Considerations: » How important is job isolation? » What's
your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
Takeaways Don't be afraid to dive in to code »
pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
Further Reading Crack open a good gem » Unicorn »
resque-pool » Adhearson
Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball
- Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
Processes & Threads Resque vs Sidekiq @jamesdabbs |
[email protected]
More Decks by James Dabbs
See All by James Dabbs
Monads @ ATLRUG
jdabbs
3
440
Other Decks in Technology
See All in Technology
飯から始まるデータエンジニア
sansantech
PRO
0
120
文書画像からの情報抽出を行うエージェントを、軌跡の正解データ無しで作れるか
sansantech
PRO
2
220
AIで実装が加速したのに、 なぜ私は仕事の大半で人と話し続けていたのか?
say_no_w
0
200
SORACOM × AWS IoT Core でSREする
butadora
0
120
市販IoT機器や自作IoT機器とHomeAssistantを駆使して家の電気代をできる限り見える化する方法の考案
harutiro
0
400
認知負荷の減らし方 ~開発にも日常業務にも活かせるノウハウ~
minorun365
PRO
12
8.3k
社内アプリ基盤「Quartz」による、 非エンジニアとAIに本番デプロイを 任せても壊れないプラットフォームの設計
staka121
PRO
1
120
OpenTelemetryで変わった 新時代のCloudWatchメトリクス監視
tsukuboshi
0
470
Taking back control of your AI development
inesmontani
PRO
0
300
AWS DevOps Agentの構造から読み解く運用 AI エージェントの設計原則/aws-devops-agent-design-principles
tomoki10
0
140
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
10
5.9k
常時稼働エージェントの作り方(Claude公式ブログを資料化)
minorun365
PRO
17
28k
Featured
See All Featured
Making the Leap to Tech Lead
cromwellryan
135
10k
So, you think you're a good person
axbom
PRO
2
2.3k
Claude Code どこまでも/ Claude Code Everywhere
nwiizo
68
61k
A Guide to Academic Writing Using Generative AI - A Workshop
ks91
PRO
1
570
Designing Dashboards & Data Visualisations in Web Apps
destraynor
232
55k
HTML-Aware ERB: The Path to Reactive Rendering @ RubyCon 2026, Rimini, Italy
marcoroth
6
960
GraphQLの誤解/rethinking-graphql
sonatard
75
12k
GraphQLとの向き合い方2022年版
quramy
50
15k
The Invisible Side of Design
smashingmag
301
52k
Product Roadmaps are Hard
iamctodd
55
13k
Imperfection Machines: The Place of Print at Facebook
scottboms
270
14k
Public Speaking Without Barfing On Your Shoes - THAT 2023
reverentgeek
1
670