Processes & Threads - Resque vs. Sidekiq

From RailsConf 2015

Transcript

1. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com
2. Read, read, read. Read everything – trash, classics, good and bad, and see how they do it. — William Faulkner, on writing
3. Background Workers
4. Resque Developed at Github, ca. 2009 » Let Redis handle the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
5. Resque — How?
6. fork(2)
7. Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1 if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
8. Forking Produces (modulo different pids) 57388: Waiting on child 57416: Child doing work 57416: a=2 57388: Done with fork. a=1
9. Forking The child process » springs into existence » performs its job » dies
10. Resque — How?
11. Resque, distilled start # Initialize, register signal handlers, &c. loop do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
12. Resque — Why?
13. “Assume chaos” — Resque
14. Forking » mitigates the risk of leaking memory » avoids the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
15. Sidekiq — Why?
16. Sidekiq By @mperham, ca. 2012 » "What if one process could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
17. Sidekiq — How?
18. Thread1 1 actually, pthread_create(3)
19. Threads Pros: shared memory Cons: shared memory
20. Race Conditions @wallet = 100 Thread.new { @wallet += 10 } Thread.new { @wallet -= 10 }
21. Race Conditions @wallet = 100 Thread.new do tmp = @wallet sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
22. Actors & Celluloid
23. class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount = amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
24. Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The Celluloid Ecosystem"
25. Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The Celluloid Ecosystem"
26. Sidekiq — How?
27. Takeaways If you're: » memory-constrained - use Sidekiq » not thread-safe - use Resque » both - ¯\_(
    )_/¯
28. Takeaways Considerations: » How important is job isolation? » What's your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
29. Takeaways Don't be afraid to dive in to code » pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
30. Further Reading Crack open a good gem » Unicorn » resque-pool » Adhearson
31. Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball - Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
32. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com