Processes & Threads - Resque vs. Sidekiq

Processes & Threads - Resque vs. Sidekiq

From RailsConf 2015

908eb2dedca30a0209477244e069b22c?s=128

James Dabbs

April 21, 2015
Tweet

Transcript

  1. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com

  2. Read, read, read. Read everything – trash, classics, good and

    bad, and see how they do it. — William Faulkner, on writing
  3. Background Workers

  4. Resque Developed at Github, ca. 2009 » Let Redis handle

    the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness
  5. Resque — How?

  6. fork(2)

  7. Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1

    if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end
  8. Forking Produces (modulo different pids) 57388: Waiting on child 57416:

    Child doing work 57416: a=2 57388: Done with fork. a=1
  9. Forking The child process » springs into existence » performs

    its job » dies
  10. Resque — How?

  11. Resque, distilled start # Initialize, register signal handlers, &c. loop

    do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.
  12. Resque — Why?

  13. “Assume chaos” — Resque

  14. Forking » mitigates the risk of leaking memory » avoids

    the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs
  15. Sidekiq — Why?

  16. Sidekiq By @mperham, ca. 2012 » "What if one process

    could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)
  17. Sidekiq — How?

  18. Thread1 1 actually, pthread_create(3)

  19. Threads Pros: shared memory Cons: shared memory

  20. Race Conditions @wallet = 100 Thread.new { @wallet += 10

    } Thread.new { @wallet -= 10 }
  21. Race Conditions @wallet = 100 Thread.new do tmp = @wallet

    sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end
  22. Actors & Celluloid

  23. class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =

    amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100
  24. Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The

    Celluloid Ecosystem"
  25. Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The

    Celluloid Ecosystem"
  26. Sidekiq — How?

  27. Takeaways If you're: » memory-constrained - use Sidekiq » not

    thread-safe - use Resque » both - ¯\_()_/¯
  28. Takeaways Considerations: » How important is job isolation? » What's

    your bottleneck? RAM? I/O? » What about the GIL? » How important is support?
  29. Takeaways Don't be afraid to dive in to code »

    pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb
  30. Further Reading Crack open a good gem » Unicorn »

    resque-pool » Adhearson
  31. Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball

    - Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails
  32. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com