Processes & Threads - Resque vs. Sidekiq

Transcript

1. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com

2. Read, read, read. Read everything – trash, classics, good and

   bad, and see how they do it. — William Faulkner, on writing

3. Background Workers

4. Resque Developed at Github, ca. 2009 » Let Redis handle

   the hard queue problems » Let Resque focus on reliability, visibility, stats and responsiveness

5. Resque — How?

6. fork(2)

7. Forking def log(str); puts "#{Process.pid}: #{str}"; end a = 1

   if pid = fork log "Waiting on child" Process.wait pid log "Done with fork. a=#{a}" else log "Child doing work" sleep 1 a += 1 log "a=#{a}" exit end

8. Forking Produces (modulo different pids) 57388: Waiting on child 57416:

   Child doing work 57416: a=2 57388: Done with fork. a=1

9. Forking The child process » springs into existence » performs

   its job » dies

10. Resque — How?

11. Resque, distilled start # Initialize, register signal handlers, &c. loop

    do if job = redis_job_queue.pop child = fork do job.process end Process.wait child else sleep polling_frequency end end shutdown # Unregister signal handlers, &c.

12. Resque — Why?

13. “Assume chaos” — Resque

14. Forking » mitigates the risk of leaking memory » avoids

    the cost of booting Rails for each job » allows using signals to e.g. shut down hung jobs

15. Sidekiq — Why?

16. Sidekiq By @mperham, ca. 2012 » "What if one process

    could do the work of 20 Resque processes?" » Performance, simplicity, support » Batteries included (failure retry, scheduled jobs, middleware)

17. Sidekiq — How?

18. Thread1 1 actually, pthread_create(3)

19. Threads Pros: shared memory Cons: shared memory

20. Race Conditions @wallet = 100 Thread.new { @wallet += 10

    } Thread.new { @wallet -= 10 }

21. Race Conditions @wallet = 100 Thread.new do tmp = @wallet

    sleep rand(0..5) @wallet = tmp + 10 end Thread.new do sleep rand(0..5) tmp = @wallet @wallet = tmp - 10 end

22. Actors & Celluloid

23. class Wallet include Celluloid attr_reader :amount def initialize(amt); @amount =

    amt; end def adjust(Δ) ; @amount += Δ ; end end @wallet = Wallet.new 100 [10, -10].each { |Δ| wallet.async.adjust(Δ) } @wallet.amount # => 100

24. Celluloid - Async @wallet.async.adjust(Δ) Diagram adapted from T. Arcieri's "The

    Celluloid Ecosystem"

25. Celluloid - Sync @wallet.amount() Diagram adapted from T. Arcieri's "The

    Celluloid Ecosystem"

26. Sidekiq — How?

27. Takeaways If you're: » memory-constrained - use Sidekiq » not

    thread-safe - use Resque » both - ¯\_(
    )_/¯

28. Takeaways Considerations: » How important is job isolation? » What's

    your bottleneck? RAM? I/O? » What about the GIL? » How important is support?

29. Takeaways Don't be afraid to dive in to code »

    pry - ls, @, wtf?!?, &c. » pry-stack_explorer, pry-byebug, pry-rescue, &c. » bundle show, bundle open » ack --rb

30. Further Reading Crack open a good gem » Unicorn »

    resque-pool » Adhearson

31. Further Reading » These slides - jamesdabbs/railsconf-2015 » T. Ball

    - Unicorn Unix Magic Tricks » J. Storimer - Understanding the GIL » J. Lane - Writing thread-safe Rails

32. Processes & Threads Resque vs Sidekiq @jamesdabbs | james@theironyard.com