Queue and SizedQueue: hidden gems in the Ruby standard library

Slides for a lightning talk showing how to implement thread pools using this two not-very-known classes in the Ruby standard library. RUG::B (Ruby Users Group Berlin) September 2014.

Transcript

1. Sergio Gil @porras iamserg.io
2. Hello, Humans! Sergio Gil @porras iamserg.io
3. list.each do |item| do_something(item) end
4. Lukas Rieder @Overbryd PStore
5. Hidden gems* in the Ruby Standard Library *They're not actually gems, that's the joke ;) Today: Queue and SizedQueue
6. list.each do |item| do_something(item) end
7. list.each do |item| do_something_slow(item) end
8. list.each do |item| http.post(item) end
10. Parallel Map
11. list.map do |item| Thread.new do do_something(item) end end.each(&:join)
12. huge_list.map do |item| Thread.new do do_something(item) end end.each(&:join)
13. huge_list = File.open('huge.txt') # * * This works as is because File (and all IO's) include Enumerable
14. huge_list = TCPSocket.new( 'somewhere.com', 1234) # * * This works as is because TCPSocket (and all IO's) include Enumerable
15. Amdahl's Law
18. Thread Pools
19. Queue
20. require 'thread' num_workers = 20 queue = Queue.new threads = num_workers.times.map do Thread.new do until (item = queue.pop) == :END do_something(item) end end end list.each { |item| queue << item } num_workers.times { queue << :END } threads.each(&:join)
21. SizedQueue
22. require 'thread' num_workers = 20 queue = SizedQueue.new(num_workers * 2) threads = num_workers.times.map do Thread.new do until (item = queue.pop) == :END do_something(item) end end end list.each { |item| queue << item } num_workers.times { queue << :END } threads.each(&:join)
23. Bonus Track: Refactoring
24. require 'thread' num_workers = 20 queue = SizedQueue.new(num_workers * 2) threads = num_workers.times.map do Thread.new do until (item = queue.pop) == :END do_something(item) end end end list.each { |item| queue << item } num_workers.times { queue << :END } threads.each(&:join)
25. require 'thread' num_workers = 20 queue = SizedQueue.new(num_workers * 2) threads = num_workers.times.map do Thread.new do until (item = queue.pop) == :END do_something(item) end end end list.each { |item| queue << item } num_workers.times { queue << :END } threads.each(&:join) U GLY
26. class WorkersPool def initialize(num_workers) ... end def process(enumerable, &block) ... end end
27. workers = WorkersPool.new(20) workers.process(huge_list) do |item| do_something(item) end
28. Dankeschön! Sergio Gil @porras iamserg.io