A tale of three web servers

A talk about three popular Ruby web servers at Amsterdam.rb. How and why are these servers different? Along the way we’ll learn some things about the options we have to let a Ruby program do multiple things at the same time.

Transcript

1. A Tale Of Three Web Servers
2. Why do these different web servers even exist?
3. All three use a similar way of receiving incoming requests and sending responses
4. All three use Zed Shaw’s HTTP parser
5. A web server has to do more than one thing at a time
6. Wikipedia Concurrency is a property of systems in which several computations are executing simultaneously, and potentially interacting with each other.
7. Three main ways of doing concurrency in Ruby • Multi-process (Unicorn) • Threading (Puma) • Event-driven (Thin) Disclaimer: For the sake of simplicity we will focus on the original strong point of each of these three servers, the story is a bit more complex in reality. There are other web servers out there too.
9. Multi-process
10. Multi-process • When you start Unicorn you start a master process • The master process does not handle requests, but controls one or more child processes that do • It starts these processes by forking itself
11. Fork is a Unix command that makes a copy of a process, with the exact state it has at the time of forking.
12. Code Console output @best_year_ever = 2014
 
 puts "#{Process.pid}: I'm the original process"
 
 if fork
 puts "#{Process.pid}: I’m the master"
 else
 puts "#{Process.pid}: I'm the child” @best_year_ever = 2015
 end
 
 puts "#{Process.pid}: The best year ever is #{@best_year_ever}" 351: I'm the original process 351: I’m the master 351: The best year ever is 2014 372: I'm the child 372: The best year ever is 2015
13. How does Unicorn do this?
14. def spawn_missing_workers
 worker_nr = -1
 until (worker_nr += 1) == @worker_processes
 worker = Worker.new(worker_nr)
 if pid = fork
 # Run in the master
 WORKERS[pid] = worker
 worker.atfork_parent
 else
 # Run in the child
 after_fork_internal
 # Start the loop that handles incoming requests
 worker_loop(worker)
 end
 end
 end
15. WORKERS.each_key do |pid|
 Process.kill(pid)
 end
16. socket = Socket.new
 loop do
 # Wait for incoming connection
 socket, addr = socket.accept
 # Process incoming request with some backend
 process_request(socket.gets)
 end
17. Memory
18. Copy on write • Memory is not copied on forking • It does get copied when it’s written too • Therefore code used by frameworks and such occupies memory only once • Introduced in Ruby 2.0 (used to be available in REE too)
20. Recap: Multi-process • Multi-process does concurrency by running separate worker processes that handle requests • If you expect your workers to break it’s easy to kill them without affecting other workers • Concurrency is limited by the number of processes • Every process uses the full amount of memory. Copy on write helps, a bit.
21. Threading
22. Threading • One process handles multiple requests by running multiple threads • Threads live in the same process and therefore share the same global state
23. You have to take the shared global state into account when using threaded code, let’s look at some examples.
24. Code Console output 5.times do |i|
 Thread.new do
 sleep rand(5)
 puts "I'm thread #{i}"
 end
 end
 
 sleep 10 I'm thread 1 I'm thread 3 I'm thread 4 I'm thread 0 I'm thread 2
25. Code Console output @best_year_ever = 2014
 
 5.times do |i|
 Thread.new do
 sleep rand(5)
 puts "I'm thread #{i} and the best year ever is #{@best_year_ever}"
 end
 end
 
 sleep 2
 @best_year_ever = 2015
 
 sleep 30 I'm thread 2 and the best year ever is 2014 I'm thread 4 and the best year ever is 2014 I'm thread 0 and the best year ever is 2014 I'm thread 1 and the best year ever is 2015 I'm thread 3 and the best year ever is 2015
26. Code Console output @total = 0
 
 100.times do |i|
 Thread.new do
 sleep rand(0.5)
 snapshot_of_total = @total
 sleep rand(0.5)
 @total = snapshot_of_total + 1
 end
 end
 
 sleep 5
 puts @total 6
27. Code Console output @total = 0
 @lock = Mutex.new
 
 100.times do |i|
 Thread.new do
 sleep rand(0.5)
 @lock.synchronize do
 snapshot_of_total = @total
 sleep rand(0.5)
 @total = snapshot_of_total + 1
 end
 end
 end
 
 sleep 60
 puts @total 100
28. Code Console output @total = 0
 
 100.times do |i|
 snapshot_of_total = @total
 sleep rand(0.5)
 @total = snapshot_of_total + 1
 end
 
 puts @total 100
29. Threads can work well to achieve concurrency, but it can be really hard to make them independent of each other.
30. Global Interpreter Lock (GIL) • Every time the interpreter runs a line of Ruby code it locks • IO operations are run outside of the GIL • If you run operations on hashes, for example, in multiple threads your program will still only utilize one CPU core • Rubinius and jRuby don’t have a GIL
31. How does Puma do this?
32. module Puma
 class ThreadPool
 def initialize(min, max, *extra, &block)
 @cond = ConditionVariable.new
 @mutex = Mutex.new
 @workers = []
 @mutex.synchronize do
 @min.times { spawn_thread }
 end 
 end
 end
 end
33. while true
 mutex.synchronize do
 while todo.empty?
 @waiting += 1
 @cond.wait mutex
 @waiting -= 1
 end
 
 work = @todo.pop
 end
 
 block.call(work, *extra)
 end

34. def <<(work)
 @mutex.synchronize do
 @todo << work
 
 if @waiting == 0 and @spawned < @max
 spawn_thread
 end
 
 @cond.signal
 end
 end
35. while @status == :run
 begin
 ios = IO.select sockets
 ios.first.each do |sock|
 if io = sock.accept_nonblock
 c = Client.new io, nil
 pool << c
 end
 end
 end
36. Recap: Threading • Server keeps a pool of worker threads • They wait for work to come in and process it outside of the server’s main lock • Concurrency is limited by size of thread pool • Worker threads use little memory compared to processes
37. Event driven
38. Event driven • One process handles multiple requests by running an event loop that schedules work • All code that gets run in this loop has to split itself up in the smallest feasible units of work
39. loop do
 events = select(operations_being_watched)
 
 events.each do |event|
 find_callback_for_event(event).call
 end
 end
40. EventMachine
41. Code Console output require 'eventmachine'
 
 EM.run do
 5.times do |i|
 EM.add_timer(rand(5)) do
 puts "I'm callback #{i}"
 end
 end
 end I'm callback 1 I'm callback 2 I'm callback 0 I'm callback 3 I'm callback 4
42. Code Console output require 'eventmachine'
 
 @best_year_ever = 2014
 
 EM.run do
 5.times do |i|
 EM.add_timer(rand(5)) do
 puts "I'm callback #{i} and” the best year ever is #{@best_year_ever}"
 end
 end
 
 EM.add_timer(2) do
 @best_year_ever = 2015
 end
 end I'm callback 1 and the best year ever is 2014 I'm callback 3 and the best year ever is 2014 I'm callback 0 and the best year ever is 2015 I'm callback 2 and the best year ever is 2015 I'm callback 4 and the best year ever is 2015
43. Code Console output require 'eventmachine'
 
 @total = 0
 
 EM.run do
 100.times do |i|
 EM.add_timer(rand(0.5)) do
 snapshot_of_total = @total
 EM.add_timer(rand(0.5)) do
 @total = snapshot_of_total + 1
 end
 end
 end
 
 EM.add_timer(5) do
 puts @total
 end
 end 4
44. We can’t fix this
45. You can only use code that works in an event- driven way, or you’ll end up with a program that’s not concurrent
46. How does Thin do this?
47. EventMachine.start_server(
 @host,
 @port,
 Connection
 ) @port,
 Connection
 )
48. EventMachine.defer(
 operation,
 callback
 )
49. module Thin
 module Connection
 def receive_data(data)
 process if @request.parse(data)
 end
 
 def process
 EventMachine.defer(
 method(:pre_process),
 method(:post_process)
 )
 end
 
 def pre_process @app.call(@request.env)
 end
 
 def post_process(result)
 @response.status, @response.headers, @response.body = *result
 @response.each do |chunk|
 send_data chunk
 end
 end
 end
 end @port,
 Connection
 )
50. Recap: Event-driven • Server runs a loop that schedules execution of operations and callbacks • Whenever an operation has to wait for something it stops, a callback gets called when the wait is over • Hardly any memory is used by the callbacks, they’re just Ruby blocks • Concurrency is an order of magnitude bigger than the other two models • All code running in the loop has to be event-driven
51. So which concurrency model should I use?
52. It depends
53. • For most apps threading makes sense, Ruby/Rails ecosystem seems to (slowly) be moving this way. • If you run highly concurrent apps with long-running streams event-driven allows you to scale • If you don’t have a high-traffic site or you expect your workers to break go for good old multi-process
54. Thank you
55. More information • http://tomayko.com/writings/unicorn-is-unix • http://www.jstorimer.com/products/working-with-unix-processes • http://ablogaboutcode.com/2012/02/06/the-ruby-global-interpreter-lock/ • http://jordanhollinger.com/2011/05/15/writing-an-ajax-long-polling-server-in-ruby-part-1 • http://dailyjs.com/2012/09/27/truth-about-event-loops/