A tale of three chat rooms

How three types of concurrency handle an audience full of excited developers. At Railsconf 2016. Code example are here: https://github.com/thijsc/three-chat-servers

Transcript

1. A TALE OF THREE CHAT ROOMS git clone git@github.com:thijsc/three-chat-servers.git To follow along and join the demo, clone the examples:
2. A TALE OF THREE CHAT ROOMS How three types of concurrency handle an audience full of excited developers. A tale of three chat rooms
3. A TALE OF THREE CHAT ROOMS A developer (me) was working on a monitoring tool called AppSignal. SOMEWHERE… IN A COUNTRY FAR FAR AWAY…
4. A TALE OF THREE CHAT ROOMS Multi-process Multi-threading Event-loop THREE TYPES OF CONCURRENCY
5. A TALE OF THREE CHAT ROOMS Multi-process Multi-threading Event-loop THREE TYPES OF CONCURRENCY Unicorn Puma Thin
6. A TALE OF THREE CHAT ROOMS Let’s try to build a (very) minimalistic Slack
7. A TALE OF THREE CHAT ROOMS code Our chat client 1 require 'socket' 2 client = TCPSocket.open(ARGV[0], 2000) 3 4 Thread.new do 5 while line = client.gets 6 puts line.chop 7 end 8 end 9 10 while input = STDIN.gets.chomp 11 client.puts input 12 end
8. A TALE OF THREE CHAT ROOMS • A master process starts multiple worker processes • The worker process handles the actual requests • The master manages the workers Multi-process
9. A TALE OF THREE CHAT ROOMS root 132 unicorn master root 133 unicorn worker_1 root 134 unicorn worker_2 root 135 unicorn worker_3 root 136 unicorn worker_4 root 137 unicorn worker_5 root 82 crypto root 81 ecrptfs-krthrea
10. A TALE OF THREE CHAT ROOMS code Start a TCP server 1 require 'socket' 3 4 puts "Starting server on port 2000" 5 6 server = TCPServer.open(2000)
11. A TALE OF THREE CHAT ROOMS code Communicate between processes 8 client_writers = [] 9 master_reader, master_writer = IO.pipe 10 11 write_incoming_messages_to_child_processes
12. A TALE OF THREE CHAT ROOMS code Start a worker process 15 loop do 16 while socket = server.accept 20 client_reader, client_writer = IO.pipe 23 client_writers.push(client_writer) .. 26 fork do 39 end 41 end 42 end
13. A TALE OF THREE CHAT ROOMS code What’s running in the worker process 27 nickname = read_line_from(socket) 29 client.puts 'Connected to chat server' 30 31 write_incoming_messages_to_client 32 33 # Read incoming messages from the client. 34 while incoming = read_line_from(socket) 35 master_writer.puts "#{nickname}: #{incoming}" 36 end
14. A TALE OF THREE CHAT ROOMS Multi-process pros :) • Write code without thinking about concurrency at all! • Each worker can crash without damaging the rest of the system
15. A TALE OF THREE CHAT ROOMS Multi-process cons :( • Each process loads the full codebase in memory • This makes it memory-intensive • Hence, it does not scale to large amounts of concurrent connections
16. A TALE OF THREE CHAT ROOMS • Allows one process to handle multiple requests at the same time • It does so by running multiple threads within a single process Multi-threading
17. A TALE OF THREE CHAT ROOMS code Start a TCP server (exactly the same) 1 require 'socket' 3 4 puts "Starting server on port 2000" 5 6 server = TCPServer.open(2000)
18. A TALE OF THREE CHAT ROOMS code Store incoming messages 8 mutex = Mutex.new 9 messages = []
19. A TALE OF THREE CHAT ROOMS code Run a thread per connection 11 loop do 14 Thread.new(server.accept) do |socket| 15 nickname = read_line_from(socket) 17 client.puts 'Connected to chat server' 19 send_incoming_messages_thread 38 read_incoming_messages 47 end 49 end
20. A TALE OF THREE CHAT ROOMS code Store a message 38 while incoming = read_line_from(socket) 39 mutex.synchronize do 40 messages.push( 41 :time => Time.now, 42 :nickname => nickname, 43 :text => incoming 44 ) 45 end 46 end
21. A TALE OF THREE CHAT ROOMS code Send messages to the client 22 loop do 23 messages_to_send = mutex.synchronize do 24 get_messages_to_send().tap do 25 sent_until = Time.now 26 end 27 end 28 messages_to_send.each do |message| 29 socket.puts message[:text] 30 end 31 sleep 0.2 32 end
22. A TALE OF THREE CHAT ROOMS root 132 puma | ——— thread #1 | ——— thread #2 | ——— thread #3 | ——— thread #4 root 82 crypto root 81 ecrptfs-krthrea
23. A TALE OF THREE CHAT ROOMS • A lock around the execution of all Ruby code • Even though our threads appear to run in parallel, only one thread is active at a time • IO operates outside of the GIL Global Interpreter Lock
24. A TALE OF THREE CHAT ROOMS If you use multiple threads you have to be careful to write all code that manipulates shared data in a thread safe way Thread safety
25. A TALE OF THREE CHAT ROOMS Multi-threading pros :) • Uses less memory than multi-process • You can share data easily because everything runs in the same process
26. A TALE OF THREE CHAT ROOMS Multi-threading cons :( • You have to make sure your code is thread safe • If a thread causes a a crash, it could take down your process • The GIL locks all operations except I/O
27. A TALE OF THREE CHAT ROOMS • For large number of concurrent I/O operations • Not actually executed at the same time • It is an efficient way to handle concurrent users Event-loop
28. A TALE OF THREE CHAT ROOMS OS [network / disk] Event queue Oldest event Event Event Event Event Event Newest event Storage in memory Event loop Add IO ready event Add event Register interest The mental model…
29. A TALE OF THREE CHAT ROOMS • We’ll use IO.select instead of registering interest with the OS • The client is implemented as a Fiber • The event loop will inform the fiber when its connection is ready Our example: Poor man’s event loop
30. A TALE OF THREE CHAT ROOMS code Fibers 1 fiber = Fiber.new do 2 loop do 3 puts Fiber.yield 4 end 5 end 6 7 fiber.resume 1 8 fiber.resume 2 9 fiber.resume 3 10 fiber.resume 4
31. A TALE OF THREE CHAT ROOMS code You’ve seen this before 1 require 'socket' 3 4 puts "Starting server on port 2000" 5 6 server = TCPServer.open(2000)
32. A TALE OF THREE CHAT ROOMS code Clients & messages 1 clients = {} 2 messages = []
33. A TALE OF THREE CHAT ROOMS code The client representation 12 Fiber.new do 13 last_write = Time.now 15 loop do 16 state = Fiber.yield 17 18 if state == :readable 19 # Read a message from the socket 37 elsif state == :writable 38 # Write messages to the socket 44 end 45 end 46 end
34. A TALE OF THREE CHAT ROOMS code Read a message 18 if state == :readable 20 incoming = read_line_from(socket) 32 $messages.push( 33 :time => Time.now, 34 :nickname => nickname, 35 :text => incoming 36 )
35. A TALE OF THREE CHAT ROOMS code Write a message 37 elsif state == :writable 38 # Write messages to the socket 39 get_messages_to_send().each do |message| 40 socket.puts message 41 end 42 43 last_write = Time.now 44 end
36. A TALE OF THREE CHAT ROOMS code Structure of the event loop 12 loop do 13 accept_incoming_connections 14 15 get_ready_connections 16 17 read_from_readable_connections 18 19 write_to_writable_connections 20 end
37. A TALE OF THREE CHAT ROOMS code Accept incoming connections 15 begin 16 socket = server.accept_nonblock 17 nickname = socket.gets.chomp 18 client = { 19 :last_write => Time.now, 20 :nickname => nickname, 21 :ip => socket.addr.last 22 } 23 clients[socket] = client 25 rescue IO::WaitReadable, Errno::EINTR 26 # No new incoming connections at the moment 27 end
38. A TALE OF THREE CHAT ROOMS code See which clients are ready 30 readable, writable = IO.select( 31 clients.keys, 32 clients.keys, 33 clients.keys, 34 0.01 35 )
39. A TALE OF THREE CHAT ROOMS code Read from readable connections 38 if readable 39 readable.each do |ready_socket| 41 incoming = read_line_from(ready_socket) 44 client = clients[ready_socket] 55 messages.push( 56 :time => Time.now, 57 :nickname => client[:nickname], 58 :text => incoming 59 ) 60 end 61 end
40. A TALE OF THREE CHAT ROOMS code Write to writable connections 64 if writable 65 writable.each do |ready_socket| 67 client = clients[ready_socket] 69 70 get_messages_to_send.each do |message| 71 ready_socket.puts message[:text] 72 end 73 74 client[:last_write] = Time.now 75 end 76 end
41. A TALE OF THREE CHAT ROOMS Event-loop pros :) • Almost no memory overhead per connection • Scales to a huge number of parallel connections
42. A TALE OF THREE CHAT ROOMS Event-loop cons :( • It can be hard to debug if your system gets more complex (callbacks) • The work done per tick of the loop must be small and predictable to avoid high latency
43. A TALE OF THREE CHAT ROOMS Multi-process Multi-threading Event-loop WHICH ONE TO USE?
44. A TALE OF THREE CHAT ROOMS Who fancies a chat? How many developers will it take to break three different ways I built a chat? Let’s see…..
45. A TALE OF THREE CHAT ROOMS git clone git@github.com:thijsc/three-chat-servers.git Clone from here: Then run: ruby src/client.rb <ip> YourNick
46. A TALE OF THREE CHAT ROOMS Thijs Cadier Co-founder AppSignal, Email Twitter Github thijs@appsignal.com @thijsc thijsc We’re hiring! If you want to help other developers build better software, contact us! Use the coupon code concurrency and get $50 credit! Thank you!