Ruby loves Unix: Applying beautiful Unix idioms to build a Ruby prefork server

The Unix programming model has existed for more than 40 years now. Unix programming techniques and principles have transcended programming languages and frameworks du jour. Also, Ruby was built for unix hacking. Implementing unix system calls in its standard library was the best perlism Ruby adopted.

Learning Unix programming and its idioms will make you a better Ruby programmer. Your ruby programs will be friendly citizens on your *nix server making them easy to administer. You will be able to daemonize your code at will. Your concurrent ruby code will be easy to grok and debug. You will have deeper understanding into the inner working of popular tools like the Resque job scheduler and
the Unicorn web server.

This talk is divided into two parts. The first part will be a dive into the basic building block of the unix programming model: processes. We will learn the application of the myriad unix system calls such as kill(2), fork(2), execve(2), pipe(2) and friends to our processes and understand various important idioms.

The next part will involve using the knowledge we've gained to build a Ruby preforking server. Our daemon will follow the classical master/worker prefork model wherein the master will fork and monitor its children and use the operating system to load balance work amongst available children. Our daemon will demonstrate classical unix idioms like daemonizing via double fork(2), administration via signals, IPC via signals and pipes,
concurrency via fork(2) and many more.

Transcript

1. <3 Unix Applying beautiful Unix idioms to build a Ruby prefork server
2. Sahil Muthoo @sahilmuthoo ThoughtWorks Studios
3. snap-ci.com The missing link between GitHub and Heroku
4. Four decades of Unix
5. Transcends languages and frameworks
6. Solving problems for four decades
7. Unicorn Resque
8. Unicorn Resque UNIX Live free or die
9. Man pages Tell your kill(2) from your kill(1)
10. System calls
11. Ruby was built for Unix hacking
12. System calls Unix Ruby fork(2) Process.fork execve(2) Kernel.exec pipe(2) IO.pipe select(2) IO.select kill(2) Process.kill
13. Processes The atoms of Unix
14. Processes have ids % irb >> Process.pid => 705 Process #705
15. Processes have parents % irb >> Process.pid => 705 >> Process.ppid => 99930 Process #705 Parent process #99930
16. Process lineage % pstree -s irb -+= 00001 root /sbin/launchd \-+= 00136 sahilm /sbin/launchd \-+= 08735 sahilm /Applications/iTerm.app \-+= 08739 sahilm -/bin/bash \--= 08830 sahilm irb
17. Processes have friendly names % irb >> $PROGRAM_NAME => "irb" >> $PROGRAM_NAME="Ponies!" => "Ponies!" Process #705 Parent process #99930 Name irb Ponies!
18. Process names are useful % bin/kaanta [kaanta master (PID: 4402)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
19. Process names are useful % bin/kaanta [kaanta master (PID: 4402)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
20. Process names are useful % bin/kaanta [kaanta master (PID: 4402)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
21. Processes have resources % irb >> STDIN.fileno => 0 >> STDOUT.fileno => 1 >> STDERR.fileno => 2 Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2
22. file == resource == file % irb >> socket = TCPServer.open => #<TCPServer:fd 3> >> socket.fileno => 3 Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
23. Processes can fork % irb >> fork { puts "Oh hai!" } => 5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
24. % irb >> fork { puts "Oh hai!" } => 5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3 => Oh hai! Process #5306 Parent process #705 Name Ponies! Resources fd0-2 fd3
25. % irb >> fork { puts "Oh hai!" } => 5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3 => Oh hai! Process #5306 Parent process #705 Name Ponies! Resources fd0-2 fd3
26. fork returns twice if fork puts "forked" else puts "didn't fork" end => forked => didn't fork Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
27. fork returns twice puts "Parent: #{Process.pid}" if fork puts "#{Process.pid} forked" else puts "#{Process.pid} didn't fork" end => Parent 705 => 705 forked => 5306 didn't fork Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
28. fork(2) is copy on write friendly
29. homework = Tempfile.new('') pid = fork do homework.write "5 * 5 = 25" end if File.zero?(homework) puts "Y U NO HOMEWORK!" else puts "Have a cookie" end
30. Y U NO HOMEWORK!
31. homework = Tempfile.new('') pid = fork do homework.write "5 * 5 = 25" end if File.zero?(homework) puts "Y U NO HOMEWORK!" else puts "Have a cookie" end Process.waitpid(pid)
32. Wait for your children waitpid(2) and friends
33. Processes have exit statuses 3.times do fork {exit([0,1].sample)} end 3.times do pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
34. wait2 returns status 3.times do fork {exit([0,1].sample)} end 3.times do pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
35. Nonzero exit statuses indicate errors 3.times do fork {exit([0,1].sample)} end 3.times do pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
36. % bin/kaanta [kaanta master (PID: 13685)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 13685)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 13712)] INFO -- up [kaanta worker 1 (PID: 13713)] INFO -- up [kaanta worker 2 (PID: 13714)] INFO -- up [kaanta master (PID: 13685)] INFO -- reaped worker 0 (PID:13712) status: 0 [kaanta worker 0 (PID: 13725)] INFO -- up % kill 13712
37. % bin/kaanta [kaanta master (PID: 13685)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 13685)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 13712)] INFO -- up [kaanta worker 1 (PID: 13713)] INFO -- up [kaanta worker 2 (PID: 13714)] INFO -- up [kaanta master (PID: 13685)] INFO -- reaped worker 0 (PID:13712) status: 0 [kaanta worker 0 (PID: 13725)] INFO -- up % kill 13712
38. Processes can receive signals % irb >> Process.pid => 14576 >> Terminated: 15 % % kill -SIGTERM 14576
39. Processes can receive signals % irb >> Process.pid => 14576 >> Terminated: 15 % % kill -15 14576
40. kill(2) isn't only for killing puts Process.pid i=0 loop do print i+=1 sleep 1 end % ruby kill.rb 15068 1234... % kill -SIGSTOP 15068 [2]+ Stopped ruby kill.rb
41. kill(2) isn't only for killing puts Process.pid i=0 loop do print i+=1 sleep 1 end % kill -SIGCONT 15068 % 5678...
42. man 7 signal
43. Processes can trap signals % irb >> Process.pid => 15369 >> trap(:TERM){puts "nope"} => "DEFAULT" >> nope % kill -SIGTERM 15369
44. Refuse to die % irb >> Process.pid => 15369 >> trap(:TERM){puts "nope"} => "DEFAULT" >> nope % kill -SIGTERM 15369
45. A more useful example pid = fork do loop do print "." sleep 1 end end trap(:TERM) do Process.kill(:TERM, pid) end Process.waitpid(pid) % kill -SIGTERM 15369
46. Children exit with a SIGCHLD trap(:CHLD) do exit end fork do sleep 1 end
47. Signals are asynchronous trap(:CHLD) do exit end fork do sleep 1 end # tight loop loop {}
48. exited = 0 trap(:CHLD) do exited +=1 exit if exited == 3 end 3.times do fork do sleep 1 end end # tight loop loop {}
49. May never exit
50. The kernel maintains a signal queue
51. Signal delivery is unreliable
52. def handler(exited) loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1 exit if exited == 3 end exited rescue Errno::ECHILD end exited = 0 trap(:CHLD) { exited = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {}
53. def handler(exited) exited rescue Errno::ECHILD end exited = 0 trap(:CHLD) { exited = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {} loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1 exit if exited == 3 end
54. def handler(exited) exited end exited = 0 trap(:CHLD) { exited = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {} loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1 exit if exited == 3 end rescue Errno::ECHILD
55. Processes can communicate reader, writer = IO.pipe fork do writer.close puts reader.gets reader.close end reader.close writer.write("Wake up!") writer.close Process.wait
56. Pipes are one-way reader, writer = IO.pipe fork do writer.close puts reader.gets reader.close end reader.close writer.write("Wake up!") writer.close Process.wait
57. select(2) tells you when stuff is ready reader, writer = IO.pipe fork do writer.puts("Do this") sleep 2 writer.puts("Do that") end loop do ret = IO.select([reader], nil, nil, 1) if ret puts reader.gets else puts "no work :(" end end Do this no work :( Do that no work :(
58. IO.select(read_array [, write_array [, error_array [, timeout]]]) -> array or nil
59. Preforking servers
60. Master Worker Worker fork(2) fork(2) select(2) on shared socket
61. Unicorn, Passenger
62. And Kaanta (Hindi for fork) github.com/sahilm/kaanta
63. One master forks many children
64. Concurrency via fork(2)
65. Faster to boot and more memory efficient
66. def start $PROGRAM_NAME="kaanta master" @master_pid = Process.pid @socket = TCPServer.open(Config.host, Config.port) spawn_workers end def spawn_workers worker_number = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next worker = Kaanta::Worker.new(@master_pid, @socket, tempfile, worker_number,logger) pid = fork { worker.start } @workers[pid] = worker end end
67. def start $PROGRAM_NAME="kaanta master" @master_pid = Process.pid @socket = TCPServer.open(Config.host, Config.port) spawn_workers end def spawn_workers worker_number = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next worker = Kaanta::Worker.new(@master_pid, @socket, tempfile, worker_number,logger) pid = fork { worker.start } @workers[pid] = worker end end
68. One socket shared amongst all workers
69. Kernel load balances connections
70. while alive && @master_pid == Process.ppid do if ret begin client = @socket.accept_nonblock rescue Errno::EAGAIN end end ret = begin IO.select([@socket], nil, nil, Config.timeout / 2) || next rescue Errno::EBADF end end
71. Easier sysadmining
72. Managed via signals to master
73. No zoo of processes to manage
74. Works well with simple tools like kill
75. loop do reap_workers case (mode = @sig_queue.shift) when nil kill_runaway_workers spawn_workers when 'QUIT', 'TERM', 'INT' # we don't handle gracefully stopping workers # to demonstrate that workers go down quickly # after the master quits by tracking their # Process.ppid. break when 'TTIN' Config.workers += 1 when 'TTOU' unless Config.workers <= 0 Config.workers -= 1 kill_worker('QUIT', @workers.keys.max) end end
76. % bin/kaanta [kaanta master (PID: 729)] INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 729)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 756)] INFO -- up [kaanta worker 1 (PID: 757)] INFO -- up [kaanta worker 2 (PID: 758)] INFO -- up [kaanta worker 3 (PID: 862)] INFO -- up % kill -TTIN 729 13712
77. Moar idioms!
78. Let's turn the unix up to 11
79. How do you kill hung workers?
80. Hand each worker an unlinked temporary file def spawn_workers worker_number = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next tempfile = Tempfile.new('') tempfile.unlink tempfile.sync = true worker = Kaanta::Worker.new(tempfile) pid = fork { init_worker(worker) } @workers[pid] = worker end end
81. Workers must chmod that tempfile periodically while alive && @master_pid == Process.ppid do tempfile.chmod(i += 1) if ret begin client = @socket.accept_nonblock command = client.gets logger.info("Executing: #{command}") client.write `#{command}` client.flush client.close rescue Errno::EAGAIN end end tempfile.chmod(i += 1) ret = begin IO.select([@socket], nil, nil, Config.timeout / 2)||next rescue Errno::EBADF end end
82. Kill workers that failed to chmod in time def kill_runaway_workers now = Time.now @workers.each_pair do |pid, worker| (now - worker.tempfile.ctime) <= Config.timeout && next kill_worker('KILL', pid) end end
83. Spawn replacements in your master loop loop do reap_workers case (mode = @sig_queue.shift) when nil kill_runaway_workers spawn_workers when 'QUIT', 'TERM', 'INT' break when 'TTIN' Config.workers += 1 when 'TTOU' unless Config.workers <= 0 Config.workers -= 1 kill_worker('QUIT', @workers.keys.max) end end end
84. fchmod(2) heartbeat
85. How do you reap dead workers?
86. def reap_workers loop do pid, status = Process.waitpid2(-1, Process::WNOHANG)||break reap_worker(pid, status) end rescue Errno::ECHILD end def reap_worker(pid, status) worker = @workers.delete(pid) worker.tempfile.close rescue nil logger.info "reaped worker #{worker.number} " \ "(PID:#{pid}) " \ "status: #{status.exitstatus}" end
87. waitpid(2) with WNOHANG
88. How does a worker detect its parent's death?
89. while alive && @master_pid == Process.ppid do tempfile.chmod(i += 1) if ret begin client = @socket.accept_nonblock command = client.gets logger.info("Executing: #{command}") client.write `#{command}` client.flush client.close rescue Errno::EAGAIN end end tempfile.chmod(i += 1) ret = begin IO.select([@socket], nil, nil, Config.timeout / 2) || next rescue Errno::EBADF end end
90. Time out IO.select in timeout / 2 seconds
91. Make sure the master pid is correct
92. More tricks in kaanta's source
93. Daemonizing a process
94. Deferred signal handling
95. Fin
96. Thank you