Working with Unix processes

A short talk I gave about Working with Unix processes based on the ebook by Jesse Storimer at the "Amsterdam.rb Reboot" on June 19th of 2012 at the 80beans office.

Transcript

1. Working with Unix processes June 2012 - amsterdam.rb
2. Dax Huiberts Twitter: daxhuiberts Github: daxhuiberts
4. Unix philosophy
5. Everything is a file
6. /etc/passwd
7. /dev/mouse
8. ps x | grep ruby
9. sockets
10. File descriptors
11. Most Unix process starts with 3 open file descriptors
12. stdin stdout stderr
13. $stdin $stdout $stderr
14. Forking
15. `ls -al`
16. Complete copy of memory and file descriptors
17. 1x 100MB + fork = 2x 100MB
18. 1x 8000MB + fork = 2x 8000MB ???
19. Copy-on-write memory
20. Kernel#fork
21. puts "parent process pid is #{Process.pid}" if fork puts "entered the if block from #{Process.pid}" else puts "entered the else block from #{Process.pid}" end
22. parent process pid is 5054 entered the if block from 5054 entered the else block from 5078
23. puts "parent process pid is #{Process.pid}" fork do puts "I'm the child: #{Process.pid}" end puts "I'm still the parent: #{Process.pid}"
24. parent process pid is 6167 I'm still the parent: 6167 I'm the child: 6191
25. Resque
26. On certain platforms, when a Resque worker reserves a job it immediately forks a child process. The child processes the job then exits. When the child has exited successfully, the worker reserves another job and repeats the process.
27. Why? Because Resque assumes chaos.
28. Why? Because Resque assumes chaos. Resque assumes your background workers will lock up, run too long, or have unwanted memory growth.
29. 1x 8000MB + fork = 2x 8000MB ???
30. Thanks to Resque's parent / child architecture, jobs that use too much memory release that memory upon completion. No unwanted growth.
31. if @child = fork srand # Reseeding procline "Forked #{@child} at #{Time.now.to_i}" Process.wait(@child) else procline "Processing #{job.queue} since #{Time.now.to_i}" perform(job, &block) exit! unless @cant_fork end
32. if @child = fork srand # Reseeding procline "Forked #{@child} at #{Time.now.to_i}" Process.wait(@child) else procline "Processing #{job.queue} since #{Time.now.to_i}" perform(job, &block) exit! unless @cant_fork end
33. if @child = fork srand # Reseeding procline "Forked #{@child} at #{Time.now.to_i}" Process.wait(@child) else procline "Processing #{job.queue} since #{Time.now.to_i}" perform(job, &block) exit! unless @cant_fork end
34. if @child = fork srand # Reseeding procline "Forked #{@child} at #{Time.now.to_i}" Process.wait(@child) else procline "Processing #{job.queue} since #{Time.now.to_i}" perform(job, &block) exit! unless @cant_fork end
35. $ ps -e -o pid,command | grep [r]esque 92099 resque: Forked 92102 at 1253142769 92102 resque: Processing file_serve since 1253142769
36. Forking
37. Complete copy of memory and file descriptors
38. Unicorn
39. Preforking
40. Uses 1 socket for all workers
41. require 'socket' # Open a socket. socket = TCPServer.open('0.0.0.0', 8080) # For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do loop do connection = socket.accept connection.puts 'Hello Readers!' connection.close end end end Process.waitall
42. require 'socket' # Open a socket. socket = TCPServer.open('0.0.0.0', 8080) # For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do loop do connection = socket.accept connection.puts 'Hello Readers!' connection.close end end end Process.waitall
43. Manages workers through signals
44. # For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do trap(:INT) { exit } loop do # ... end end end # Forward any relevant signals to the child processes. trap(:INT) do wpids.each { |wpid| Process.kill(:INT, wpid) } exit end Process.waitall
45. # For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do trap(:INT) { exit } loop do # ... end end end # Forward any relevant signals to the child processes. trap(:INT) do wpids.each { |wpid| Process.kill(:INT, wpid) } exit end Process.waitall
47. Thank you!
48. Questions?