Working with Unix processes

Slide 1

Slide 1 text

Working with Unix processes June 2012 - amsterdam.rb

Slide 2

Slide 2 text

Dax Huiberts Twitter: daxhuiberts Github: daxhuiberts

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Unix philosophy

Slide 5

Slide 5 text

Everything is a file

Slide 6

Slide 6 text

/etc/passwd

Slide 7

Slide 7 text

/dev/mouse

Slide 8

Slide 8 text

ps x | grep ruby

Slide 9

Slide 9 text

sockets

Slide 10

Slide 10 text

File descriptors

Slide 11

Slide 11 text

Most Unix process starts with 3 open file descriptors

Slide 12

Slide 12 text

stdin stdout stderr

Slide 13

Slide 13 text

$stdin $stdout $stderr

Slide 14

Slide 14 text

Forking

Slide 15

Slide 15 text

`ls -al`

Slide 16

Slide 16 text

Complete copy of memory and file descriptors

Slide 17

Slide 17 text

1x 100MB + fork = 2x 100MB

Slide 18

Slide 18 text

1x 8000MB + fork = 2x 8000MB ???

Slide 19

Slide 19 text

Copy-on-write memory

Slide 20

Slide 20 text

Kernel#fork

Slide 21

Slide 21 text

puts "parent process pid is #{Process.pid}" if fork puts "entered the if block from #{Process.pid}" else puts "entered the else block from #{Process.pid}" end

Slide 22

Slide 22 text

parent process pid is 5054 entered the if block from 5054 entered the else block from 5078

Slide 23

Slide 23 text

puts "parent process pid is #{Process.pid}" fork do puts "I'm the child: #{Process.pid}" end puts "I'm still the parent: #{Process.pid}"

Slide 24

Slide 24 text

parent process pid is 6167 I'm still the parent: 6167 I'm the child: 6191

Slide 25

Slide 25 text

Resque

Slide 26

Slide 26 text

On certain platforms, when a Resque worker reserves a job it immediately forks a child process. The child processes the job then exits. When the child has exited successfully, the worker reserves another job and repeats the process.

Slide 27

Slide 27 text

Why? Because Resque assumes chaos.

Slide 28

Slide 28 text

Why? Because Resque assumes chaos. Resque assumes your background workers will lock up, run too long, or have unwanted memory growth.

Slide 29

Slide 29 text

1x 8000MB + fork = 2x 8000MB ???

Slide 30

Slide 30 text

Thanks to Resque's parent / child architecture, jobs that use too much memory release that memory upon completion. No unwanted growth.

Slide 31

Slide 31 text

if @child = fork srand # Reseeding procline "Forked #{@child} at #{Time.now.to_i}" Process.wait(@child) else procline "Processing #{job.queue} since #{Time.now.to_i}" perform(job, &block) exit! unless @cant_fork end

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

$ ps -e -o pid,command | grep [r]esque 92099 resque: Forked 92102 at 1253142769 92102 resque: Processing file_serve since 1253142769

Slide 36

Slide 36 text

Forking

Slide 37

Slide 37 text

Complete copy of memory and file descriptors

Slide 38

Slide 38 text

Unicorn

Slide 39

Slide 39 text

Preforking

Slide 40

Slide 40 text

Uses 1 socket for all workers

Slide 41

Slide 41 text

require 'socket' # Open a socket. socket = TCPServer.open('0.0.0.0', 8080) # For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do loop do connection = socket.accept connection.puts 'Hello Readers!' connection.close end end end Process.waitall

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Manages workers through signals

Slide 44

Slide 44 text

# For keeping track of child process (worker) pids. wpids = [] # Create 5 workers. 5.times do |i| wpids << fork do trap(:INT) { exit } loop do # ... end end end # Forward any relevant signals to the child processes. trap(:INT) do wpids.each { |wpid| Process.kill(:INT, wpid) } exit end Process.waitall