Hello! Thorsten Ball mrnugget / @thorstenball Software Developer

No content

@thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

@thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

@thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

@thorstenball Unicorn Unix Magic Tricks Master-Worker Achitecture $ pstree | grep unicorn \-+= 27185 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27210 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27211 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27212 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27213 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27214 mrnugget unicorn worker[4] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27215 mrnugget unicorn worker[5] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27216 mrnugget unicorn worker[6] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27217 mrnugget unicorn worker[7] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27218 mrnugget unicorn worker[8] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27219 mrnugget unicorn worker[9] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27220 mrnugget unicorn worker[10] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27221 mrnugget unicorn worker[11] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27222 mrnugget unicorn worker[12] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27223 mrnugget unicorn worker[13] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27224 mrnugget unicorn worker[14] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27225 mrnugget unicorn worker[15] -c simple_unicorn_config.rb -l0.0.0.0:8080

@thorstenball Unicorn Unix Magic Tricks Hot Reload $ pstree | grep unicorn \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080

@thorstenball Unicorn Unix Magic Tricks Signals TERM QUIT HUP USR1 USR2 WINCH TTIN TTOU

@thorstenball Unicorn Unix Magic Tricks Pre-Loading

@thorstenball Unicorn Unix Magic Tricks ./PHILOSOPHY I MEAN, COME ON!

@thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

@thorstenball Unicorn Unix Magic Tricks How does Unicorn work?

@thorstenball Unicorn Unix Magic Tricks fork, fork, fork

@thorstenball Unicorn Unix Magic Tricks fork(2) • system call (man 2 fork) • splits processes in two (“a fork in the road”) • fork(2) splits a process into parent and child • children inherit a lot from their parent processes (data, stack, heap, working directory, …)

@thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) # fork.rb ! child_pid = fork do puts "[child] child_pid: #{child_pid}" puts "[child] Process ID: #{Process.pid}" puts "[child] Parent Process ID: #{Process.ppid}" end Process.wait(child_pid) puts "[parent] child_pid: #{child_pid}" puts "[parent] Process ID: #{Process.pid}"

@thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) # fork.rb ! child_pid = fork do puts "[child] child_pid: #{child_pid}" puts "[child] Process ID: #{Process.pid}" puts "[child] Parent Process ID: #{Process.ppid}" end Process.wait(child_pid) puts "[parent] child_pid: #{child_pid}" puts "[parent] Process ID: #{Process.pid}" $ ruby fork.rb [child] child_pid: [child] Process ID: 29715 [child] Parent Process ID: 29695 [parent] child_pid: 29715 [parent] Process ID: 29695

@thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) $ pstree | grep fork | \-+= 29695 mrnugget ruby fork.rb | \--- 29715 mrnugget ruby fork.rb $ ruby fork.rb [child] child_pid: [child] Process ID: 29715 [child] Parent Process ID: 29695 [parent] child_pid: 29715 [parent] Process ID: 29695

@thorstenball Unicorn Unix Magic Tricks How does Unicorn use fork(2)? # spawn_missing_workers worker_nr = -1 until (worker_nr += 1) == @worker_processes WORKERS.value?(worker_nr) and next worker = Worker.new(worker_nr) before_fork.call(self, worker) if pid = fork WORKERS[pid] = worker worker.atfork_parent else after_fork_internal worker_loop(worker) exit end end

@thorstenball Unicorn Unix Magic Tricks Pipes! $ grep ‘wat’ journal.txt | wc -l 84

@thorstenball Unicorn Unix Magic Tricks pipe(2) • system call (man 2 pipe) • We can use them outside the shell! • Pipes are two file descriptors • One read end, one write end • Great for communication between processes • Pipes are inherited when forking

@thorstenball Unicorn Unix Magic Tricks Ruby and pipe(2) # pipe.rb read_end, write_end = IO.pipe fork do read_end.close write_end.write('Hello from your child!') write_end.close end write_end.close Process.wait message = read_end.read read_end.close puts "Received from child: '#{message}'"

@thorstenball Unicorn Unix Magic Tricks Ruby and pipe(2) # pipe.rb read_end, write_end = IO.pipe fork do read_end.close write_end.write('Hello from your child!') write_end.close end write_end.close Process.wait message = read_end.read read_end.close puts "Received from child: ‘#{message}'" # => Received from child: 'Hello from your child!'

@thorstenball Unicorn Unix Magic Tricks How does Unicorn use pipe(2)? • Pipe between each worker process and master process • Pipe only used inside master process • Pipe used to coordinate daemonization of process

@thorstenball Unicorn Unix Magic Tricks sockets & select(2)

@thorstenball Unicorn Unix Magic Tricks sockets • Unix networking is sockets • TCP sockets, UDP sockets, Unix domain sockets • sockets are files • system call socket(2) returns a socket

@thorstenball Unicorn Unix Magic Tricks socket lifecycle • socket(2) • bind(2) • listen(2) • accept(2)

@thorstenball Unicorn Unix Magic Tricks select(2) • system call (man 2 select) • monitor file descriptors until they are ready to read/ write • blocking

@thorstenball Unicorn Unix Magic Tricks Ruby, sockets and accept(2) sock1 = Socket.new(:INET, :STREAM) addr1 = Socket.pack_sockaddr_in(8888, '0.0.0.0') sock1.bind(addr1) sock1.listen(10) sock2 = Socket.new(:INET, :STREAM) addr2 = Socket.pack_sockaddr_in(9999, '0.0.0.0') sock2.bind(addr2) sock2.listen(10) 5.times do fork do loop do readable, _, _ = IO.select([sock1, sock2]) connection, _ = readable.first.accept puts "[#{Process.pid}] #{connection.read}" connection.close end end end Process.wait

@thorstenball Unicorn Unix Magic Tricks Ruby, sockets and accept(2) $ ruby tcp_sockets_example.rb [31605] foobar1 [31607] foobar2 [31605] foobar3 [31607] foobar4 [31609] foobar5 $ echo 'foobar1' | nc localhost 9999 $ echo 'foobar2' | nc localhost 9999 $ echo 'foobar3' | nc localhost 8888 $ echo 'foobar4' | nc localhost 8888 $ echo 'foobar5' | nc localhost 9999 Server Client

@thorstenball Unicorn Unix Magic Tricks Unicorn and sockets • Master does socket, bind, listen • Workers inherit listening socket and pipe • Workers call select and accept on socket and pipe • Workers pass connections to Rack/Rails app • Unicorn even allows multiple listening sockets!

@thorstenball Unicorn Unix Magic Tricks Signals $ kill -9 8433

@thorstenball Unicorn Unix Magic Tricks Signal Handling • You can define signal actions • You can ignore signals • You can redefine signal actions • Some signals can’t be caught/ignored (KILL)

@thorstenball Unicorn Unix Magic Tricks Ruby and signal handling # signals.rb trap(:SIGKILL) do puts "You won't see this" end trap(:SIGQUIT) do puts "SIGQUIT received" end trap(:SIGUSR1) do puts "SIGUSR1 received" end puts "My PID is #{Process.pid}. Send me some signals!" sleep 100

@thorstenball Unicorn Unix Magic Tricks $ ruby signals.rb My PID is 31950. Send me some signals! SIGUSR1 received SIGQUIT received zsh: killed ruby signals.rb $ kill -USR1 31950 $ kill -QUIT 31950 $ kill -KILL 31950 ! Server Client Ruby and signal handling

@thorstenball Unicorn Unix Magic Tricks Unicorn and signals • QUIT - graceful shutdown, let workers finish the work • TERM and INT - immediate shutdown • USR1 - reopen log files • USR2 - hot reload! • TTIN/TTOU - increase/decrease the number of workers • HUP - reload the configuration file • WINCH - keep master running, gracefully stop workers

@thorstenball Unicorn Unix Magic Tricks Unicorn and signals • Master process sets up a self-pipe and a queue • Signal handlers write signal name into the queue, write to self- pipe • Workers inherit signal handlers, ignore most of them • Master process calls IO.select on self-pipe to check for signals in main loop • Signals are sent from master to worker via pipes

@thorstenball Unicorn Unix Magic Tricks Magic?

@thorstenball Unicorn Unix Magic Tricks Pre-Loading

@thorstenball Unicorn Unix Magic Tricks Pre-Loading • Master boots up, defines lambda that loads application • If preload_app is true, master calls the lambda • Application is loaded into memory • Master calls fork, spawns workers • Workers have application in memory

@thorstenball Unicorn Unix Magic Tricks Scaling workers with signals $ kill -TTIN 17032 $ kill -TTOU 17032

@thorstenball Unicorn Unix Magic Tricks Scaling workers with signals • Master process traps TTIN and TTOU signals • Signal handlers write signal to queue, awake master • In the master main loop: master reads signal, changes worker_processes count • In new iteration of main loop the master sees that workers need to be increased/decreased • Master spawns missing workers or writes QUIT signal to worker pipe

@thorstenball Unicorn Unix Magic Tricks Scaling workers with signals def spawn_missing_workers worker_nr = -1 until (worker_nr += 1) == @worker_processes WORKERS.value?(worker_nr) and next worker = Worker.new(worker_nr) before_fork.call(self, worker) if pid = fork WORKERS[pid] = worker worker.atfork_parent else after_fork_internal worker_loop(worker) exit end end rescue => e @logger.error(e) rescue nil exit! end

@thorstenball Unicorn Unix Magic Tricks Scaling workers with signals def maintain_worker_count (off = WORKERS.size - worker_processes) == 0 and return off < 0 and return spawn_missing_workers WORKERS.each_value { |w| w.nr >= worker_processes and w.soft_kill(:QUIT) } end

@thorstenball Unicorn Unix Magic Tricks And now…

@thorstenball Unicorn Unix Magic Tricks Hot Reload! $ pstree | grep unicorn \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080

@thorstenball Unicorn Unix Magic Tricks Hot Reload • Master and worker processes are happily doing their work • Master process receives USR2 signal • Signal handler queues up signal, writes to self-pipe • Master process reads signal and calls its #reexec method

@thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec def reexec if reexec_pid > 0 begin Process.kill(0, reexec_pid) logger.error "reexec-ed child already running PID:#{reexec_pid}" return rescue Errno::ESRCH self.reexec_pid = 0 end end if pid old_pid = "#{pid}.oldbin" begin self.pid = old_pid # clear the path for a new pid file rescue ArgumentError logger.error "old PID:#{valid_pid?(old_pid)} running with " \ "existing pid=#{old_pid}, refusing rexec" return rescue => e logger.error "error writing pid=#{old_pid} #{e.class} #{e.message}" return end end self.reexec_pid = fork do listener_fds = {} LISTENERS.each do |sock| # IO#close_on_exec= will be available on any future version of # Ruby that sets FD_CLOEXEC by default on new file descriptors # ref: http://redmine.ruby-lang.org/issues/5041 sock.close_on_exec = false if sock.respond_to?(:close_on_exec=) listener_fds[sock.fileno] = sock end ENV['UNICORN_FD'] = listener_fds.keys.join(',') Dir.chdir(START_CTX[:cwd]) cmd = [ START_CTX[0] ].concat(START_CTX[:argv]) # avoid leaking FDs we don't know about, but let before_exec # unset FD_CLOEXEC, if anything else in the app eventually # relies on FD inheritence. (3..1024).each do |io| next if listener_fds.include?(io) io = IO.for_fd(io) rescue next prevent_autoclose(io) io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) end # exec(command, hash) works in at least 1.9.1+, but will only be # required in 1.9.4/2.0.0 at earliest. cmd << listener_fds if RUBY_VERSION >= "1.9.1" logger.info "cmd: #{cmd}" logger.info "executing #{cmd.inspect} (in #{Dir.pwd})" before_exec.call(self) exec(*cmd) end proc_name 'master (old)' end

@thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec • Return if already re-executing • Write PID to pidfile.pid.old • fork! Parent saves PID of new child and returns • Child writes FD number of listening sockets to ENV variable • Child closes unneeded sockets and files • Child calls exec with the original arguments: turns into new Unicorn master process

@thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec • New master process boots up with new application code • New master process checks ENV for socket FDs • Casts socket FDs into socket objects (IO.for_fd) • Spawns off workers, which start select/accept loop • No “address already in use”: sockets are inherited! • Two sets of master/worker processes running • Old process is now safe to be killed

@thorstenball Unicorn Unix Magic Tricks No magic! Just Unix.

@thorstenball Unicorn Unix Magic Tricks Why?

@thorstenball Unicorn Unix Magic Tricks That’s why! • Debugging • Design and Architecture • One more level of abstraction • Concurrency • It’s not magic!

@thorstenball Unicorn Unix Magic Tricks Thank you! … and talk to me!