Unicorn Unix Magic Tricks

Unicorn Unix Magic Tricks

A talk about the Unicorn webserver, how it works, why it looks like magic and why, in the end, it's not magic but Unix that's behind the great features of Unicorn.

Given at ArrrrCamp 2014 in Ghent, Belgium.

324b2e4d8ae9fcbd7b2983f13481075a?s=128

Thorsten Ball

October 02, 2014
Tweet

Transcript

  1. Hello! Thorsten Ball mrnugget / @thorstenball Software Developer

  2. None
  3. @thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

  4. @thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

  5. @thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

  6. @thorstenball Unicorn Unix Magic Tricks Master-Worker Achitecture $ pstree |

    grep unicorn \-+= 27185 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27210 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27211 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27212 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27213 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27214 mrnugget unicorn worker[4] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27215 mrnugget unicorn worker[5] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27216 mrnugget unicorn worker[6] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27217 mrnugget unicorn worker[7] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27218 mrnugget unicorn worker[8] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27219 mrnugget unicorn worker[9] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27220 mrnugget unicorn worker[10] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27221 mrnugget unicorn worker[11] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27222 mrnugget unicorn worker[12] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27223 mrnugget unicorn worker[13] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27224 mrnugget unicorn worker[14] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27225 mrnugget unicorn worker[15] -c simple_unicorn_config.rb -l0.0.0.0:8080
  7. @thorstenball Unicorn Unix Magic Tricks Hot Reload $ pstree |

    grep unicorn \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
  8. @thorstenball Unicorn Unix Magic Tricks Signals TERM QUIT HUP USR1

    USR2 WINCH TTIN TTOU
  9. @thorstenball Unicorn Unix Magic Tricks Pre-Loading

  10. @thorstenball Unicorn Unix Magic Tricks ./PHILOSOPHY I MEAN, COME ON!

  11. @thorstenball Unicorn Unix Magic Tricks Unicorn Unix Magic Tricks

  12. @thorstenball Unicorn Unix Magic Tricks How does Unicorn work?

  13. @thorstenball Unicorn Unix Magic Tricks fork, fork, fork

  14. @thorstenball Unicorn Unix Magic Tricks fork(2) • system call (man

    2 fork) • splits processes in two (“a fork in the road”) • fork(2) splits a process into parent and child • children inherit a lot from their parent processes (data, stack, heap, working directory, …)
  15. @thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) # fork.rb

    ! child_pid = fork do puts "[child] child_pid: #{child_pid}" puts "[child] Process ID: #{Process.pid}" puts "[child] Parent Process ID: #{Process.ppid}" end Process.wait(child_pid) puts "[parent] child_pid: #{child_pid}" puts "[parent] Process ID: #{Process.pid}"
  16. @thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) # fork.rb

    ! child_pid = fork do puts "[child] child_pid: #{child_pid}" puts "[child] Process ID: #{Process.pid}" puts "[child] Parent Process ID: #{Process.ppid}" end Process.wait(child_pid) puts "[parent] child_pid: #{child_pid}" puts "[parent] Process ID: #{Process.pid}" $ ruby fork.rb [child] child_pid: [child] Process ID: 29715 [child] Parent Process ID: 29695 [parent] child_pid: 29715 [parent] Process ID: 29695
  17. @thorstenball Unicorn Unix Magic Tricks Ruby and fork(2) $ pstree

    | grep fork | \-+= 29695 mrnugget ruby fork.rb | \--- 29715 mrnugget ruby fork.rb $ ruby fork.rb [child] child_pid: [child] Process ID: 29715 [child] Parent Process ID: 29695 [parent] child_pid: 29715 [parent] Process ID: 29695
  18. @thorstenball Unicorn Unix Magic Tricks How does Unicorn use fork(2)?

    # spawn_missing_workers worker_nr = -1 until (worker_nr += 1) == @worker_processes WORKERS.value?(worker_nr) and next worker = Worker.new(worker_nr) before_fork.call(self, worker) if pid = fork WORKERS[pid] = worker worker.atfork_parent else after_fork_internal worker_loop(worker) exit end end
  19. @thorstenball Unicorn Unix Magic Tricks Pipes! $ grep ‘wat’ journal.txt

    | wc -l 84
  20. @thorstenball Unicorn Unix Magic Tricks pipe(2) • system call (man

    2 pipe) • We can use them outside the shell! • Pipes are two file descriptors • One read end, one write end • Great for communication between processes • Pipes are inherited when forking
  21. @thorstenball Unicorn Unix Magic Tricks Ruby and pipe(2) # pipe.rb

    read_end, write_end = IO.pipe fork do read_end.close write_end.write('Hello from your child!') write_end.close end write_end.close Process.wait message = read_end.read read_end.close puts "Received from child: '#{message}'"
  22. @thorstenball Unicorn Unix Magic Tricks Ruby and pipe(2) # pipe.rb

    read_end, write_end = IO.pipe fork do read_end.close write_end.write('Hello from your child!') write_end.close end write_end.close Process.wait message = read_end.read read_end.close puts "Received from child: ‘#{message}'" # => Received from child: 'Hello from your child!'
  23. @thorstenball Unicorn Unix Magic Tricks How does Unicorn use pipe(2)?

    • Pipe between each worker process and master process • Pipe only used inside master process • Pipe used to coordinate daemonization of process
  24. @thorstenball Unicorn Unix Magic Tricks sockets & select(2)

  25. @thorstenball Unicorn Unix Magic Tricks sockets • Unix networking is

    sockets • TCP sockets, UDP sockets, Unix domain sockets • sockets are files • system call socket(2) returns a socket
  26. @thorstenball Unicorn Unix Magic Tricks socket lifecycle • socket(2) •

    bind(2) • listen(2) • accept(2)
  27. @thorstenball Unicorn Unix Magic Tricks select(2) • system call (man

    2 select) • monitor file descriptors until they are ready to read/ write • blocking
  28. @thorstenball Unicorn Unix Magic Tricks Ruby, sockets and accept(2) sock1

    = Socket.new(:INET, :STREAM) addr1 = Socket.pack_sockaddr_in(8888, '0.0.0.0') sock1.bind(addr1) sock1.listen(10) sock2 = Socket.new(:INET, :STREAM) addr2 = Socket.pack_sockaddr_in(9999, '0.0.0.0') sock2.bind(addr2) sock2.listen(10) 5.times do fork do loop do readable, _, _ = IO.select([sock1, sock2]) connection, _ = readable.first.accept puts "[#{Process.pid}] #{connection.read}" connection.close end end end Process.wait
  29. @thorstenball Unicorn Unix Magic Tricks Ruby, sockets and accept(2) $

    ruby tcp_sockets_example.rb [31605] foobar1 [31607] foobar2 [31605] foobar3 [31607] foobar4 [31609] foobar5 $ echo 'foobar1' | nc localhost 9999 $ echo 'foobar2' | nc localhost 9999 $ echo 'foobar3' | nc localhost 8888 $ echo 'foobar4' | nc localhost 8888 $ echo 'foobar5' | nc localhost 9999 Server Client
  30. @thorstenball Unicorn Unix Magic Tricks Unicorn and sockets • Master

    does socket, bind, listen • Workers inherit listening socket and pipe • Workers call select and accept on socket and pipe • Workers pass connections to Rack/Rails app • Unicorn even allows multiple listening sockets!
  31. @thorstenball Unicorn Unix Magic Tricks Signals $ kill -9 8433

  32. @thorstenball Unicorn Unix Magic Tricks Signal Handling • You can

    define signal actions • You can ignore signals • You can redefine signal actions • Some signals can’t be caught/ignored (KILL)
  33. @thorstenball Unicorn Unix Magic Tricks Ruby and signal handling #

    signals.rb trap(:SIGKILL) do puts "You won't see this" end trap(:SIGQUIT) do puts "SIGQUIT received" end trap(:SIGUSR1) do puts "SIGUSR1 received" end puts "My PID is #{Process.pid}. Send me some signals!" sleep 100
  34. @thorstenball Unicorn Unix Magic Tricks $ ruby signals.rb My PID

    is 31950. Send me some signals! SIGUSR1 received SIGQUIT received zsh: killed ruby signals.rb $ kill -USR1 31950 $ kill -QUIT 31950 $ kill -KILL 31950 ! Server Client Ruby and signal handling
  35. @thorstenball Unicorn Unix Magic Tricks Unicorn and signals • QUIT

    - graceful shutdown, let workers finish the work • TERM and INT - immediate shutdown • USR1 - reopen log files • USR2 - hot reload! • TTIN/TTOU - increase/decrease the number of workers • HUP - reload the configuration file • WINCH - keep master running, gracefully stop workers
  36. @thorstenball Unicorn Unix Magic Tricks Unicorn and signals • Master

    process sets up a self-pipe and a queue • Signal handlers write signal name into the queue, write to self- pipe • Workers inherit signal handlers, ignore most of them • Master process calls IO.select on self-pipe to check for signals in main loop • Signals are sent from master to worker via pipes
  37. @thorstenball Unicorn Unix Magic Tricks Magic?

  38. @thorstenball Unicorn Unix Magic Tricks Pre-Loading

  39. @thorstenball Unicorn Unix Magic Tricks Pre-Loading • Master boots up,

    defines lambda that loads application • If preload_app is true, master calls the lambda • Application is loaded into memory • Master calls fork, spawns workers • Workers have application in memory
  40. @thorstenball Unicorn Unix Magic Tricks Scaling workers with signals $

    kill -TTIN 17032 $ kill -TTOU 17032
  41. @thorstenball Unicorn Unix Magic Tricks Scaling workers with signals •

    Master process traps TTIN and TTOU signals • Signal handlers write signal to queue, awake master • In the master main loop: master reads signal, changes worker_processes count • In new iteration of main loop the master sees that workers need to be increased/decreased • Master spawns missing workers or writes QUIT signal to worker pipe
  42. @thorstenball Unicorn Unix Magic Tricks Scaling workers with signals def

    spawn_missing_workers worker_nr = -1 until (worker_nr += 1) == @worker_processes WORKERS.value?(worker_nr) and next worker = Worker.new(worker_nr) before_fork.call(self, worker) if pid = fork WORKERS[pid] = worker worker.atfork_parent else after_fork_internal worker_loop(worker) exit end end rescue => e @logger.error(e) rescue nil exit! end
  43. @thorstenball Unicorn Unix Magic Tricks Scaling workers with signals def

    maintain_worker_count (off = WORKERS.size - worker_processes) == 0 and return off < 0 and return spawn_missing_workers WORKERS.each_value { |w| w.nr >= worker_processes and w.soft_kill(:QUIT) } end
  44. @thorstenball Unicorn Unix Magic Tricks And now…

  45. @thorstenball Unicorn Unix Magic Tricks Hot Reload! $ pstree |

    grep unicorn \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080 \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080 |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080 \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
  46. @thorstenball Unicorn Unix Magic Tricks Hot Reload • Master and

    worker processes are happily doing their work • Master process receives USR2 signal • Signal handler queues up signal, writes to self-pipe • Master process reads signal and calls its #reexec method
  47. @thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec def

    reexec if reexec_pid > 0 begin Process.kill(0, reexec_pid) logger.error "reexec-ed child already running PID:#{reexec_pid}" return rescue Errno::ESRCH self.reexec_pid = 0 end end if pid old_pid = "#{pid}.oldbin" begin self.pid = old_pid # clear the path for a new pid file rescue ArgumentError logger.error "old PID:#{valid_pid?(old_pid)} running with " \ "existing pid=#{old_pid}, refusing rexec" return rescue => e logger.error "error writing pid=#{old_pid} #{e.class} #{e.message}" return end end self.reexec_pid = fork do listener_fds = {} LISTENERS.each do |sock| # IO#close_on_exec= will be available on any future version of # Ruby that sets FD_CLOEXEC by default on new file descriptors # ref: http://redmine.ruby-lang.org/issues/5041 sock.close_on_exec = false if sock.respond_to?(:close_on_exec=) listener_fds[sock.fileno] = sock end ENV['UNICORN_FD'] = listener_fds.keys.join(',') Dir.chdir(START_CTX[:cwd]) cmd = [ START_CTX[0] ].concat(START_CTX[:argv]) # avoid leaking FDs we don't know about, but let before_exec # unset FD_CLOEXEC, if anything else in the app eventually # relies on FD inheritence. (3..1024).each do |io| next if listener_fds.include?(io) io = IO.for_fd(io) rescue next prevent_autoclose(io) io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) end # exec(command, hash) works in at least 1.9.1+, but will only be # required in 1.9.4/2.0.0 at earliest. cmd << listener_fds if RUBY_VERSION >= "1.9.1" logger.info "cmd: #{cmd}" logger.info "executing #{cmd.inspect} (in #{Dir.pwd})" before_exec.call(self) exec(*cmd) end proc_name 'master (old)' end
  48. @thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec •

    Return if already re-executing • Write PID to pidfile.pid.old • fork! Parent saves PID of new child and returns • Child writes FD number of listening sockets to ENV variable • Child closes unneeded sockets and files • Child calls exec with the original arguments: turns into new Unicorn master process
  49. @thorstenball Unicorn Unix Magic Tricks Hot Reload - #reexec •

    New master process boots up with new application code • New master process checks ENV for socket FDs • Casts socket FDs into socket objects (IO.for_fd) • Spawns off workers, which start select/accept loop • No “address already in use”: sockets are inherited! • Two sets of master/worker processes running • Old process is now safe to be killed
  50. @thorstenball Unicorn Unix Magic Tricks No magic! Just Unix.

  51. @thorstenball Unicorn Unix Magic Tricks Why?

  52. @thorstenball Unicorn Unix Magic Tricks That’s why! • Debugging •

    Design and Architecture • One more level of abstraction • Concurrency • It’s not magic!
  53. @thorstenball Unicorn Unix Magic Tricks Thank you! … and talk

    to me!