Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unicorn Unix Magic Tricks

Unicorn Unix Magic Tricks

A talk about the Unicorn webserver, how it works, why it looks like magic and why, in the end, it's not magic but Unix that's behind the great features of Unicorn.

Given at ArrrrCamp 2014 in Ghent, Belgium.

Thorsten Ball

October 02, 2014
Tweet

More Decks by Thorsten Ball

Other Decks in Programming

Transcript

  1. Hello!
    Thorsten Ball
    mrnugget / @thorstenball
    Software Developer

    View Slide

  2. View Slide

  3. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn Unix Magic Tricks

    View Slide

  4. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn Unix Magic Tricks

    View Slide

  5. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn Unix Magic Tricks

    View Slide

  6. @thorstenball
    Unicorn Unix Magic Tricks
    Master-Worker Achitecture
    $ pstree | grep unicorn
    \-+= 27185 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27210 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27211 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27212 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27213 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27214 mrnugget unicorn worker[4] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27215 mrnugget unicorn worker[5] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27216 mrnugget unicorn worker[6] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27217 mrnugget unicorn worker[7] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27218 mrnugget unicorn worker[8] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27219 mrnugget unicorn worker[9] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27220 mrnugget unicorn worker[10] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27221 mrnugget unicorn worker[11] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27222 mrnugget unicorn worker[12] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27223 mrnugget unicorn worker[13] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27224 mrnugget unicorn worker[14] -c simple_unicorn_config.rb -l0.0.0.0:8080
    \--- 27225 mrnugget unicorn worker[15] -c simple_unicorn_config.rb -l0.0.0.0:8080

    View Slide

  7. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload
    $ pstree | grep unicorn
    \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
    \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
    \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080

    View Slide

  8. @thorstenball
    Unicorn Unix Magic Tricks
    Signals
    TERM
    QUIT
    HUP
    USR1
    USR2
    WINCH
    TTIN
    TTOU

    View Slide

  9. @thorstenball
    Unicorn Unix Magic Tricks
    Pre-Loading

    View Slide

  10. @thorstenball
    Unicorn Unix Magic Tricks
    ./PHILOSOPHY
    I MEAN, COME ON!

    View Slide

  11. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn Unix Magic Tricks

    View Slide

  12. @thorstenball
    Unicorn Unix Magic Tricks
    How does Unicorn work?

    View Slide

  13. @thorstenball
    Unicorn Unix Magic Tricks
    fork, fork, fork

    View Slide

  14. @thorstenball
    Unicorn Unix Magic Tricks
    fork(2)
    • system call (man 2 fork)
    • splits processes in two (“a fork in the road”)
    • fork(2) splits a process into parent and child
    • children inherit a lot from their parent processes (data,
    stack, heap, working directory, …)

    View Slide

  15. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and fork(2)
    # fork.rb
    !
    child_pid = fork do
    puts "[child] child_pid: #{child_pid}"
    puts "[child] Process ID: #{Process.pid}"
    puts "[child] Parent Process ID: #{Process.ppid}"
    end
    Process.wait(child_pid)
    puts "[parent] child_pid: #{child_pid}"
    puts "[parent] Process ID: #{Process.pid}"

    View Slide

  16. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and fork(2)
    # fork.rb
    !
    child_pid = fork do
    puts "[child] child_pid: #{child_pid}"
    puts "[child] Process ID: #{Process.pid}"
    puts "[child] Parent Process ID: #{Process.ppid}"
    end
    Process.wait(child_pid)
    puts "[parent] child_pid: #{child_pid}"
    puts "[parent] Process ID: #{Process.pid}"
    $ ruby fork.rb
    [child] child_pid:
    [child] Process ID: 29715
    [child] Parent Process ID: 29695
    [parent] child_pid: 29715
    [parent] Process ID: 29695

    View Slide

  17. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and fork(2)
    $ pstree | grep fork
    | \-+= 29695 mrnugget ruby fork.rb
    | \--- 29715 mrnugget ruby fork.rb
    $ ruby fork.rb
    [child] child_pid:
    [child] Process ID: 29715
    [child] Parent Process ID: 29695
    [parent] child_pid: 29715
    [parent] Process ID: 29695

    View Slide

  18. @thorstenball
    Unicorn Unix Magic Tricks
    How does Unicorn use fork(2)?
    # spawn_missing_workers
    worker_nr = -1
    until (worker_nr += 1) == @worker_processes
    WORKERS.value?(worker_nr) and next
    worker = Worker.new(worker_nr)
    before_fork.call(self, worker)
    if pid = fork
    WORKERS[pid] = worker
    worker.atfork_parent
    else
    after_fork_internal
    worker_loop(worker)
    exit
    end
    end

    View Slide

  19. @thorstenball
    Unicorn Unix Magic Tricks
    Pipes!
    $ grep ‘wat’ journal.txt | wc -l
    84

    View Slide

  20. @thorstenball
    Unicorn Unix Magic Tricks
    pipe(2)
    • system call (man 2 pipe)
    • We can use them outside the shell!
    • Pipes are two file descriptors
    • One read end, one write end
    • Great for communication between processes
    • Pipes are inherited when forking

    View Slide

  21. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and pipe(2)
    # pipe.rb
    read_end, write_end = IO.pipe
    fork do
    read_end.close
    write_end.write('Hello from your child!')
    write_end.close
    end
    write_end.close
    Process.wait
    message = read_end.read
    read_end.close
    puts "Received from child: '#{message}'"

    View Slide

  22. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and pipe(2)
    # pipe.rb
    read_end, write_end = IO.pipe
    fork do
    read_end.close
    write_end.write('Hello from your child!')
    write_end.close
    end
    write_end.close
    Process.wait
    message = read_end.read
    read_end.close
    puts "Received from child: ‘#{message}'"
    # => Received from child: 'Hello from your child!'

    View Slide

  23. @thorstenball
    Unicorn Unix Magic Tricks
    How does Unicorn use pipe(2)?
    • Pipe between each worker process and master process
    • Pipe only used inside master process
    • Pipe used to coordinate daemonization of process

    View Slide

  24. @thorstenball
    Unicorn Unix Magic Tricks
    sockets & select(2)

    View Slide

  25. @thorstenball
    Unicorn Unix Magic Tricks
    sockets
    • Unix networking is sockets
    • TCP sockets, UDP sockets, Unix domain sockets
    • sockets are files
    • system call socket(2) returns a socket

    View Slide

  26. @thorstenball
    Unicorn Unix Magic Tricks
    socket lifecycle
    • socket(2)
    • bind(2)
    • listen(2)
    • accept(2)

    View Slide

  27. @thorstenball
    Unicorn Unix Magic Tricks
    select(2)
    • system call (man 2 select)
    • monitor file descriptors until they are ready to read/
    write
    • blocking

    View Slide

  28. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby, sockets and accept(2)
    sock1 = Socket.new(:INET, :STREAM)
    addr1 = Socket.pack_sockaddr_in(8888, '0.0.0.0')
    sock1.bind(addr1)
    sock1.listen(10)
    sock2 = Socket.new(:INET, :STREAM)
    addr2 = Socket.pack_sockaddr_in(9999, '0.0.0.0')
    sock2.bind(addr2)
    sock2.listen(10)
    5.times do
    fork do
    loop do
    readable, _, _ = IO.select([sock1, sock2])
    connection, _ = readable.first.accept
    puts "[#{Process.pid}] #{connection.read}"
    connection.close
    end
    end
    end
    Process.wait

    View Slide

  29. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby, sockets and accept(2)
    $ ruby tcp_sockets_example.rb
    [31605] foobar1
    [31607] foobar2
    [31605] foobar3
    [31607] foobar4
    [31609] foobar5
    $ echo 'foobar1' | nc localhost 9999
    $ echo 'foobar2' | nc localhost 9999
    $ echo 'foobar3' | nc localhost 8888
    $ echo 'foobar4' | nc localhost 8888
    $ echo 'foobar5' | nc localhost 9999
    Server Client

    View Slide

  30. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn and sockets
    • Master does socket, bind, listen
    • Workers inherit listening socket and pipe
    • Workers call select and accept on socket and pipe
    • Workers pass connections to Rack/Rails app
    • Unicorn even allows multiple listening sockets!

    View Slide

  31. @thorstenball
    Unicorn Unix Magic Tricks
    Signals
    $ kill -9 8433

    View Slide

  32. @thorstenball
    Unicorn Unix Magic Tricks
    Signal Handling
    • You can define signal actions
    • You can ignore signals
    • You can redefine signal actions
    • Some signals can’t be caught/ignored (KILL)

    View Slide

  33. @thorstenball
    Unicorn Unix Magic Tricks
    Ruby and signal handling
    # signals.rb
    trap(:SIGKILL) do
    puts "You won't see this"
    end
    trap(:SIGQUIT) do
    puts "SIGQUIT received"
    end
    trap(:SIGUSR1) do
    puts "SIGUSR1 received"
    end
    puts "My PID is #{Process.pid}. Send me some signals!"
    sleep 100

    View Slide

  34. @thorstenball
    Unicorn Unix Magic Tricks
    $ ruby signals.rb
    My PID is 31950. Send me some signals!
    SIGUSR1 received
    SIGQUIT received
    zsh: killed ruby signals.rb
    $ kill -USR1 31950
    $ kill -QUIT 31950
    $ kill -KILL 31950
    !
    Server Client
    Ruby and signal handling

    View Slide

  35. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn and signals
    • QUIT - graceful shutdown, let workers finish the work
    • TERM and INT - immediate shutdown
    • USR1 - reopen log files
    • USR2 - hot reload!
    • TTIN/TTOU - increase/decrease the number of workers
    • HUP - reload the configuration file
    • WINCH - keep master running, gracefully stop workers

    View Slide

  36. @thorstenball
    Unicorn Unix Magic Tricks
    Unicorn and signals
    • Master process sets up a self-pipe and a queue
    • Signal handlers write signal name into the queue, write to self-
    pipe
    • Workers inherit signal handlers, ignore most of them
    • Master process calls IO.select on self-pipe to check for
    signals in main loop
    • Signals are sent from master to worker via pipes

    View Slide

  37. @thorstenball
    Unicorn Unix Magic Tricks
    Magic?

    View Slide

  38. @thorstenball
    Unicorn Unix Magic Tricks
    Pre-Loading

    View Slide

  39. @thorstenball
    Unicorn Unix Magic Tricks
    Pre-Loading
    • Master boots up, defines lambda that loads application
    • If preload_app is true, master calls the lambda
    • Application is loaded into memory
    • Master calls fork, spawns workers
    • Workers have application in memory

    View Slide

  40. @thorstenball
    Unicorn Unix Magic Tricks
    Scaling workers with signals
    $ kill -TTIN 17032
    $ kill -TTOU 17032

    View Slide

  41. @thorstenball
    Unicorn Unix Magic Tricks
    Scaling workers with signals
    • Master process traps TTIN and TTOU signals
    • Signal handlers write signal to queue, awake master
    • In the master main loop: master reads signal, changes
    worker_processes count
    • In new iteration of main loop the master sees that workers need to
    be increased/decreased
    • Master spawns missing workers or writes QUIT signal to worker pipe

    View Slide

  42. @thorstenball
    Unicorn Unix Magic Tricks
    Scaling workers with signals
    def spawn_missing_workers
    worker_nr = -1
    until (worker_nr += 1) == @worker_processes
    WORKERS.value?(worker_nr) and next
    worker = Worker.new(worker_nr)
    before_fork.call(self, worker)
    if pid = fork
    WORKERS[pid] = worker
    worker.atfork_parent
    else
    after_fork_internal
    worker_loop(worker)
    exit
    end
    end
    rescue => e
    @logger.error(e) rescue nil
    exit!
    end

    View Slide

  43. @thorstenball
    Unicorn Unix Magic Tricks
    Scaling workers with signals
    def maintain_worker_count
    (off = WORKERS.size - worker_processes) == 0 and return
    off < 0 and return spawn_missing_workers
    WORKERS.each_value { |w| w.nr >= worker_processes and w.soft_kill(:QUIT) }
    end

    View Slide

  44. @thorstenball
    Unicorn Unix Magic Tricks
    And now…

    View Slide

  45. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload!
    $ pstree | grep unicorn
    \-+= 27426 mrnugget unicorn master (old) -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27451 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27452 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27453 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27454 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080
    \-+- 27467 mrnugget unicorn master -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27475 mrnugget unicorn worker[0] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27476 mrnugget unicorn worker[1] -c simple_unicorn_config.rb -l0.0.0.0:8080
    |--- 27477 mrnugget unicorn worker[2] -c simple_unicorn_config.rb -l0.0.0.0:8080
    \--- 27478 mrnugget unicorn worker[3] -c simple_unicorn_config.rb -l0.0.0.0:8080

    View Slide

  46. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload
    • Master and worker processes are happily doing their
    work
    • Master process receives USR2 signal
    • Signal handler queues up signal, writes to self-pipe
    • Master process reads signal and calls its #reexec
    method

    View Slide

  47. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload - #reexec
    def reexec
    if reexec_pid > 0
    begin
    Process.kill(0, reexec_pid)
    logger.error "reexec-ed child already running PID:#{reexec_pid}"
    return
    rescue Errno::ESRCH
    self.reexec_pid = 0
    end
    end
    if pid
    old_pid = "#{pid}.oldbin"
    begin
    self.pid = old_pid # clear the path for a new pid file
    rescue ArgumentError
    logger.error "old PID:#{valid_pid?(old_pid)} running with " \
    "existing pid=#{old_pid}, refusing rexec"
    return
    rescue => e
    logger.error "error writing pid=#{old_pid} #{e.class} #{e.message}"
    return
    end
    end
    self.reexec_pid = fork do
    listener_fds = {}
    LISTENERS.each do |sock|
    # IO#close_on_exec= will be available on any future version of
    # Ruby that sets FD_CLOEXEC by default on new file descriptors
    # ref: http://redmine.ruby-lang.org/issues/5041
    sock.close_on_exec = false if sock.respond_to?(:close_on_exec=)
    listener_fds[sock.fileno] = sock
    end
    ENV['UNICORN_FD'] = listener_fds.keys.join(',')
    Dir.chdir(START_CTX[:cwd])
    cmd = [ START_CTX[0] ].concat(START_CTX[:argv])
    # avoid leaking FDs we don't know about, but let before_exec
    # unset FD_CLOEXEC, if anything else in the app eventually
    # relies on FD inheritence.
    (3..1024).each do |io|
    next if listener_fds.include?(io)
    io = IO.for_fd(io) rescue next
    prevent_autoclose(io)
    io.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC)
    end
    # exec(command, hash) works in at least 1.9.1+, but will only be
    # required in 1.9.4/2.0.0 at earliest.
    cmd << listener_fds if RUBY_VERSION >= "1.9.1"
    logger.info "cmd: #{cmd}"
    logger.info "executing #{cmd.inspect} (in #{Dir.pwd})"
    before_exec.call(self)
    exec(*cmd)
    end
    proc_name 'master (old)'
    end

    View Slide

  48. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload - #reexec
    • Return if already re-executing
    • Write PID to pidfile.pid.old
    • fork! Parent saves PID of new child and returns
    • Child writes FD number of listening sockets to ENV variable
    • Child closes unneeded sockets and files
    • Child calls exec with the original arguments: turns into new Unicorn
    master process

    View Slide

  49. @thorstenball
    Unicorn Unix Magic Tricks
    Hot Reload - #reexec
    • New master process boots up with new application code
    • New master process checks ENV for socket FDs
    • Casts socket FDs into socket objects (IO.for_fd)
    • Spawns off workers, which start select/accept loop
    • No “address already in use”: sockets are inherited!
    • Two sets of master/worker processes running
    • Old process is now safe to be killed

    View Slide

  50. @thorstenball
    Unicorn Unix Magic Tricks
    No magic! Just Unix.

    View Slide

  51. @thorstenball
    Unicorn Unix Magic Tricks
    Why?

    View Slide

  52. @thorstenball
    Unicorn Unix Magic Tricks
    That’s why!
    • Debugging
    • Design and Architecture
    • One more level of abstraction
    • Concurrency
    • It’s not magic!

    View Slide

  53. @thorstenball
    Unicorn Unix Magic Tricks
    Thank you!
    … and talk to me!

    View Slide