Upgrade to Pro — share decks privately, control downloads, hide ads and more …

I love UNIX and so can you

I love UNIX and so can you

John Downey

July 14, 2016
Tweet

More Decks by John Downey

Other Decks in Technology

Transcript

  1. I ❤ UNIX
    and so can you
    @jtdowney 1

    View full-size slide

  2. History
    @jtdowney 6

    View full-size slide

  3. Research UNIX
    @jtdowney 7

    View full-size slide

  4. 1. a hierarchical file system incorporating demountable
    volumes
    2. compatible file, device, and inter-process I/O
    3. the ability to initiate asynchronous processes
    4. system command language selectable on a per-user
    basis
    5. over 100 subsystems including a dozen languages
    @jtdowney 9

    View full-size slide

  5. Why was UNIX successful
    1. The UNIX system was written in a high-level
    language.
    2. The UNIX system was distributed in source form.
    3. The UNIX system provided powerful primitives
    normally found in only those operating systems that
    ran on much more expensive hardware.
    @jtdowney 10

    View full-size slide

  6. Berkley Software
    Distribution (BSD)
    @jtdowney 11

    View full-size slide

  7. @jtdowney 12

    View full-size slide

  8. UNIX Wars
    @jtdowney 13

    View full-size slide

  9. What does UNIX mean
    now?
    @jtdowney 14

    View full-size slide

  10. Officially it means
    4 POSIX
    4 Single UNIX Specification
    4 The Open Group certifies compatible systems
    4 Solaris
    4 OS X
    @jtdowney 15

    View full-size slide

  11. @jtdowney 16

    View full-size slide

  12. Unofficially
    4 UNIX-like systems
    4 POSIX
    4 Aren't certified for the Single UNIX Specification
    4 Linux
    4 BSD (Free/Net/Open)
    4 illumos
    @jtdowney 17

    View full-size slide


  13. @jtdowney 18

    View full-size slide

  14. UNIX Philosophy
    4 Write programs that do one thing and do it well.
    4 Write programs to work together.
    4 Write programs to handle text streams, because that
    is a universal interface.
    @jtdowney 19

    View full-size slide

  15. Functional operations on data
    4 filter (grep)
    4 map (sed / awk)
    4 reduce (uniq / wc / awk)
    @jtdowney 20

    View full-size slide

  16. netstat -an | grep -i established
    @jtdowney 21

    View full-size slide

  17. Everything is a file
    (descriptor)
    @jtdowney 22

    View full-size slide

  18. netstat -an
    % lsof -p 9576
    COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    netstat 9576 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault
    netstat 9576 pair rtd DIR 202,1 4096 2 /
    netstat 9576 pair txt REG 202,1 119784 10616898 /bin/netstat
    netstat 9576 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so
    netstat 9576 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so
    netstat 9576 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive
    netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 23

    View full-size slide

  19. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 24

    View full-size slide

  20. % tty
    /dev/pts/3
    @jtdowney 25

    View full-size slide

  21. Every process starts with 3 file
    descriptors open
    4 0 = STDIN
    4 1 = STDOUT
    4 2 = STDERR
    @jtdowney 26

    View full-size slide

  22. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3
    netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 27

    View full-size slide

  23. netstat -an | grep -i established
    % lsof -p 12538
    COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    netstat 12538 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault
    netstat 12538 pair rtd DIR 202,1 4096 2 /
    netstat 12538 pair txt REG 202,1 119784 10616898 /bin/netstat
    netstat 12538 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so
    netstat 12538 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so
    netstat 12538 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive
    netstat 12538 pair 0u CHR 136,3 0t0 6 /dev/pts/3
    netstat 12538 pair 1w FIFO 0,8 0t0 735322872 pipe
    netstat 12538 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 28

    View full-size slide

  24. netstat -an | grep -i established
    % lsof -p 12539
    COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    grep 12539 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault
    grep 12539 pair rtd DIR 202,1 4096 2 /
    grep 12539 pair txt REG 202,1 175488 10616876 /bin/grep
    grep 12539 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so
    grep 12539 pair mem REG 202,1 14768 2886786 /lib/x86_64-linux-gnu/libdl-2.13.so
    grep 12539 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so
    grep 12539 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive
    grep 12539 pair mem REG 202,1 26066 11807983 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
    grep 12539 pair 0r FIFO 0,8 0t0 735322872 pipe
    grep 12539 pair 1u CHR 136,3 0t0 6 /dev/pts/3
    grep 12539 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 29

    View full-size slide

  25. Shell Redirection
    @jtdowney 30

    View full-size slide

  26. grep test < /dev/zero
    % lsof -p 13828
    COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    grep 13828 pair cwd DIR 202,1 4096 7864322 /home/pair
    grep 13828 pair rtd DIR 202,1 4096 2 /
    grep 13828 pair txt REG 202,1 175488 10616876 /bin/grep
    grep 13828 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so
    grep 13828 pair mem REG 202,1 14768 2886786 /lib/x86_64-linux-gnu/libdl-2.13.so
    grep 13828 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so
    grep 13828 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive
    grep 13828 pair mem REG 202,1 26066 11807983 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
    grep 13828 pair 0r CHR 1,5 0t0 1030 /dev/zero
    grep 13828 pair 1u CHR 136,3 0t0 6 /dev/pts/3
    grep 13828 pair 2u CHR 136,3 0t0 6 /dev/pts/3
    @jtdowney 31

    View full-size slide

  27. System Calls
    @jtdowney 32

    View full-size slide

  28. % echo "Hello World" > test
    % cat test
    Hello World
    @jtdowney 33

    View full-size slide

  29. % strace cat test
    execve("/bin/cat", ["cat", "test"], [/* 86 vars */]) = 0
    ...
    open("test", O_RDONLY) = 3
    read(3, "Hello World\n", 32768) = 12
    write(1, "Hello World\n", 12) = 12
    read(3, "", 32768) = 0
    close(3) = 0
    close(1) = 0
    close(2) = 0
    exit_group(0) = ?
    @jtdowney 34

    View full-size slide

  30. Common file-related system calls
    4 creat - create a file
    4 open - open a file
    4 read - read from a file descriptor
    4 write - write to a file descriptor
    4 close - close a file descriptor
    @jtdowney 35

    View full-size slide

  31. Expose special files in mount points
    % mount
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=3855906,mode=755)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3089000k,mode=755)
    /dev/xvda1 on / type ext4 (rw,relatime,stripe=128,data=ordered)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6177980k)
    @jtdowney 36

    View full-size slide

  32. % ls /proc
    1/ 111/ 13717/ 17293/ 20/ 26/ 28952/ 34/ 4125/ 44/ 4716/
    10/ 112/ 13719/ 173/ 208/ 26538/ 28962/ 35/ 4136/ 4403/ 4718/
    100/ 113/ 13720/ 174/ 209/ 26549/ 29/ 36/ 42/ 4422/ 4721/
    101/ 117/ 13926/ 176/ 21/ 267/ 3/ 37/ 4236/ 4445/ 4723/
    102/ 12/ 14/ 18/ 22/ 27/ 30/ 38/ 4276/ 45/ 4769/
    103/ 13/ 15/ 185/ 23/ 274/ 31/ 39/ 43/ 46/ 4776/
    108/ 13329/ 1533/ 186/ 24/ 28/ 32/ 3980/ 4349/ 4618/ 4778/
    109/ 13334/ 16/ 19/ 249/ 28162/ 33/ 4/ 4361/ 4683/ 4795/
    11/ 13337/ 16276/ 1969/ 25/ 28883/ 335/ 40/ 4366/ 47/ 48/
    110/ 13346/ 17/ 2/ 250/ 28888/ 339/ 41/ 4387/ 4711/ 4806/
    @jtdowney 37

    View full-size slide

  33. % ls /proc/self
    attr/ coredump_filter gid_map mountinfo oom_score sessionid task/
    autogroup cpuset io mounts oom_score_adj smaps timers
    auxv cwd@ limits mountstats pagemap stack uid_map
    cgroup environ loginuid net/ personality stat wchan
    clear_refs exe@ map_files/ ns/ projid_map statm
    cmdline fd/ maps numa_maps root@ status
    comm fdinfo/ mem oom_adj sched syscall
    @jtdowney 38

    View full-size slide

  34. % cat /proc/self/status
    Name: zsh
    State: S (sleeping)
    Tgid: 20292
    Ngid: 0
    Pid: 20292
    PPid: 7596
    TracerPid: 0
    Uid: 1000 1000 1000 1000
    Gid: 1000 1000 1000 1000
    FDSize: 64
    Groups: 4 106 918 1000 1100 2000
    VmPeak: 45772 kB
    @jtdowney 39

    View full-size slide

  35. Process Model
    @jtdowney 40

    View full-size slide

  36. 4 pid - process id
    4 ppid - parent process id
    4 uid - user id
    4 gid - group id
    @jtdowney 41

    View full-size slide

  37. % pstree
    init─┬─zsh───pstree
    ├─7*[getty]
    ├─2*[master─┬─pickup]
    │ └─qmgr]
    ├─atd
    ├─cron───cron───sh───vpair───{vpair}
    ├─docker─┬─docker-containe───9*[{docker-containe}]
    │ └─24*[{docker}]
    ├─epmd
    ├─haproxy
    ├─java───128*[{java}]
    @jtdowney 42

    View full-size slide

  38. fork()
    @jtdowney 43

    View full-size slide

  39. What does it get?
    4 Get a copy of the file descriptors
    4 Easy to use pipes for inter-process
    communication
    4 Share virtual memory (copy on write)
    @jtdowney 44

    View full-size slide

  40. require "logger"
    logger = Logger.new($stdout)
    logger.info "starting parent"
    reader, writer = IO.pipe
    fork do
    writer.close
    logger.info "starting child"
    logger.info reader.gets
    end
    reader.close
    writer.write "sent from parent process"
    @jtdowney 45

    View full-size slide

  41. % ruby fork_test.rb
    I, [2016-07-14T16:52:38.440600 #28966] INFO -- : starting parent
    I, [2016-07-14T16:52:38.442533 #28968] INFO -- : starting child
    I, [2016-07-14T16:52:38.444770 #28968] INFO -- : sent from parent process
    @jtdowney 46

    View full-size slide

  42. Signals
    @jtdowney 47

    View full-size slide

  43. Signal Value Action Comment
    ──────────────────────────────────────────────────────────────────────
    SIGHUP 1 Term Hangup detected on controlling terminal
    or death of controlling process
    SIGINT 2 Term Interrupt from keyboard
    SIGQUIT 3 Core Quit from keyboard
    SIGILL 4 Core Illegal Instruction
    SIGABRT 6 Core Abort signal from abort(3)
    SIGFPE 8 Core Floating point exception
    SIGKILL 9 Term Kill signal
    SIGSEGV 11 Core Invalid memory reference
    SIGPIPE 13 Term Broken pipe: write to pipe with no
    readers
    SIGALRM 14 Term Timer signal from alarm(2)
    SIGTERM 15 Term Termination signal
    SIGUSR1 30,10,16 Term User-defined signal 1
    SIGUSR2 31,12,17 Term User-defined signal 2
    SIGCHLD 20,17,18 Ign Child stopped or terminated
    SIGCONT 19,18,25 Cont Continue if stopped
    SIGSTOP 17,19,23 Stop Stop process
    SIGTSTP 18,20,24 Stop Stop typed at tty
    SIGTTIN 21,21,26 Stop tty input for background process
    SIGTTOU 22,22,27 Stop tty output for background process
    The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored.
    @jtdowney 48

    View full-size slide

  44. 4 SIGHUP - sent when you close a terminal on a running
    process
    4 SIGINT - sent when you CTRL-C
    4 SIGTERM - sent with plain kill
    4 SIGKILL - not messing around, uncatchable
    @jtdowney 49

    View full-size slide

  45. Questions
    @jtdowney 50

    View full-size slide

  46. References
    4 Books
    4 The Daemon, the Gnu, and the Penguin by Peter H. Salus, et al
    4 The Design and Implementation of the 4.4 BSD Operating System by Marshall Kirk McKusick, et al
    4 Lions' Commentary on Unix 6th Edition by John Lions
    4 Papers
    4 The UNIX Time-Sharing System by Dennis M. Ritchie and Ken Thompson
    4 The Evolution of the Unix Time-sharing System by Dennis M. Ritchie
    4 Articles
    4 https://en.wikipedia.org/wiki/History_of_Unix
    4 https://en.wikipedia.org/wiki/Unix_philosophy
    @jtdowney 51

    View full-size slide