Slide 1

Slide 1 text

I ❤ UNIX and so can you @jtdowney 1

Slide 2

Slide 2 text

@jtdowney 2

Slide 3

Slide 3 text

@jtdowney 3

Slide 4

Slide 4 text

@jtdowney 4

Slide 5

Slide 5 text

@jtdowney 5

Slide 6

Slide 6 text

History @jtdowney 6

Slide 7

Slide 7 text

Research UNIX @jtdowney 7

Slide 8

Slide 8 text

@jtdowney 8

Slide 9

Slide 9 text

1. a hierarchical file system incorporating demountable volumes 2. compatible file, device, and inter-process I/O 3. the ability to initiate asynchronous processes 4. system command language selectable on a per-user basis 5. over 100 subsystems including a dozen languages @jtdowney 9

Slide 10

Slide 10 text

Why was UNIX successful 1. The UNIX system was written in a high-level language. 2. The UNIX system was distributed in source form. 3. The UNIX system provided powerful primitives normally found in only those operating systems that ran on much more expensive hardware. @jtdowney 10

Slide 11

Slide 11 text

Berkley Software Distribution (BSD) @jtdowney 11

Slide 12

Slide 12 text

@jtdowney 12

Slide 13

Slide 13 text

UNIX Wars @jtdowney 13

Slide 14

Slide 14 text

What does UNIX mean now? @jtdowney 14

Slide 15

Slide 15 text

Officially it means 4 POSIX 4 Single UNIX Specification 4 The Open Group certifies compatible systems 4 Solaris 4 OS X @jtdowney 15

Slide 16

Slide 16 text

@jtdowney 16

Slide 17

Slide 17 text

Unofficially 4 UNIX-like systems 4 POSIX 4 Aren't certified for the Single UNIX Specification 4 Linux 4 BSD (Free/Net/Open) 4 illumos @jtdowney 17

Slide 18

Slide 18 text

❤ @jtdowney 18

Slide 19

Slide 19 text

UNIX Philosophy 4 Write programs that do one thing and do it well. 4 Write programs to work together. 4 Write programs to handle text streams, because that is a universal interface. @jtdowney 19

Slide 20

Slide 20 text

Functional operations on data 4 filter (grep) 4 map (sed / awk) 4 reduce (uniq / wc / awk) @jtdowney 20

Slide 21

Slide 21 text

netstat -an | grep -i established @jtdowney 21

Slide 22

Slide 22 text

Everything is a file (descriptor) @jtdowney 22

Slide 23

Slide 23 text

netstat -an % lsof -p 9576 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME netstat 9576 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault netstat 9576 pair rtd DIR 202,1 4096 2 / netstat 9576 pair txt REG 202,1 119784 10616898 /bin/netstat netstat 9576 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so netstat 9576 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so netstat 9576 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 23

Slide 24

Slide 24 text

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 24

Slide 25

Slide 25 text

% tty /dev/pts/3 @jtdowney 25

Slide 26

Slide 26 text

Every process starts with 3 file descriptors open 4 0 = STDIN 4 1 = STDOUT 4 2 = STDERR @jtdowney 26

Slide 27

Slide 27 text

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME netstat 9576 pair 0u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 1u CHR 136,3 0t0 6 /dev/pts/3 netstat 9576 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 27

Slide 28

Slide 28 text

netstat -an | grep -i established % lsof -p 12538 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME netstat 12538 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault netstat 12538 pair rtd DIR 202,1 4096 2 / netstat 12538 pair txt REG 202,1 119784 10616898 /bin/netstat netstat 12538 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so netstat 12538 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so netstat 12538 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive netstat 12538 pair 0u CHR 136,3 0t0 6 /dev/pts/3 netstat 12538 pair 1w FIFO 0,8 0t0 735322872 pipe netstat 12538 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 28

Slide 29

Slide 29 text

netstat -an | grep -i established % lsof -p 12539 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME grep 12539 pair cwd DIR 202,1 4096 8787609 /home/pair/bt/vault grep 12539 pair rtd DIR 202,1 4096 2 / grep 12539 pair txt REG 202,1 175488 10616876 /bin/grep grep 12539 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so grep 12539 pair mem REG 202,1 14768 2886786 /lib/x86_64-linux-gnu/libdl-2.13.so grep 12539 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so grep 12539 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive grep 12539 pair mem REG 202,1 26066 11807983 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache grep 12539 pair 0r FIFO 0,8 0t0 735322872 pipe grep 12539 pair 1u CHR 136,3 0t0 6 /dev/pts/3 grep 12539 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 29

Slide 30

Slide 30 text

Shell Redirection @jtdowney 30

Slide 31

Slide 31 text

grep test < /dev/zero % lsof -p 13828 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME grep 13828 pair cwd DIR 202,1 4096 7864322 /home/pair grep 13828 pair rtd DIR 202,1 4096 2 / grep 13828 pair txt REG 202,1 175488 10616876 /bin/grep grep 13828 pair mem REG 202,1 1607696 2886783 /lib/x86_64-linux-gnu/libc-2.13.so grep 13828 pair mem REG 202,1 14768 2886786 /lib/x86_64-linux-gnu/libdl-2.13.so grep 13828 pair mem REG 202,1 136936 2886666 /lib/x86_64-linux-gnu/ld-2.13.so grep 13828 pair mem REG 202,1 1534672 11798922 /usr/lib/locale/locale-archive grep 13828 pair mem REG 202,1 26066 11807983 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache grep 13828 pair 0r CHR 1,5 0t0 1030 /dev/zero grep 13828 pair 1u CHR 136,3 0t0 6 /dev/pts/3 grep 13828 pair 2u CHR 136,3 0t0 6 /dev/pts/3 @jtdowney 31

Slide 32

Slide 32 text

System Calls @jtdowney 32

Slide 33

Slide 33 text

% echo "Hello World" > test % cat test Hello World @jtdowney 33

Slide 34

Slide 34 text

% strace cat test execve("/bin/cat", ["cat", "test"], [/* 86 vars */]) = 0 ... open("test", O_RDONLY) = 3 read(3, "Hello World\n", 32768) = 12 write(1, "Hello World\n", 12) = 12 read(3, "", 32768) = 0 close(3) = 0 close(1) = 0 close(2) = 0 exit_group(0) = ? @jtdowney 34

Slide 35

Slide 35 text

Common file-related system calls 4 creat - create a file 4 open - open a file 4 read - read from a file descriptor 4 write - write to a file descriptor 4 close - close a file descriptor @jtdowney 35

Slide 36

Slide 36 text

Expose special files in mount points % mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=3855906,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3089000k,mode=755) /dev/xvda1 on / type ext4 (rw,relatime,stripe=128,data=ordered) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6177980k) @jtdowney 36

Slide 37

Slide 37 text

% ls /proc 1/ 111/ 13717/ 17293/ 20/ 26/ 28952/ 34/ 4125/ 44/ 4716/ 10/ 112/ 13719/ 173/ 208/ 26538/ 28962/ 35/ 4136/ 4403/ 4718/ 100/ 113/ 13720/ 174/ 209/ 26549/ 29/ 36/ 42/ 4422/ 4721/ 101/ 117/ 13926/ 176/ 21/ 267/ 3/ 37/ 4236/ 4445/ 4723/ 102/ 12/ 14/ 18/ 22/ 27/ 30/ 38/ 4276/ 45/ 4769/ 103/ 13/ 15/ 185/ 23/ 274/ 31/ 39/ 43/ 46/ 4776/ 108/ 13329/ 1533/ 186/ 24/ 28/ 32/ 3980/ 4349/ 4618/ 4778/ 109/ 13334/ 16/ 19/ 249/ 28162/ 33/ 4/ 4361/ 4683/ 4795/ 11/ 13337/ 16276/ 1969/ 25/ 28883/ 335/ 40/ 4366/ 47/ 48/ 110/ 13346/ 17/ 2/ 250/ 28888/ 339/ 41/ 4387/ 4711/ 4806/ @jtdowney 37

Slide 38

Slide 38 text

% ls /proc/self attr/ coredump_filter gid_map mountinfo oom_score sessionid task/ autogroup cpuset io mounts oom_score_adj smaps timers auxv cwd@ limits mountstats pagemap stack uid_map cgroup environ loginuid net/ personality stat wchan clear_refs exe@ map_files/ ns/ projid_map statm cmdline fd/ maps numa_maps root@ status comm fdinfo/ mem oom_adj sched syscall @jtdowney 38

Slide 39

Slide 39 text

% cat /proc/self/status Name: zsh State: S (sleeping) Tgid: 20292 Ngid: 0 Pid: 20292 PPid: 7596 TracerPid: 0 Uid: 1000 1000 1000 1000 Gid: 1000 1000 1000 1000 FDSize: 64 Groups: 4 106 918 1000 1100 2000 VmPeak: 45772 kB @jtdowney 39

Slide 40

Slide 40 text

Process Model @jtdowney 40

Slide 41

Slide 41 text

4 pid - process id 4 ppid - parent process id 4 uid - user id 4 gid - group id @jtdowney 41

Slide 42

Slide 42 text

% pstree init─┬─zsh───pstree ├─7*[getty] ├─2*[master─┬─pickup] │ └─qmgr] ├─atd ├─cron───cron───sh───vpair───{vpair} ├─docker─┬─docker-containe───9*[{docker-containe}] │ └─24*[{docker}] ├─epmd ├─haproxy ├─java───128*[{java}] @jtdowney 42

Slide 43

Slide 43 text

fork() @jtdowney 43

Slide 44

Slide 44 text

What does it get? 4 Get a copy of the file descriptors 4 Easy to use pipes for inter-process communication 4 Share virtual memory (copy on write) @jtdowney 44

Slide 45

Slide 45 text

require "logger" logger = Logger.new($stdout) logger.info "starting parent" reader, writer = IO.pipe fork do writer.close logger.info "starting child" logger.info reader.gets end reader.close writer.write "sent from parent process" @jtdowney 45

Slide 46

Slide 46 text

% ruby fork_test.rb I, [2016-07-14T16:52:38.440600 #28966] INFO -- : starting parent I, [2016-07-14T16:52:38.442533 #28968] INFO -- : starting child I, [2016-07-14T16:52:38.444770 #28968] INFO -- : sent from parent process @jtdowney 46

Slide 47

Slide 47 text

Signals @jtdowney 47

Slide 48

Slide 48 text

Signal Value Action Comment ────────────────────────────────────────────────────────────────────── SIGHUP 1 Term Hangup detected on controlling terminal or death of controlling process SIGINT 2 Term Interrupt from keyboard SIGQUIT 3 Core Quit from keyboard SIGILL 4 Core Illegal Instruction SIGABRT 6 Core Abort signal from abort(3) SIGFPE 8 Core Floating point exception SIGKILL 9 Term Kill signal SIGSEGV 11 Core Invalid memory reference SIGPIPE 13 Term Broken pipe: write to pipe with no readers SIGALRM 14 Term Timer signal from alarm(2) SIGTERM 15 Term Termination signal SIGUSR1 30,10,16 Term User-defined signal 1 SIGUSR2 31,12,17 Term User-defined signal 2 SIGCHLD 20,17,18 Ign Child stopped or terminated SIGCONT 19,18,25 Cont Continue if stopped SIGSTOP 17,19,23 Stop Stop process SIGTSTP 18,20,24 Stop Stop typed at tty SIGTTIN 21,21,26 Stop tty input for background process SIGTTOU 22,22,27 Stop tty output for background process The signals SIGKILL and SIGSTOP cannot be caught, blocked, or ignored. @jtdowney 48

Slide 49

Slide 49 text

4 SIGHUP - sent when you close a terminal on a running process 4 SIGINT - sent when you CTRL-C 4 SIGTERM - sent with plain kill 4 SIGKILL - not messing around, uncatchable @jtdowney 49

Slide 50

Slide 50 text

Questions @jtdowney 50

Slide 51

Slide 51 text

References 4 Books 4 The Daemon, the Gnu, and the Penguin by Peter H. Salus, et al 4 The Design and Implementation of the 4.4 BSD Operating System by Marshall Kirk McKusick, et al 4 Lions' Commentary on Unix 6th Edition by John Lions 4 Papers 4 The UNIX Time-Sharing System by Dennis M. Ritchie and Ken Thompson 4 The Evolution of the Unix Time-sharing System by Dennis M. Ritchie 4 Articles 4 https://en.wikipedia.org/wiki/History_of_Unix 4 https://en.wikipedia.org/wiki/Unix_philosophy @jtdowney 51