Christian Heimes - File descriptors, Unix sockets and other POSIX wizardry

Have you ever wondered how the OS manages open files and network connections, what this 'file descriptor' thing actually is all about, or what's so special about Unix sockets? In my talk I will give you a quick tour into the I/O layer and process model of Unix-like operating systems. You will learn how to securely identify and efficiently share resources between processes.

https://us.pycon.org/2016/schedule/presentation/2019/

1. File descriptors, Unix sockets and other POSIX magic PyCon US 2016 Christian Heimes christian@python.org @ChristianHeimes
2. File descriptors, PyCon US 2016 2 Who am I • from Hamburg / Germany • Python core contributor since 2008 • PEP 379, 452, 456 • ssl, hashlib • Python Security Response Team • I put bytes and b'' into Python 2.6
3. File descriptors, PyCon US 2016 3 Professional Life • Senior Software Engineer with Red Hat • Security Engineering & Identity Management • FreeIPA IdM • Dogtag PKI
4. File descriptors, PyCon US 2016 4 Custodia Secrets for Containers
5. File descriptors, PyCon US 2016 5 Recipes Concepts & Tools Linux / Unix only Python 3
6. File descriptors, PyCon US 2016 6 Agenda • File descriptor • Operating System 101 • File descriptors and processes • Networking and sockets • Unix sockets, containers and sandboxing • Bonus track
7. File descriptors, PyCon US 2016 7 Simplifications ahead
8. Introducing file descriptors
9. File descriptors, PyCon US 2016 9 Everything is a file
10. File descriptors, PyCon US 2016 10 Every I/O operating uses a file descriptor
11. File descriptors, PyCon US 2016 11 File descriptors are used for • reading / writing files • directories • devices • inter-process communication • network communication • I/O multiplexing • file system monitoring • …
12. File descriptors, PyCon US 2016 12
13. File descriptors, PyCon US 2016 13 Standard numbers • 0: standard input (sys.stdin) • 1: standard output (sys.stdout) • 2: standard error output (sys.stderr) • -1: error (Python raises an exception)
14. File descriptors, PyCon US 2016 14 Simple example with open('example.txt') as f: print(f.read()) with open('example.txt') as f: print(f.read())
15. File descriptors, PyCon US 2016 15 Reference file by descriptor with open('example.txt') as f: stat = os.stat(f.fileno()) os.chmod(f.fileno(), 0o640) with open('example.txt') as f: stat = os.stat(f.fileno()) os.chmod(f.fileno(), 0o640)
16. File descriptors, PyCon US 2016 16 Directory file descriptor flags = os.O_RDONLY | os.O_DIRECTORY etcdir = os.open('/etc', flags) # read /etc/passwd stat = os.stat('passwd', dir_fd=etcdir)) os.close(etcdir) flags = os.O_RDONLY | os.O_DIRECTORY etcdir = os.open('/etc', flags) # read /etc/passwd stat = os.stat('passwd', dir_fd=etcdir)) os.close(etcdir)
17. File descriptors, PyCon US 2016 17 Control hardware (silence please)
18. File descriptors, PyCon US 2016 18
19. File descriptors, PyCon US 2016 19 Open CD drive with Python import os, fcntl CDROMEJECT = 0x5309 # Linux specific! fd = os.open('/dev/cdrom', os.O_RDONLY | os.O_NONBLOCK) fcntl.ioctl(fd, CDROMEJECT, 0) os.close(fd) import os, fcntl CDROMEJECT = 0x5309 # Linux specific! fd = os.open('/dev/cdrom', os.O_RDONLY | os.O_NONBLOCK) fcntl.ioctl(fd, CDROMEJECT, 0) os.close(fd)
20. Operating System 101
21. File descriptors, PyCon US 2016 21 Dark Ages application application application application disk disk memory memory network network
22. File descriptors, PyCon US 2016 22 Modern Operating Systems application application application application disk disk memory memory network network Kernel Kernel
23. File descriptors, PyCon US 2016 23 syscall
24. File descriptors, PyCon US 2016 24 Simple example # read.py with open('example.txt') as f: print(f.read()) # read.py with open('example.txt') as f: print(f.read())
25. File descriptors, PyCon US 2016 25 Simple example $ strace ./read.py open("example.txt", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0664, st_size=19, …}) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "Python is awesome!\n", 20) = 19 read(3, "", 1) = 0 write(1, "Python is awesome!\n", 19) = 19 write(1, "\n", 1) = 1 close(3) = 0 $ strace ./read.py open("example.txt", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0664, st_size=19, …}) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "Python is awesome!\n", 20) = 19 read(3, "", 1) = 0 write(1, "Python is awesome!\n", 19) = 19 write(1, "\n", 1) = 1 close(3) = 0
26. File descriptors, PyCon US 2016 26 open file table (global)
27. File descriptors, PyCon US 2016 27 file descriptor table (process)
28. File descriptors, PyCon US 2016 28 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 open file table entry 23 ... ... ... ... process 2 file descriptor table open 'example.txt'
29. File descriptors, PyCon US 2016 29 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table open 'example.txt' a second time
30. File descriptors, PyCon US 2016 30 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table duplicate file descriptor 4
31. File descriptors, PyCon US 2016 31 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table ... ... entry 117 ... ... process 2 file descriptor table rename and close
32. File descriptors, PyCon US 2016 32 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table fd 7 send file descriptor to another process
33. File descriptors, PyCon US 2016 33 process and global state file descriptor table maps to open file table cloexec flags open file table position mode owner locks credentials reference count ...
34. File descriptors, PyCon US 2016 34
35. File descriptors, PyCon US 2016 35 Spawn a new process fork & exec
36. File descriptors, PyCon US 2016 36 fork() • creates a clone of the current process • inherits copy of file descriptor table • inherits copy of memory • shallow copy with Copy-on-Write
37. File descriptors, PyCon US 2016 37 fork() example import os f = open('example.txt', mode='rb', buffering=0) pid = os.fork() print("getpid {:>5}, fork {:>5}, fd {}: {}".format( os.getpid(), pid, f.fileno(), f.read(6))) f.close() $ python3 fork.py getpid 17364, fork 17365, fd 3: b'Python' getpid 17365, fork 0, fd 3: b' is aw' import os f = open('example.txt', mode='rb', buffering=0) pid = os.fork() print("getpid {:>5}, fork {:>5}, fd {}: {}".format( os.getpid(), pid, f.fileno(), f.read(6))) f.close() $ python3 fork.py getpid 17364, fork 17365, fd 3: b'Python' getpid 17365, fork 0, fd 3: b' is aw'
38. File descriptors, PyCon US 2016 38 exec() • replaces current program code • file descriptors are inherited by default • unless cloexec flag is set • potential security risk • PEP 446 -- Make newly created file descriptors non-inheritable (Victor Stinner)
39. File descriptors, PyCon US 2016 39 Summary • applications must go through the Kernel • syscalls call into kernel space • file descriptors point to global table • global tables contains mode and position • new process with fork() and exec()
40. subprocess.PIPE $ ls | grep py
41. File descriptors, PyCon US 2016 41
42. File descriptors, PyCon US 2016 43 pipe example import os readend, writeend = os.pipe() pid = os.fork() if pid != 0: # parent process os.close(writeend) with open(readend) as f: print(f.read()) else: # child process os.close(readend) os.dup2(writeend, 1) # 1: stdout os.execl('/bin/ls', # programm 'ls', '-l', 'example.txt') # argv 0, 1, 2 import os readend, writeend = os.pipe() pid = os.fork() if pid != 0: # parent process os.close(writeend) with open(readend) as f: print(f.read()) else: # child process os.close(readend) os.dup2(writeend, 1) # 1: stdout os.execl('/bin/ls', # programm 'ls', '-l', 'example.txt') # argv 0, 1, 2 1 2 3 4 6 5 7
43. network sockets
44. File descriptors, PyCon US 2016 45
45. File descriptors, PyCon US 2016 46
46. File descriptors, PyCon US 2016 47 socket parameters • addressing / routing • IPv4: AF_INET • IPv6: AF_INET6 • flow control • TCP: SOCK_STREAM • UDP: SOCK_DGRAM
47. File descriptors, PyCon US 2016 48 socket server from socket import socket, SOCK_STREAM,AF_INET server = socket(AF_INET, SOCK_STREAM) server.bind(('0.0.0.0', 443)) server.listen(1) while True: conn, addr = server.accept() from socket import socket, SOCK_STREAM,AF_INET server = socket(AF_INET, SOCK_STREAM) server.bind(('0.0.0.0', 443)) server.listen(1) while True: conn, addr = server.accept()
48. File descriptors, PyCon US 2016 49 socket client from socket import (socket, SOCK_STREAM, AF_INET, AF_INET6) cl4 = socket(AF_INET, SOCK_STREAM) cl4.connect(('104.130.43.121', 443)) cl6 = socket(AF_INET6, SOCK_STREAM) cl6.connect(('2001:4802:7901:0:e60a:1375:0:5', 443)) from socket import (socket, SOCK_STREAM, AF_INET, AF_INET6) cl4 = socket(AF_INET, SOCK_STREAM) cl4.connect(('104.130.43.121', 443)) cl6 = socket(AF_INET6, SOCK_STREAM) cl6.connect(('2001:4802:7901:0:e60a:1375:0:5', 443))
49. Unix Sockets (Unix Domain Sockets / Local Sockets)
50. File descriptors, PyCon US 2016 51
51. File descriptors, PyCon US 2016 52
52. File descriptors, PyCon US 2016 53 Unix sockets from socket import (socket, socketpair, SOCK_STREAM, AF_UNIX) server = socket(AF_UNIX, SOCK_STREAM) server.bind('/path/to/file') client = socket(AF_UNIX, SOCK_STREAM) client.connect('/path/to/file') a, b = socketpair() from socket import (socket, socketpair, SOCK_STREAM, AF_UNIX) server = socket(AF_UNIX, SOCK_STREAM) server.bind('/path/to/file') client = socket(AF_UNIX, SOCK_STREAM) client.connect('/path/to/file') a, b = socketpair()
53. File descriptors, PyCon US 2016 54 peer credentials import socket, struct def getpeercred(sock): fmt = "iII" size = struct.calcsize(fmt) raw = sock.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, size) pid, uid, gid = struct.unpack(fmt, raw) return pid, uid, gid >>> getpeercred(uds) (31362, 0, 0) import socket, struct def getpeercred(sock): fmt = "iII" size = struct.calcsize(fmt) raw = sock.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, size) pid, uid, gid = struct.unpack(fmt, raw) return pid, uid, gid >>> getpeercred(uds) (31362, 0, 0)
54. File descriptors, PyCon US 2016 55 peer security context import socket SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31) def getpeersec(sock): raw = sock.getsockopt(socket.SOL_SOCKET, SO_PEERSEC, 256) return raw.rstrip(b'\x00').decode('utf-8') >>> getpeersec(uds) 'system_u:system_r:svirt_lxc_net_t:s0:c560,c872' import socket SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31) def getpeersec(sock): raw = sock.getsockopt(socket.SOL_SOCKET, SO_PEERSEC, 256) return raw.rstrip(b'\x00').decode('utf-8') >>> getpeersec(uds) 'system_u:system_r:svirt_lxc_net_t:s0:c560,c872'
55. File descriptors, PyCon US 2016 56 Unix sockets and containers • Unix sockets work between containers • SELinux! • Kernel translates PID name space • pid 31362 • Multi-Category Security (MCS) Separation • system_u:system_r:svirt_lxc_net_t:s0:c560,c872
56. File descriptors, PyCon US 2016 57 Docker # cat /proc/31336/cgroup 7:pids:/system.slice/docker-de20f4cb....scope # docker inspect de20f4cb... ... "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c560,c872", ... "Config": { "Labels": { "io.kubernetes.container.name": "custodia-pwmgr", "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz", "io.kubernetes.pod.namespace": "default", ... }} # cat /proc/31336/cgroup 7:pids:/system.slice/docker-de20f4cb....scope # docker inspect de20f4cb... ... "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c560,c872", ... "Config": { "Labels": { "io.kubernetes.container.name": "custodia-pwmgr", "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz", "io.kubernetes.pod.namespace": "default", ... }}
57. File descriptors, PyCon US 2016 58 sending file descriptors # sender with open(filename) as f: send_fds(sender, b'\x01', [f.fileno()]) # receiver msg, fds = recv_fds(rcv, msglen=1, maxfds=1) with open(fds[0]) as f: f.read() # sender with open(filename) as f: send_fds(sender, b'\x01', [f.fileno()]) # receiver msg, fds = recv_fds(rcv, msglen=1, maxfds=1) with open(fds[0]) as f: f.read()
58. File descriptors, PyCon US 2016 59 seccomp sandboxing syscall filter OpenSSH, Tor, Firefox media plugins, Google Chrome sandbox disk disk network network broker broker • read, write, close • open, socket, unlink • fork, exec
59. Beyond
60. File descriptors, PyCon US 2016 61 memory mapped I/0 • mmap • numpy.memmap
61. File descriptors, PyCon US 2016 62 memory fd • temporary file in memory • file sealing
62. File descriptors, PyCon US 2016 63 Efficient I/O with zero-copy • sendfile() • copy_file_range() • splice() / vmsplice() / tee() • AF_KTLS (Kernel space TLS)
63. File descriptors, PyCon US 2016 64 event-driven I/O • select • poll • epoll
64. File descriptors, PyCon US 2016 65 Open Space 4pm, room C120+C121 @ChristianHeimes / christian@python.org
65. THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews
66. Images sources • https://commons.wikimedia.org/wiki/File:Washington,_D.C._Miss_Helen_ Ringwald_works_with_the_pneumatic_tubes.jpg ( • https://commons.wikimedia.org/wiki/File:Rohrpostbuechse.jpg (CC BY- SA 3.0) • https://commons.wikimedia.org/wiki/File:Datasette_c2n.jpg (CC BY-SA 3.0)