File descriptors, Unix sockets and other POSIX magic PyCon US 2016 Christian Heimes [email protected] @ChristianHeimes

File descriptors, PyCon US 2016 2 Who am I • from Hamburg / Germany • Python core contributor since 2008 • PEP 379, 452, 456 • ssl, hashlib • Python Security Response Team • I put bytes and b'' into Python 2.6

File descriptors, PyCon US 2016 3 Professional Life • Senior Software Engineer with Red Hat • Security Engineering & Identity Management • FreeIPA IdM • Dogtag PKI

File descriptors, PyCon US 2016 4 Custodia Secrets for Containers

File descriptors, PyCon US 2016 5 Recipes Concepts & Tools Linux / Unix only Python 3

File descriptors, PyCon US 2016 6 Agenda • File descriptor • Operating System 101 • File descriptors and processes • Networking and sockets • Unix sockets, containers and sandboxing • Bonus track

File descriptors, PyCon US 2016 7 Simplifications ahead

Introducing file descriptors

File descriptors, PyCon US 2016 9 Everything is a file

File descriptors, PyCon US 2016 10 Every I/O operating uses a file descriptor

File descriptors, PyCon US 2016 11 File descriptors are used for • reading / writing files • directories • devices • inter-process communication • network communication • I/O multiplexing • file system monitoring • …

File descriptors, PyCon US 2016 12

File descriptors, PyCon US 2016 13 Standard numbers • 0: standard input (sys.stdin) • 1: standard output (sys.stdout) • 2: standard error output (sys.stderr) • -1: error (Python raises an exception)

File descriptors, PyCon US 2016 14 Simple example with open('example.txt') as f: print(f.read()) with open('example.txt') as f: print(f.read())

File descriptors, PyCon US 2016 15 Reference file by descriptor with open('example.txt') as f: stat = os.stat(f.fileno()) os.chmod(f.fileno(), 0o640) with open('example.txt') as f: stat = os.stat(f.fileno()) os.chmod(f.fileno(), 0o640)

File descriptors, PyCon US 2016 16 Directory file descriptor flags = os.O_RDONLY | os.O_DIRECTORY etcdir = os.open('/etc', flags) # read /etc/passwd stat = os.stat('passwd', dir_fd=etcdir)) os.close(etcdir) flags = os.O_RDONLY | os.O_DIRECTORY etcdir = os.open('/etc', flags) # read /etc/passwd stat = os.stat('passwd', dir_fd=etcdir)) os.close(etcdir)

File descriptors, PyCon US 2016 17 Control hardware (silence please)

File descriptors, PyCon US 2016 18

File descriptors, PyCon US 2016 19 Open CD drive with Python import os, fcntl CDROMEJECT = 0x5309 # Linux specific! fd = os.open('/dev/cdrom', os.O_RDONLY | os.O_NONBLOCK) fcntl.ioctl(fd, CDROMEJECT, 0) os.close(fd) import os, fcntl CDROMEJECT = 0x5309 # Linux specific! fd = os.open('/dev/cdrom', os.O_RDONLY | os.O_NONBLOCK) fcntl.ioctl(fd, CDROMEJECT, 0) os.close(fd)

Operating System 101

File descriptors, PyCon US 2016 21 Dark Ages application application application application disk disk memory memory network network

File descriptors, PyCon US 2016 22 Modern Operating Systems application application application application disk disk memory memory network network Kernel Kernel

File descriptors, PyCon US 2016 23 syscall

File descriptors, PyCon US 2016 24 Simple example # read.py with open('example.txt') as f: print(f.read()) # read.py with open('example.txt') as f: print(f.read())

File descriptors, PyCon US 2016 25 Simple example $ strace ./read.py open("example.txt", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0664, st_size=19, …}) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "Python is awesome!\n", 20) = 19 read(3, "", 1) = 0 write(1, "Python is awesome!\n", 19) = 19 write(1, "\n", 1) = 1 close(3) = 0 $ strace ./read.py open("example.txt", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0664, st_size=19, …}) = 0 lseek(3, 0, SEEK_CUR) = 0 read(3, "Python is awesome!\n", 20) = 19 read(3, "", 1) = 0 write(1, "Python is awesome!\n", 19) = 19 write(1, "\n", 1) = 1 close(3) = 0

File descriptors, PyCon US 2016 26 open file table (global)

File descriptors, PyCon US 2016 27 file descriptor table (process)

File descriptors, PyCon US 2016 28 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 open file table entry 23 ... ... ... ... process 2 file descriptor table open 'example.txt'

File descriptors, PyCon US 2016 29 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table open 'example.txt' a second time

File descriptors, PyCon US 2016 30 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table duplicate file descriptor 4

File descriptors, PyCon US 2016 31 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table ... ... entry 117 ... ... process 2 file descriptor table rename and close

File descriptors, PyCon US 2016 32 open file / file descriptor tables example.txt example.txt process 1 file descriptor table fd 3 fd 4 fd 5 open file table entry 23 ... entry 117 ... ... process 2 file descriptor table fd 7 send file descriptor to another process

File descriptors, PyCon US 2016 33 process and global state file descriptor table maps to open file table cloexec flags open file table position mode owner locks credentials reference count ...

File descriptors, PyCon US 2016 34

File descriptors, PyCon US 2016 35 Spawn a new process fork & exec

File descriptors, PyCon US 2016 36 fork() • creates a clone of the current process • inherits copy of file descriptor table • inherits copy of memory • shallow copy with Copy-on-Write

File descriptors, PyCon US 2016 37 fork() example import os f = open('example.txt', mode='rb', buffering=0) pid = os.fork() print("getpid {:>5}, fork {:>5}, fd {}: {}".format( os.getpid(), pid, f.fileno(), f.read(6))) f.close() $ python3 fork.py getpid 17364, fork 17365, fd 3: b'Python' getpid 17365, fork 0, fd 3: b' is aw' import os f = open('example.txt', mode='rb', buffering=0) pid = os.fork() print("getpid {:>5}, fork {:>5}, fd {}: {}".format( os.getpid(), pid, f.fileno(), f.read(6))) f.close() $ python3 fork.py getpid 17364, fork 17365, fd 3: b'Python' getpid 17365, fork 0, fd 3: b' is aw'

File descriptors, PyCon US 2016 38 exec() • replaces current program code • file descriptors are inherited by default • unless cloexec flag is set • potential security risk • PEP 446 -- Make newly created file descriptors non-inheritable (Victor Stinner)

File descriptors, PyCon US 2016 39 Summary • applications must go through the Kernel • syscalls call into kernel space • file descriptors point to global table • global tables contains mode and position • new process with fork() and exec()

subprocess.PIPE $ ls | grep py

File descriptors, PyCon US 2016 41

File descriptors, PyCon US 2016 43 pipe example import os readend, writeend = os.pipe() pid = os.fork() if pid != 0: # parent process os.close(writeend) with open(readend) as f: print(f.read()) else: # child process os.close(readend) os.dup2(writeend, 1) # 1: stdout os.execl('/bin/ls', # programm 'ls', '-l', 'example.txt') # argv 0, 1, 2 import os readend, writeend = os.pipe() pid = os.fork() if pid != 0: # parent process os.close(writeend) with open(readend) as f: print(f.read()) else: # child process os.close(readend) os.dup2(writeend, 1) # 1: stdout os.execl('/bin/ls', # programm 'ls', '-l', 'example.txt') # argv 0, 1, 2 1 2 3 4 6 5 7

network sockets

File descriptors, PyCon US 2016 45

File descriptors, PyCon US 2016 46

File descriptors, PyCon US 2016 47 socket parameters • addressing / routing • IPv4: AF_INET • IPv6: AF_INET6 • flow control • TCP: SOCK_STREAM • UDP: SOCK_DGRAM

File descriptors, PyCon US 2016 48 socket server from socket import socket, SOCK_STREAM,AF_INET server = socket(AF_INET, SOCK_STREAM) server.bind(('0.0.0.0', 443)) server.listen(1) while True: conn, addr = server.accept() from socket import socket, SOCK_STREAM,AF_INET server = socket(AF_INET, SOCK_STREAM) server.bind(('0.0.0.0', 443)) server.listen(1) while True: conn, addr = server.accept()

File descriptors, PyCon US 2016 49 socket client from socket import (socket, SOCK_STREAM, AF_INET, AF_INET6) cl4 = socket(AF_INET, SOCK_STREAM) cl4.connect(('104.130.43.121', 443)) cl6 = socket(AF_INET6, SOCK_STREAM) cl6.connect(('2001:4802:7901:0:e60a:1375:0:5', 443)) from socket import (socket, SOCK_STREAM, AF_INET, AF_INET6) cl4 = socket(AF_INET, SOCK_STREAM) cl4.connect(('104.130.43.121', 443)) cl6 = socket(AF_INET6, SOCK_STREAM) cl6.connect(('2001:4802:7901:0:e60a:1375:0:5', 443))

Unix Sockets (Unix Domain Sockets / Local Sockets)

File descriptors, PyCon US 2016 51

File descriptors, PyCon US 2016 52

File descriptors, PyCon US 2016 53 Unix sockets from socket import (socket, socketpair, SOCK_STREAM, AF_UNIX) server = socket(AF_UNIX, SOCK_STREAM) server.bind('/path/to/file') client = socket(AF_UNIX, SOCK_STREAM) client.connect('/path/to/file') a, b = socketpair() from socket import (socket, socketpair, SOCK_STREAM, AF_UNIX) server = socket(AF_UNIX, SOCK_STREAM) server.bind('/path/to/file') client = socket(AF_UNIX, SOCK_STREAM) client.connect('/path/to/file') a, b = socketpair()

File descriptors, PyCon US 2016 54 peer credentials import socket, struct def getpeercred(sock): fmt = "iII" size = struct.calcsize(fmt) raw = sock.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, size) pid, uid, gid = struct.unpack(fmt, raw) return pid, uid, gid >>> getpeercred(uds) (31362, 0, 0) import socket, struct def getpeercred(sock): fmt = "iII" size = struct.calcsize(fmt) raw = sock.getsockopt(socket.SOL_SOCKET, socket.SO_PEERCRED, size) pid, uid, gid = struct.unpack(fmt, raw) return pid, uid, gid >>> getpeercred(uds) (31362, 0, 0)

File descriptors, PyCon US 2016 55 peer security context import socket SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31) def getpeersec(sock): raw = sock.getsockopt(socket.SOL_SOCKET, SO_PEERSEC, 256) return raw.rstrip(b'\x00').decode('utf-8') >>> getpeersec(uds) 'system_u:system_r:svirt_lxc_net_t:s0:c560,c872' import socket SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31) def getpeersec(sock): raw = sock.getsockopt(socket.SOL_SOCKET, SO_PEERSEC, 256) return raw.rstrip(b'\x00').decode('utf-8') >>> getpeersec(uds) 'system_u:system_r:svirt_lxc_net_t:s0:c560,c872'

File descriptors, PyCon US 2016 56 Unix sockets and containers ● Unix sockets work between containers ● SELinux! ● Kernel translates PID name space • pid 31362 ● Multi-Category Security (MCS) Separation • system_u:system_r:svirt_lxc_net_t:s0:c560,c872

File descriptors, PyCon US 2016 57 Docker # cat /proc/31336/cgroup 7:pids:/system.slice/docker-de20f4cb....scope # docker inspect de20f4cb... ... "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c560,c872", ... "Config": { "Labels": { "io.kubernetes.container.name": "custodia-pwmgr", "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz", "io.kubernetes.pod.namespace": "default", ... }} # cat /proc/31336/cgroup 7:pids:/system.slice/docker-de20f4cb....scope # docker inspect de20f4cb... ... "ProcessLabel": "system_u:system_r:svirt_lxc_net_t:s0:c560,c872", ... "Config": { "Labels": { "io.kubernetes.container.name": "custodia-pwmgr", "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz", "io.kubernetes.pod.namespace": "default", ... }}

File descriptors, PyCon US 2016 58 sending file descriptors # sender with open(filename) as f: send_fds(sender, b'\x01', [f.fileno()]) # receiver msg, fds = recv_fds(rcv, msglen=1, maxfds=1) with open(fds[0]) as f: f.read() # sender with open(filename) as f: send_fds(sender, b'\x01', [f.fileno()]) # receiver msg, fds = recv_fds(rcv, msglen=1, maxfds=1) with open(fds[0]) as f: f.read()

File descriptors, PyCon US 2016 59 seccomp sandboxing syscall filter OpenSSH, Tor, Firefox media plugins, Google Chrome sandbox disk disk network network broker broker ● read, write, close ● open, socket, unlink ● fork, exec

Beyond

File descriptors, PyCon US 2016 61 memory mapped I/0 • mmap • numpy.memmap

File descriptors, PyCon US 2016 62 memory fd • temporary file in memory • file sealing

File descriptors, PyCon US 2016 63 Efficient I/O with zero-copy • sendfile() • copy_file_range() • splice() / vmsplice() / tee() • AF_KTLS (Kernel space TLS)

File descriptors, PyCon US 2016 64 event-driven I/O • select • poll • epoll

File descriptors, PyCon US 2016 65 Open Space 4pm, room C120+C121 @ChristianHeimes / [email protected]

THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews

Images sources ● https://commons.wikimedia.org/wiki/File:Washington,_D.C._Miss_Helen_ Ringwald_works_with_the_pneumatic_tubes.jpg ( ● https://commons.wikimedia.org/wiki/File:Rohrpostbuechse.jpg (CC BY- SA 3.0) ● https://commons.wikimedia.org/wiki/File:Datasette_c2n.jpg (CC BY-SA 3.0)