PyCon 2016, File descriptors, Unix sockets and other POSIX magic

Have you ever wondered how the OS manages open files and network connections, what this 'file descriptor' thing actually is all about, or what's so special about Unix sockets? In my talk I will give you a quick tour into the I/O layer and process model of Unix-like operating systems. You will learn how to securely identify and efficiently share resources between processes.

https://us.pycon.org/2016/schedule/presentation/2019/

Transcript

1. File descriptors, Unix sockets and
   other POSIX magic
   PyCon US 2016
   Christian Heimes
   christian@python.org
   @ChristianHeimes
   

   View Slide

2. File descriptors, PyCon US 2016
   2
   Who am I
   • from Hamburg / Germany
   • Python core contributor since 2008
   • PEP 379, 452, 456
   • ssl, hashlib
   • Python Security Response Team
   • I put bytes and b'' into Python 2.6
   

   View Slide

3. File descriptors, PyCon US 2016
   3
   Professional Life
   • Senior Software Engineer with Red Hat
   • Security Engineering & Identity Management
   • FreeIPA IdM
   • Dogtag PKI
   

   View Slide

4. File descriptors, PyCon US 2016
   4
   Custodia
   Secrets for Containers
   

   View Slide

5. File descriptors, PyCon US 2016
   5
   Recipes
   Concepts & Tools
   Linux / Unix only
   Python 3
   

   View Slide

6. File descriptors, PyCon US 2016
   6
   Agenda
   • File descriptor
   • Operating System 101
   • File descriptors and processes
   • Networking and sockets
   • Unix sockets, containers and sandboxing
   • Bonus track
   

   View Slide

7. File descriptors, PyCon US 2016
   7
   Simplifications ahead
   

   View Slide

8. Introducing
   file descriptors
   

   View Slide

9. File descriptors, PyCon US 2016
   9
   Everything is a file
   

   View Slide

10. File descriptors, PyCon US 2016
    10
    Every I/O operating
    uses a file descriptor
    

    View Slide

11. File descriptors, PyCon US 2016
    11
    File descriptors are used for
    • reading / writing
    files
    • directories
    • devices
    • inter-process
    communication
    • network
    communication
    • I/O multiplexing
    • file system
    monitoring
    • …
    

    View Slide

12. File descriptors, PyCon US 2016
    12
    

    View Slide

13. File descriptors, PyCon US 2016
    13
    Standard numbers
    • 0: standard input (sys.stdin)
    • 1: standard output (sys.stdout)
    • 2: standard error output (sys.stderr)
    • -1: error (Python raises an exception)
    

    View Slide

14. File descriptors, PyCon US 2016
    14
    Simple example
    with open('example.txt') as f:
    print(f.read())
    with open('example.txt') as f:
    print(f.read())
    

    View Slide

15. File descriptors, PyCon US 2016
    15
    Reference file by descriptor
    with open('example.txt') as f:
    stat = os.stat(f.fileno())
    os.chmod(f.fileno(), 0o640)
    with open('example.txt') as f:
    stat = os.stat(f.fileno())
    os.chmod(f.fileno(), 0o640)
    

    View Slide

16. File descriptors, PyCon US 2016
    16
    Directory file descriptor
    flags = os.O_RDONLY | os.O_DIRECTORY
    etcdir = os.open('/etc', flags)
    # read /etc/passwd
    stat = os.stat('passwd', dir_fd=etcdir))
    os.close(etcdir)
    flags = os.O_RDONLY | os.O_DIRECTORY
    etcdir = os.open('/etc', flags)
    # read /etc/passwd
    stat = os.stat('passwd', dir_fd=etcdir))
    os.close(etcdir)
    

    View Slide

17. File descriptors, PyCon US 2016
    17
    Control hardware
    (silence please)
    

    View Slide

18. File descriptors, PyCon US 2016
    18
    

    View Slide

19. File descriptors, PyCon US 2016
    19
    Open CD drive with Python
    import os, fcntl
    CDROMEJECT = 0x5309 # Linux specific!
    fd = os.open('/dev/cdrom',
    os.O_RDONLY | os.O_NONBLOCK)
    fcntl.ioctl(fd, CDROMEJECT, 0)
    os.close(fd)
    import os, fcntl
    CDROMEJECT = 0x5309 # Linux specific!
    fd = os.open('/dev/cdrom',
    os.O_RDONLY | os.O_NONBLOCK)
    fcntl.ioctl(fd, CDROMEJECT, 0)
    os.close(fd)
    

    View Slide

20. Operating System 101
    

    View Slide

21. File descriptors, PyCon US 2016
    21
    Dark Ages
    application
    application
    application
    application
    disk
    disk memory
    memory
    network
    network
    

    View Slide

22. File descriptors, PyCon US 2016
    22
    Modern Operating Systems
    application
    application application
    application
    disk
    disk memory
    memory
    network
    network
    Kernel
    Kernel
    

    View Slide

23. File descriptors, PyCon US 2016
    23
    syscall
    

    View Slide

24. File descriptors, PyCon US 2016
    24
    Simple example
    # read.py
    with open('example.txt') as f:
    print(f.read())
    # read.py
    with open('example.txt') as f:
    print(f.read())
    

    View Slide

25. File descriptors, PyCon US 2016
    25
    Simple example
    $ strace ./read.py
    open("example.txt", O_RDONLY|O_CLOEXEC) = 3
    fstat(3, {st_mode=S_IFREG|0664,
    st_size=19, …}) = 0
    lseek(3, 0, SEEK_CUR) = 0
    read(3, "Python is awesome!\n", 20) = 19
    read(3, "", 1) = 0
    write(1, "Python is awesome!\n", 19) = 19
    write(1, "\n", 1) = 1
    close(3) = 0
    $ strace ./read.py
    open("example.txt", O_RDONLY|O_CLOEXEC) = 3
    fstat(3, {st_mode=S_IFREG|0664,
    st_size=19, …}) = 0
    lseek(3, 0, SEEK_CUR) = 0
    read(3, "Python is awesome!\n", 20) = 19
    read(3, "", 1) = 0
    write(1, "Python is awesome!\n", 19) = 19
    write(1, "\n", 1) = 1
    close(3) = 0
    

    View Slide

26. File descriptors, PyCon US 2016
    26
    open file table
    (global)
    

    View Slide

27. File descriptors, PyCon US 2016
    27
    file descriptor table
    (process)
    

    View Slide

28. File descriptors, PyCon US 2016
    28
    open file / file descriptor tables
    example.txt
    example.txt
    process 1
    file descriptor table
    fd 3
    open file table
    entry 23
    ...
    ...
    ...
    ...
    process 2
    file descriptor table
    open 'example.txt'
    

    View Slide

29. File descriptors, PyCon US 2016
    29
    open file / file descriptor tables
    example.txt
    example.txt
    process 1
    file descriptor table
    fd 3
    fd 4
    open file table
    entry 23
    ...
    entry 117
    ...
    ...
    process 2
    file descriptor table
    open 'example.txt' a second time
    

    View Slide

30. File descriptors, PyCon US 2016
    30
    open file / file descriptor tables
    example.txt
    example.txt
    process 1
    file descriptor table
    fd 3
    fd 4
    fd 5
    open file table
    entry 23
    ...
    entry 117
    ...
    ...
    process 2
    file descriptor table
    duplicate file descriptor 4
    

    View Slide

31. File descriptors, PyCon US 2016
    31
    open file / file descriptor tables
    example.txt
    example.txt
    process 1
    file descriptor table
    fd 3
    fd 4
    fd 5
    open file table
    ...
    ...
    entry 117
    ...
    ...
    process 2
    file descriptor table
    rename and close
    

    View Slide

32. File descriptors, PyCon US 2016
    32
    open file / file descriptor tables
    example.txt
    example.txt
    process 1
    file descriptor table
    fd 3
    fd 4
    fd 5
    open file table
    entry 23
    ...
    entry 117
    ...
    ...
    process 2
    file descriptor table
    fd 7
    send file descriptor to another process
    

    View Slide

33. File descriptors, PyCon US 2016
    33
    process and global state
    file descriptor table
    maps to open file table
    cloexec flags
    open file table
    position
    mode
    owner
    locks
    credentials
    reference count
    ...
    

    View Slide

34. File descriptors, PyCon US 2016
    34
    

    View Slide

35. File descriptors, PyCon US 2016
    35
    Spawn a new process
    fork & exec
    

    View Slide

36. File descriptors, PyCon US 2016
    36
    fork()
    • creates a clone of the current process
    • inherits copy of file descriptor table
    • inherits copy of memory
    • shallow copy with Copy-on-Write
    

    View Slide

37. File descriptors, PyCon US 2016
    37
    fork() example
    import os
    f = open('example.txt', mode='rb', buffering=0)
    pid = os.fork()
    print("getpid {:>5}, fork {:>5}, fd {}: {}".format(
    os.getpid(), pid, f.fileno(), f.read(6)))
    f.close()
    $ python3 fork.py
    getpid 17364, fork 17365, fd 3: b'Python'
    getpid 17365, fork 0, fd 3: b' is aw'
    import os
    f = open('example.txt', mode='rb', buffering=0)
    pid = os.fork()
    print("getpid {:>5}, fork {:>5}, fd {}: {}".format(
    os.getpid(), pid, f.fileno(), f.read(6)))
    f.close()
    $ python3 fork.py
    getpid 17364, fork 17365, fd 3: b'Python'
    getpid 17365, fork 0, fd 3: b' is aw'
    

    View Slide

38. File descriptors, PyCon US 2016
    38
    exec()
    • replaces current program code
    • file descriptors are inherited by default
    • unless cloexec flag is set
    • potential security risk
    • PEP 446 -- Make newly created file descriptors
    non-inheritable (Victor Stinner)
    

    View Slide

39. File descriptors, PyCon US 2016
    39
    Summary
    • applications must go through the Kernel
    • syscalls call into kernel space
    • file descriptors point to global table
    • global tables contains mode and position
    • new process with fork() and exec()
    

    View Slide

40. subprocess.PIPE
    $ ls | grep py
    

    View Slide

41. File descriptors, PyCon US 2016
    41
    

    View Slide

42. File descriptors, PyCon US 2016
    43
    pipe example
    import os
    readend, writeend = os.pipe()
    pid = os.fork()
    if pid != 0: # parent process
    os.close(writeend)
    with open(readend) as f:
    print(f.read())
    else: # child process
    os.close(readend)
    os.dup2(writeend, 1) # 1: stdout
    os.execl('/bin/ls', # programm
    'ls', '-l', 'example.txt') # argv 0, 1, 2
    import os
    readend, writeend = os.pipe()
    pid = os.fork()
    if pid != 0: # parent process
    os.close(writeend)
    with open(readend) as f:
    print(f.read())
    else: # child process
    os.close(readend)
    os.dup2(writeend, 1) # 1: stdout
    os.execl('/bin/ls', # programm
    'ls', '-l', 'example.txt') # argv 0, 1, 2
    1
    2
    3
    4
    6
    5
    7
    

    View Slide

43. network
    sockets
    

    View Slide

44. File descriptors, PyCon US 2016
    45
    

    View Slide

45. File descriptors, PyCon US 2016
    46
    

    View Slide

46. File descriptors, PyCon US 2016
    47
    socket parameters
    • addressing / routing
    • IPv4: AF_INET
    • IPv6: AF_INET6
    • flow control
    • TCP: SOCK_STREAM
    • UDP: SOCK_DGRAM
    

    View Slide

47. File descriptors, PyCon US 2016
    48
    socket server
    from socket import socket,
    SOCK_STREAM,AF_INET
    server = socket(AF_INET, SOCK_STREAM)
    server.bind(('0.0.0.0', 443))
    server.listen(1)
    while True:
    conn, addr = server.accept()
    from socket import socket,
    SOCK_STREAM,AF_INET
    server = socket(AF_INET, SOCK_STREAM)
    server.bind(('0.0.0.0', 443))
    server.listen(1)
    while True:
    conn, addr = server.accept()
    

    View Slide

48. File descriptors, PyCon US 2016
    49
    socket client
    from socket import (socket, SOCK_STREAM,
    AF_INET, AF_INET6)
    cl4 = socket(AF_INET, SOCK_STREAM)
    cl4.connect(('104.130.43.121', 443))
    cl6 = socket(AF_INET6, SOCK_STREAM)
    cl6.connect(('2001:4802:7901:0:e60a:1375:0:5',
    443))
    from socket import (socket, SOCK_STREAM,
    AF_INET, AF_INET6)
    cl4 = socket(AF_INET, SOCK_STREAM)
    cl4.connect(('104.130.43.121', 443))
    cl6 = socket(AF_INET6, SOCK_STREAM)
    cl6.connect(('2001:4802:7901:0:e60a:1375:0:5',
    443))
    

    View Slide

49. Unix Sockets
    (Unix Domain Sockets / Local Sockets)
    

    View Slide

50. File descriptors, PyCon US 2016
    51
    

    View Slide

51. File descriptors, PyCon US 2016
    52
    

    View Slide

52. File descriptors, PyCon US 2016
    53
    Unix sockets
    from socket import (socket, socketpair,
    SOCK_STREAM, AF_UNIX)
    server = socket(AF_UNIX, SOCK_STREAM)
    server.bind('/path/to/file')
    client = socket(AF_UNIX, SOCK_STREAM)
    client.connect('/path/to/file')
    a, b = socketpair()
    from socket import (socket, socketpair,
    SOCK_STREAM, AF_UNIX)
    server = socket(AF_UNIX, SOCK_STREAM)
    server.bind('/path/to/file')
    client = socket(AF_UNIX, SOCK_STREAM)
    client.connect('/path/to/file')
    a, b = socketpair()
    

    View Slide

53. File descriptors, PyCon US 2016
    54
    peer credentials
    import socket, struct
    def getpeercred(sock):
    fmt = "iII"
    size = struct.calcsize(fmt)
    raw = sock.getsockopt(socket.SOL_SOCKET,
    socket.SO_PEERCRED, size)
    pid, uid, gid = struct.unpack(fmt, raw)
    return pid, uid, gid
    >>> getpeercred(uds)
    (31362, 0, 0)
    import socket, struct
    def getpeercred(sock):
    fmt = "iII"
    size = struct.calcsize(fmt)
    raw = sock.getsockopt(socket.SOL_SOCKET,
    socket.SO_PEERCRED, size)
    pid, uid, gid = struct.unpack(fmt, raw)
    return pid, uid, gid
    >>> getpeercred(uds)
    (31362, 0, 0)
    

    View Slide

54. File descriptors, PyCon US 2016
    55
    peer security context
    import socket
    SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31)
    def getpeersec(sock):
    raw = sock.getsockopt(socket.SOL_SOCKET,
    SO_PEERSEC, 256)
    return raw.rstrip(b'\x00').decode('utf-8')
    >>> getpeersec(uds)
    'system_u:system_r:svirt_lxc_net_t:s0:c560,c872'
    import socket
    SO_PEERSEC = getattr(socket, 'SO_PEERSEC', 31)
    def getpeersec(sock):
    raw = sock.getsockopt(socket.SOL_SOCKET,
    SO_PEERSEC, 256)
    return raw.rstrip(b'\x00').decode('utf-8')
    >>> getpeersec(uds)
    'system_u:system_r:svirt_lxc_net_t:s0:c560,c872'
    

    View Slide

55. File descriptors, PyCon US 2016
    56
    Unix sockets and containers
    ●
    Unix sockets work between containers
    ●
    SELinux!
    ●
    Kernel translates PID name space
    • pid 31362
    ●
    Multi-Category Security (MCS) Separation
    • system_u:system_r:svirt_lxc_net_t:s0:c560,c872
    

    View Slide

56. File descriptors, PyCon US 2016
    57
    Docker
    # cat /proc/31336/cgroup
    7:pids:/system.slice/docker-de20f4cb....scope
    # docker inspect de20f4cb...
    ...
    "ProcessLabel":
    "system_u:system_r:svirt_lxc_net_t:s0:c560,c872",
    ...
    "Config": {
    "Labels": {
    "io.kubernetes.container.name": "custodia-pwmgr",
    "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz",
    "io.kubernetes.pod.namespace": "default",
    ...
    }}
    # cat /proc/31336/cgroup
    7:pids:/system.slice/docker-de20f4cb....scope
    # docker inspect de20f4cb...
    ...
    "ProcessLabel":
    "system_u:system_r:svirt_lxc_net_t:s0:c560,c872",
    ...
    "Config": {
    "Labels": {
    "io.kubernetes.container.name": "custodia-pwmgr",
    "io.kubernetes.pod.name": "custodia-pwmgr-tqxtz",
    "io.kubernetes.pod.namespace": "default",
    ...
    }}
    

    View Slide

57. File descriptors, PyCon US 2016
    58
    sending file descriptors
    # sender
    with open(filename) as f:
    send_fds(sender, b'\x01', [f.fileno()])
    # receiver
    msg, fds = recv_fds(rcv, msglen=1, maxfds=1)
    with open(fds[0]) as f:
    f.read()
    # sender
    with open(filename) as f:
    send_fds(sender, b'\x01', [f.fileno()])
    # receiver
    msg, fds = recv_fds(rcv, msglen=1, maxfds=1)
    with open(fds[0]) as f:
    f.read()
    

    View Slide

58. File descriptors, PyCon US 2016
    59
    seccomp sandboxing
    syscall filter
    OpenSSH, Tor, Firefox media plugins, Google Chrome
    sandbox
    disk
    disk network
    network
    broker
    broker
    ●
    read, write, close
    ●
    open, socket, unlink
    ●
    fork, exec
    

    View Slide

59. Beyond
    

    View Slide

60. File descriptors, PyCon US 2016
    61
    memory mapped I/0
    • mmap
    • numpy.memmap
    

    View Slide

61. File descriptors, PyCon US 2016
    62
    memory fd
    • temporary file in memory
    • file sealing
    

    View Slide

62. File descriptors, PyCon US 2016
    63
    Efficient I/O with zero-copy
    • sendfile()
    • copy_file_range()
    • splice() / vmsplice() / tee()
    • AF_KTLS (Kernel space TLS)
    

    View Slide

63. File descriptors, PyCon US 2016
    64
    event-driven I/O
    • select
    • poll
    • epoll
    

    View Slide

64. File descriptors, PyCon US 2016
    65
    Open Space
    4pm, room C120+C121
    @ChristianHeimes / christian@python.org
    

    View Slide

65. THANK YOU
    plus.google.com/+RedHat
    linkedin.com/company/red-hat
    youtube.com/user/RedHatVideos
    facebook.com/redhatinc
    twitter.com/RedHatNews
    

    View Slide

66. Images sources
    ●
    https://commons.wikimedia.org/wiki/File:Washington,_D.C._Miss_Helen_
    Ringwald_works_with_the_pneumatic_tubes.jpg (
    ●
    https://commons.wikimedia.org/wiki/File:Rohrpostbuechse.jpg (CC BY-
    SA 3.0)
    ●
    https://commons.wikimedia.org/wiki/File:Datasette_c2n.jpg (CC BY-SA
    3.0)
    

    View Slide