Subprocess to FFI by Christine Spang

Subprocess to FFI by Christine Spang

D21717ea76044d31115c573d368e6ff4?s=128

PyCon 2014

April 12, 2014
Tweet

Transcript

  1. Subprocess to FFI Memory, Performance, and Why You Shouldn’t Shell

    Out!
  2. Christine Spang

  3. Inbox is a startup in San Francisco that I co-founded

    funded by top VCs building a new email platform.
  4. We ❤ Python

  5. CPython 2.7 on Linux EXTERNAL BINARY C LIBRARY VS.

  6. Why you shouldn’t $hell out! And sometimes why you should...

  7. iconv

  8. iconv convert-utf8

  9. SUBPROCESS ( USUALLY PART OF LARGER SYSTEM. )

  10. photo credit: http://flic.kr/p/hBMUP5

  11. subprocess.check_call( [‘iconv’, ‘-f’, encoding, ‘-t’, ‘utf-8’, filename]) Let’s go source

    diving...
  12. self.pid = os.fork() >>> from inspect import getsourcefile >>> import

    subprocess >>> getsourcefile(subprocess) ‘/usr/lib/python2.7/subprocess.py’
  13. photo credit: http://flic.kr/p/hBMUP5

  14. the API between a userspace application (like convert-utf8) and the

    operating system’s kernel a system call (syscall)
  15. fork()

  16. None
  17. waitpid() ✓ convert- utf8 execvp() parent process fork() child process

    fork() _exit() $ iconv -f encoding -t utf-8 filename checks exit code and raises exception if child process failed convert-utf8 subprocess.check_call()
  18. creates the child process by making a copy of the

    parent process. The child process inherits the parent’s memory pages: the program data is shared between the two processes, and the data, heap, and stack are given to the child copy-on-write. fork()
  19. -> TWO PROCESSES fork() USING MORE TOTAL MEMORY THAN THE

    ENTIRE SYSTEM HAS ALLOCATED COPY - ON - WRITE
  20. photo credit: http://flic.kr/p/7eypMU

  21. OVERCOMMIT When overcommit_memory flag is 0, the kernel attempts to

    estimate the amount of free memory left when userspace requests more memory. — docs from Kernel.org
  22. OVERCOMMIT OOM Kill SOMETIMES MORE COMPLICATED

  23. simple and easy flexible enough throws native Python exceptions why

    shell out? using subprocess module photo credit: http://flic.kr/p/6UTVj7
  24. the dangers... of forking your process significant overhead (fork, file

    I/O vs memory) limited API parsing stdout/stderr flushing, buffering, deadlocks issues w/pipes
  25. DO IT ANYWAY ( USUALLY )

  26. Wrapping C libraries from Python Another option:

  27. a way to call functions and use data structures provided

    by one language in another language. FFI: foreign function interface
  28. The usual suspects

  29. C extension write lots of C with Python’s C API

    The usual suspects
  30. C extension write lots of C with Python’s C API

    The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky
  31. C extension write lots of C with Python’s C API

    The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky Cython Python/C hybrid language, more for optimizing speed than wrapping
  32. C extension write lots of C with Python’s C API

    The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky Cython Python/C hybrid language, more for optimizing speed than wrapping CFFI written to address ctypes shortcomings, ABI or API (needs compiler) interface
  33. Wrapping libiconv: C extension

  34. Wrapping libiconv: CFFI

  35. Write less C. Python C extension: 252 lines of C

    CFFI wrapper: 120 lines of Python/C (~40 lines actually interface with C) (you are not a superhuman)
  36. What did we learn?

  37. $HELLING OUT IS EXPENSIVE IN BOTH MEMORY AND COMPUTATION

  38. TO MAKE IT FASTER & HAVE MORE CONTROL... WRAP YOUR

    LIBRARIES WITH CFFI (usually)
  39. KNOW YOUR OS EVEN WHEN USING A HIGH-LEVEL LANGUAGE…

  40. Say hi ! follow @spang spang@inboxapp.com all examples on GitHub

    Like this? Come work at Inbox! :)