Subprocess to FFI by Christine Spang

Transcript

1. Subprocess to FFI Memory, Performance, and Why You Shouldn’t Shell Out!
2. Christine Spang
3. Inbox is a startup in San Francisco that I co-founded funded by top VCs building a new email platform.
4. We ❤ Python
5. CPython 2.7 on Linux EXTERNAL BINARY C LIBRARY VS.
6. Why you shouldn’t $hell out! And sometimes why you should...
7. iconv
8. iconv convert-utf8
9. SUBPROCESS ( USUALLY PART OF LARGER SYSTEM. )
10. photo credit: http://flic.kr/p/hBMUP5
11. subprocess.check_call( [‘iconv’, ‘-f’, encoding, ‘-t’, ‘utf-8’, filename]) Let’s go source diving...
12. self.pid = os.fork() >>> from inspect import getsourcefile >>> import subprocess >>> getsourcefile(subprocess) ‘/usr/lib/python2.7/subprocess.py’
13. photo credit: http://flic.kr/p/hBMUP5
14. the API between a userspace application (like convert-utf8) and the operating system’s kernel a system call (syscall)
15. fork()
17. waitpid() ✓ convert- utf8 execvp() parent process fork() child process fork() _exit() $ iconv -f encoding -t utf-8 filename checks exit code and raises exception if child process failed convert-utf8 subprocess.check_call()
18. creates the child process by making a copy of the parent process. The child process inherits the parent’s memory pages: the program data is shared between the two processes, and the data, heap, and stack are given to the child copy-on-write. fork()
19. -> TWO PROCESSES fork() USING MORE TOTAL MEMORY THAN THE ENTIRE SYSTEM HAS ALLOCATED COPY - ON - WRITE
20. photo credit: http://flic.kr/p/7eypMU
21. OVERCOMMIT When overcommit_memory flag is 0, the kernel attempts to estimate the amount of free memory left when userspace requests more memory. — docs from Kernel.org
22. OVERCOMMIT OOM Kill SOMETIMES MORE COMPLICATED
23. simple and easy flexible enough throws native Python exceptions why shell out? using subprocess module photo credit: http://flic.kr/p/6UTVj7
24. the dangers... of forking your process significant overhead (fork, file I/O vs memory) limited API parsing stdout/stderr flushing, buffering, deadlocks issues w/pipes
25. DO IT ANYWAY ( USUALLY )
26. Wrapping C libraries from Python Another option:
27. a way to call functions and use data structures provided by one language in another language. FFI: foreign function interface
28. The usual suspects
29. C extension write lots of C with Python’s C API The usual suspects
30. C extension write lots of C with Python’s C API The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky
31. C extension write lots of C with Python’s C API The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky Cython Python/C hybrid language, more for optimizing speed than wrapping
32. C extension write lots of C with Python’s C API The usual suspects ctypes standard library (wraps libffi), no C compiler needed, but tedious and clunky Cython Python/C hybrid language, more for optimizing speed than wrapping CFFI written to address ctypes shortcomings, ABI or API (needs compiler) interface
33. Wrapping libiconv: C extension
34. Wrapping libiconv: CFFI
35. Write less C. Python C extension: 252 lines of C CFFI wrapper: 120 lines of Python/C (~40 lines actually interface with C) (you are not a superhuman)
36. What did we learn?
37. $HELLING OUT IS EXPENSIVE IN BOTH MEMORY AND COMPUTATION
38. TO MAKE IT FASTER & HAVE MORE CONTROL... WRAP YOUR LIBRARIES WITH CFFI (usually)
39. KNOW YOUR OS EVEN WHEN USING A HIGH-LEVEL LANGUAGE…
40. Say hi ! follow @spang spang@inboxapp.com all examples on GitHub Like this? Come work at Inbox! :)