Paul Kehrer - Reliably Distributing Compiled Modules

Shipping Python libraries is easy! ...until you want to use a C library. How do you easily and reliably deliver software to users when they may not have the libraries you depend on, or even a compiler? How do you handle the significant differences between linux, OS X, Windows, FreeBSD, and other platforms Python runs on?

https://us.pycon.org/2016/schedule/presentation/2145/

Transcript

1. Reliably Distributing Compiled Modules Paul Kehrer (reaperhulk) – PyCon 2016
2. Who am I? • I’m Paul Kehrer • I write software, kiteboard, and bake things • I am an inveterate Simpsons quoter. • @reaperhulk on Twitter • @reaperhulk on Freenode • @reaperhulk on GitHub • …you get the idea
3. What I work on • Projects for Rackspace Managed Security • Python Cryptographic Authority (PyCA) • cryptography • pyOpenSSL • PyNaCl • bcrypt • Frinkiac (but that’s a separate talk)
4. What do we want to do? pip install cryptography
5. What is a compiled module? • Any module that uses ”binary” code
6. What is a compiled module? • Any module that uses “binary” code • Less facetiously, a module that contains some sort of build process that results in architecture and operating system specific output that can be executed or calls into such code even if it does not invoke the compilation itself. • This is a bit of a simplification, but will do for our purposes.
7. Installing… Well, we need a compiler right? apt-get install build-essential …or yum install make automake gcc if RHEL/CentOS
8. pip install cryptography
9. Installing… (.venv)root@pycon2016:~# pip install cryptography ...snip... Installing collected packages: cryptography, idna, pyasn1, six, setuptools, enum34, ipaddress, cffi, pycparser Running setup.py install for cryptography c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ...snip...
10. (.venv)root@pycon2016:~# pip install cryptography Downloading/unpacking cryptography Downloading cryptography-1.3.2.tar.gz (383kB): 383kB downloaded Running setup.py (path:/root/.venv/build/cryptography/setup.py) egg_info for package cryptography no previously-included directories found matching 'docs/_build' warning: no previously-included files matching '*' found under directory 'vectors' Downloading/unpacking idna>=2.0 (from cryptography) Downloading idna-2.1-py2.py3-none-any.whl (54kB): 54kB downloaded Downloading/unpacking pyasn1>=0.1.8 (from cryptography) Downloading pyasn1-0.1.9-py2.py3-none-any.whl Downloading/unpacking six>=1.4.1 (from cryptography) Downloading six-1.10.0-py2.py3-none-any.whl Downloading/unpacking setuptools>=11.3 (from cryptography) Downloading setuptools-21.2.0-py2.py3-none-any.whl (509kB): 509kB downloaded Downloading/unpacking enum34 (from cryptography) Downloading enum34-1.1.6-py2-none-any.whl Downloading/unpacking ipaddress (from cryptography) Downloading ipaddress-1.0.16-py27-none-any.whl Downloading/unpacking cffi>=1.4.1 (from cryptography) Downloading cffi-1.6.0.tar.gz (397kB): 397kB downloaded Running setup.py (path:/root/.venv/build/cffi/setup.py) egg_info for package cffi Downloading/unpacking pycparser (from cffi>=1.4.1->cryptography) Downloading pycparser-2.14.tar.gz (223kB): 223kB downloaded Running setup.py (path:/root/.venv/build/pycparser/setup.py) egg_info for package pycparser warning: no previously-included files matching 'yacctab.*' found under directory 'tests' warning: no previously-included files matching 'lextab.*' found under directory 'tests' warning: no previously-included files matching 'yacctab.*' found under directory 'examples' warning: no previously-included files matching 'lextab.*' found under directory 'examples' Installing collected packages: cryptography, idna, pyasn1, six, setuptools, enum34, ipaddress, cffi, pycparser Running setup.py install for cryptography
11. c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. Traceback (most recent call last): File "<string>", line 1, in <module> File "/root/.venv/build/cryptography/setup.py", line 335, in <module> **keywords_with_side_effects(sys.argv) File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__ self.fetch_build_eggs(attrs.pop('setup_requires')) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs replace_conflicting=True File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match return self.obtain(req, installer) # try and download/install File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain return installer(requirement) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg return cmd.easy_install(req) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 593, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 623, in install_item dists = self.install_eggs(spec, download, tmpdir)
12. File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 809, in install_eggs return self.build_and_install(setup_script, setup_base) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 1015, in build_and_install self.run_setup(setup_script, setup_base, args) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 1003, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux- gnu-gcc' failed with exit status 1 Complete output from command /root/.venv/bin/python -c "import setuptools, tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ag4Mxm-record/install-record.txt --single-version-externally-managed --compile -- install-headers /root/.venv/include/site/python2.7: c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. Traceback (most recent call last): File "<string>", line 1, in <module> File "/root/.venv/build/cryptography/setup.py", line 335, in <module> **keywords_with_side_effects(sys.argv) File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__ self.fetch_build_eggs(attrs.pop('setup_requires')) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs
13. replace_conflicting=True File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match return self.obtain(req, installer) # try and download/install File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain return installer(requirement) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg return cmd.easy_install(req) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 593, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 809, in install_eggs return self.build_and_install(setup_script, setup_base) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1015, in build_and_install self.run_setup(setup_script, setup_base, args) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1003, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- Cleaning up... Command /root/.venv/bin/python -c "import setuptools, tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ag4Mxm- record/install-record.txt --single-version-externally-managed --compile --install-headers /root/.venv/include/site/python2.7 failed with error code 1 in /root/.venv/build/cryptography Storing debug log for failure in /root/.pip/pip.log
14. What went wrong?
15. What went wrong? • No Python.h
16. What went wrong? • No Python.h
17. What went wrong? • No Python.h
18. What went wrong? • No Python.h • apt-get install python-dev
19. How about now? (.venv)root@pycon2016:~# pip install cryptography ...snip... Running setup.py install for cryptography c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory #include <ffi.h> ...snip...
20. yum install libffi-devel (Wait, weren’t we using apt before?)
21. This time… ...snip... generating cffi module 'build/temp.linux-x86_64- 2.7/_openssl.c' building '_openssl' extension x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64- 2.7/_openssl.c -o build/temp.linux-x86_64- 2.7/build/temp.linux-x86_64-2.7/_openssl.o build/temp.linux-x86_64-2.7/_openssl.c:423:30: fatal error: openssl/opensslv.h: No such file or directory #include <openssl/opensslv.h> ^ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ...snip...
22. brew install openssl
23. brew install openssl apt-get install libssl-dev
24. brew install openssl apt-get install libssl-dev yum install openssl-devel
25. brew install openssl apt-get install libssl-dev yum install openssl-devel dnf install openssl-devel
26. One final time… ...snip... Successfully installed cryptography idna pyasn1 six setuptools enum34 ipaddress cffi pycparser
27. The real prerequisites • The abstract requirements consist of • compiler - Typically, but not guaranteed to be, gcc, clang, or msvc • library - If you're linking against one • headers - So the code can know what function signatures are available in the library *Implicit in the following is an assumption that the user has root/admin privileges. If they do not then compiling a module may become extremely difficult.
28. Compiling on OS X • xcode-select --install • Unless the user’s OS X is older than 10.9…
29. Compiling on OS X • xcode-select --install • Unless the user’s OS X is older than 10.9… • If your software depends on libraries outside the OS X default • homebrew • macports
30. Compiling on Windows • Download Visual Studio (on occasion someone who wants to make your life hard might choose mingw) • For Python 2.7 the user will need "Microsoft Visual C++ Compiler for Python 2.7" • Python 3.3 and 3.4 use Visual Studio 2010 • Python 3.5 uses Visual Studio 2015
31. Compiling on Windows • Architecture matters • 32-bit and 64-bit Pythons • VS2010 64-bit compilers aren’t free • No package manager for getting third party libraries
32. Compiling on Linux • On Linux each distribution has its own view of what packages provide the basic compiler tools. • apt-get install build-essential • yum install make automake gcc • …and its own opinion of what packages provide the required headers/libraries.
33. Common Problems • C toolchain lookup paths
34. Common Problems • C toolchain lookup paths • Unsupported library versions
35. Common Problems • C toolchain lookup paths • Unsupported library versions • Windows CRT/compiler restrictions
36. Common Problems • C toolchain lookup paths • Unsupported library versions • Windows CRT/compiler restrictions • People do terrible things to their computers • Install software from random tarballs or installers that end up in /usr/local, but leave conflicting versions from elsewhere in their paths. • Edit /etc/ld.so.confand screw up library path searches. • Add LD_LIBRARY_PATH or DYLD_LIBRARY_PATH env variables to their .bash_profile without understanding the ramifications
37. This is not reasonable
38. Let’s avoid this
39. Let’s avoid this (Apparently 130x better)
40. Wheels
41. Wheels • PEP 427/491/513 (and probably more) • Provides a way to distribute Python artifacts. • Superior in a variety of ways to eggs and sdists • Install packages with binary dependencies sans root • pip can both produce and install them • Tags are used to denote what a given wheel is compatible with: • e.g. cryptography-1.3.2-cp27-cp27mu- macosx_10_10_x86_64.whl
42. Look deep within yourself
43. Questions for your project • What platforms do you care about?
44. Questions for your project • What platforms do you care about? • What versions of Python do you care about?
45. Questions for your project • What platforms do you care about? • What versions of Python do you care about? • Can you rely on the library you need to be present on all platforms?
46. What’s Required (OS X) • An OS X machine • Any recent release will do (but why aren’t you on El Capitan?)
47. What’s Required (OS X) • An OS X machine • Any recent release will do, but why aren’t you on El Capitan? • Challenges • OS X SDK versions • Universal wheels • UCS2/UCS4
48. Handling OS X SDK versions • Build against python.org Pythons • Not pyenvor system python • pip’s handling of OS X SDK restrictions has changed in the recent past
49. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64
50. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64 • python.org pythons are universal
51. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64 • python.org pythons are universal • Libraries you link against must also be universal • lipo -info /path/to/lib
52. UCS2/UCS4 Symbol not found: _PyUnicodeUCS2_AsASCIIString
53. UCS2/UCS4 • Python has two ways to compile unicode support in 2.x (and < 3.3, but ignore those releases) • UCS2 and UCS4. • The resulting Python has a different ABI so we need different wheels. • pip/wheel did not handle this until recently (wheel 0.27.0 and pip 8.1) • Use a pyenv 2.7 to handle this case.
54. What’s Required (OS X) • Pythons from python.org • One pyenv Python 2.7 • Patience to understand all the permutations
55. What’s Required (Windows) • Make life simple by building a separate VM for 32- bit vs 64-bit Python. • This way Python can just live in C:\Python35, C:\Python27, et cetera • Install Microsoft Visual C++ Compiler for Python 2.7 • Install Visual Studio 2015 Express for Python 3.5 • Install Visual Studio 2010 for Python 3.3 and 3.4 • You’re on your own for libraries L
56. What’s Required (manylinux1) docker run -it --rm -v /path/to/python/code:/io quay.io/pypa/manylinux1_x86_64 /bin/bash • https://github.com/pypa/manylinux
57. What’s Required Automation
58. Caveats • Users need the right versions of pip • 8.1+ if you want to avoid quite a few issues
59. Caveats • Users need the right versions of pip • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it
60. Caveats • Users need the right versions of pip • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it • Unusual architectures aren’t supported by wheels on PyPI
61. Caveats • Users need the right versions of pip/wheel • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it • Unusual architectures aren’t supported by wheels on PyPI • Download size and memory footprint increase
62. Good luck Go forth and build wheels!
63. Thank you
64. Resources • manylinux1: https://github.com/pypa/manylinux • Wheel builders mailing list: https://mail.python.org/mailman/listinfo/wheel- builders • Blog: https://langui.sh • Twitter: https://twitter.com/reaperhulk • Simpsons related technology: https://frinkiac.com