Reliably Distributing Binary Modules by Paul Kehrer

Shipping Python libraries is easy! ...until you want to use a C library. How do you easily and reliably deliver software to users when they may not have the libraries you depend on, or even a compiler? How do you handle the significant differences between linux, OS X, Windows, FreeBSD, and other platforms Python runs on?

We'll walk through:

The requirements for building C modules in Python.
Why those requirements contribute to bad UX and create barriers to adoption.
How wheels can be used to surmount many of these challenges.
What kind of infrastructure is required to build these wheels.
The advantages and potential disadvantages of binary distribution.

Transcript

1. Reliably Distributing Compiled Modules Paul Kehrer (reaperhulk) – PyCon ZA 2016
2. Reliably Distributing Compiled Modules Paul Kehrer (reaperhulk) – PyCon ZA 2016 Contains some recycled material
3. Who am I? • I’m Paul Kehrer • I write software, kiteboard, and bake things. • I am an inveterate Simpsons quoter. • @reaperhulk on Twitter • @reaperhulk on Freenode • @reaperhulk on GitHub • …you get the idea
4. What I work on • Projects for Rackspace Managed Security • Python Cryptographic Authority (PyCA) • cryptography • pyOpenSSL • PyNaCl • bcrypt • Frinkiac & Morbotron (but that’s a separate talk)
5. What do we want to do? pip install cryptography
6. What is a compiled module? • Any module that uses ”binary” code
7. What is a compiled module? • Any module that uses “binary” code • Less facetiously, a module that contains some sort of build process that results in architecture and operating system specific output that can be executed or calls into such code even if it does not invoke the compilation itself. • This is a bit of a simplification, but will do for our purposes.
8. Installing… Well, we need a compiler right? apt-get install build-essential …or yum install make gcc if RHEL/CentOS
9. pip install cryptography
10. Installing… (.venv)root@pycon2016:~# pip install cryptography ...snip... Installing collected packages: cryptography, idna, pyasn1, six, setuptools, enum34, ipaddress, cffi, pycparser Running setup.py install for cryptography c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ...snip...
11. (.venv)root@pycon2016:~# pip install cryptography Downloading/unpacking cryptography Downloading cryptography-1.3.2.tar.gz (383kB): 383kB downloaded Running setup.py (path:/root/.venv/build/cryptography/setup.py) egg_info for package cryptography no previously-included directories found matching 'docs/_build' warning: no previously-included files matching '*' found under directory 'vectors' Downloading/unpacking idna>=2.0 (from cryptography) Downloading idna-2.1-py2.py3-none-any.whl (54kB): 54kB downloaded Downloading/unpacking pyasn1>=0.1.8 (from cryptography) Downloading pyasn1-0.1.9-py2.py3-none-any.whl Downloading/unpacking six>=1.4.1 (from cryptography) Downloading six-1.10.0-py2.py3-none-any.whl Downloading/unpacking setuptools>=11.3 (from cryptography) Downloading setuptools-21.2.0-py2.py3-none-any.whl (509kB): 509kB downloaded Downloading/unpacking enum34 (from cryptography) Downloading enum34-1.1.6-py2-none-any.whl Downloading/unpacking ipaddress (from cryptography) Downloading ipaddress-1.0.16-py27-none-any.whl Downloading/unpacking cffi>=1.4.1 (from cryptography) Downloading cffi-1.6.0.tar.gz (397kB): 397kB downloaded Running setup.py (path:/root/.venv/build/cffi/setup.py) egg_info for package cffi Downloading/unpacking pycparser (from cffi>=1.4.1->cryptography) Downloading pycparser-2.14.tar.gz (223kB): 223kB downloaded Running setup.py (path:/root/.venv/build/pycparser/setup.py) egg_info for package pycparser warning: no previously-included files matching 'yacctab.*' found under directory 'tests' warning: no previously-included files matching 'lextab.*' found under directory 'tests' warning: no previously-included files matching 'yacctab.*' found under directory 'examples' warning: no previously-included files matching 'lextab.*' found under directory 'examples' Installing collected packages: cryptography, idna, pyasn1, six, setuptools, enum34, ipaddress, cffi, pycparser Running setup.py install for cryptography
12. c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. Traceback (most recent call last): File "<string>", line 1, in <module> File "/root/.venv/build/cryptography/setup.py", line 335, in <module> **keywords_with_side_effects(sys.argv) File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__ self.fetch_build_eggs(attrs.pop('setup_requires')) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs replace_conflicting=True File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match return self.obtain(req, installer) # try and download/install File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain return installer(requirement) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg return cmd.easy_install(req) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 593, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 623, in install_item dists = self.install_eggs(spec, download, tmpdir)
13. File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 809, in install_eggs return self.build_and_install(setup_script, setup_base) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 1015, in build_and_install self.run_setup(setup_script, setup_base, args) File "/root/.venv/local/lib/python2.7/site- packages/setuptools/command/easy_install.py", line 1003, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux- gnu-gcc' failed with exit status 1 Complete output from command /root/.venv/bin/python -c "import setuptools, tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ag4Mxm-record/install-record.txt --single-version-externally-managed --compile -- install-headers /root/.venv/include/site/python2.7: c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. Traceback (most recent call last): File "<string>", line 1, in <module> File "/root/.venv/build/cryptography/setup.py", line 335, in <module> **keywords_with_side_effects(sys.argv) File "/usr/lib/python2.7/distutils/core.py", line 111, in setup _setup_distribution = dist = klass(attrs) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in __init__ self.fetch_build_eggs(attrs.pop('setup_requires')) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in fetch_build_eggs
14. replace_conflicting=True File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve dist = best[req.key] = env.best_match(req, ws, installer) File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match return self.obtain(req, installer) # try and download/install File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain return installer(requirement) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in fetch_build_egg return cmd.easy_install(req) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 593, in easy_install return self.install_item(spec, dist.location, tmpdir, deps) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623, in install_item dists = self.install_eggs(spec, download, tmpdir) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 809, in install_eggs return self.build_and_install(setup_script, setup_base) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1015, in build_and_install self.run_setup(setup_script, setup_base, args) File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1003, in run_setup raise DistutilsError("Setup script exited with %s" % (v.args[0],)) distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ---------------------------------------- Cleaning up... Command /root/.venv/bin/python -c "import setuptools, tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ag4Mxm- record/install-record.txt --single-version-externally-managed --compile --install-headers /root/.venv/include/site/python2.7 failed with error code 1 in /root/.venv/build/cryptography Storing debug log for failure in /root/.pip/pip.log
15. What went wrong?
16. What went wrong? • No Python.h
17. What went wrong? • No Python.h • apt-get install python-dev
18. How about now? (.venv)root@pycon2016:~# pip install cryptography ...snip... Running setup.py install for cryptography c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory #include <ffi.h> ...snip...
19. yum install libffi-devel (Wait, weren’t we using apt before?)
20. This time… ...snip... generating cffi module 'build/temp.linux-x86_64- 2.7/_openssl.c' building '_openssl' extension x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c build/temp.linux-x86_64- 2.7/_openssl.c -o build/temp.linux-x86_64- 2.7/build/temp.linux-x86_64-2.7/_openssl.o build/temp.linux-x86_64-2.7/_openssl.c:423:30: fatal error: openssl/opensslv.h: No such file or directory #include <openssl/opensslv.h> ^ compilation terminated. error: command 'x86_64-linux-gnu-gcc' failed with exit status 1 ...snip...
21. brew install openssl
22. brew install openssl apt-get install libssl-dev
23. brew install openssl apt-get install libssl-dev yum install openssl-devel
24. brew install openssl apt-get install libssl-dev yum install openssl-devel dnf install openssl-devel
25. One final time… ...snip... Successfully installed cryptography idna pyasn1 six setuptools enum34 ipaddress cffi pycparser
26. The real prerequisites • The abstract requirements consist of • compiler - Typically, but not guaranteed to be, gcc, clang, or msvc • library - If you're linking against one • headers - So the code can know what function signatures are available in the library *Implicit in the following is an assumption that the user has root/admin privileges. If they do not then compiling a module may become extremely difficult.
27. Compiling on OS X • xcode-select --install • Unless the user’s OS X is older than 10.9…
28. Compiling on OS X • xcode-select --install • Unless the user’s OS X is older than 10.9… • If your software depends on libraries outside the OS X default • homebrew • macports
29. Compiling on Windows • Download Visual Studio (on occasion someone who wants to make your life hard might choose mingw) • For Python 2.7 the user will need "Microsoft Visual C++ Compiler for Python 2.7" • Python 3.3 and 3.4 use Visual Studio 2010 • Python 3.5 uses Visual Studio 2015
30. Compiling on Windows • Architecture matters • 32-bit and 64-bit Pythons • VS2010 64-bit compilers don’t come with Express • No package manager for getting third party libraries
31. Compiling on Linux • On Linux each distribution has its own view of what packages provide the basic compiler tools. • apt-get install build-essential • yum install make automake gcc • …and its own opinion of what packages provide the required headers/libraries.
32. Common Problems • C toolchain lookup paths
33. Common Problems • C toolchain lookup paths • Unsupported library versions
34. Common Problems • C toolchain lookup paths • Unsupported library versions • Windows CRT/compiler restrictions
35. Common Problems • C toolchain lookup paths • Unsupported library versions • Windows CRT/compiler restrictions • People do terrible things to their computers • Install software from random tarballs or installers that end up in /usr/local, but leave conflicting versions from elsewhere in their paths. • Edit /etc/ld.so.conf and screw up library path searches. • Add LD_LIBRARY_PATH or DYLD_LIBRARY_PATH env variables to their .bash_profile without understanding the ramifications
36. This is not reasonable
37. Let’s avoid this
38. Let’s avoid this (Apparently 130x better)
39. Wheels
40. Wheels • PEP 427/491/513 (and probably more) • Provides a way to distribute Python artifacts. • Superior in a variety of ways to eggs and sdists • Install packages with binary dependencies sans root • pip can both produce and install them • Tags are used to denote what a given wheel is compatible with: • e.g. cryptography-1.3.2-cp27-cp27mu- macosx_10_10_x86_64.whl
41. Look deep within yourself
42. Questions for your project • What platforms do you care about?
43. Questions for your project • What platforms do you care about? • What versions of Python do you care about?
44. Questions for your project • What platforms do you care about? • What versions of Python do you care about? • Can you rely on the library you need to be present on all platforms?
45. What’s Required (OS X) • An OS X machine • Any recent release will do, but you should really be on El Capitan or Sierra.
46. What’s Required (OS X) • An OS X machine • Any recent release will do, but you should really be on El Capitan or Sierra. • Challenges • OS X SDK versions • Universal wheels • UCS2/UCS4
47. Handling OS X SDK versions • Build against python.org Pythons • Not pyenv or system python • pip’s handling of OS X SDK restrictions has changed in the recent past
48. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64
49. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64 • python.org pythons are universal
50. Universal Support • Universal wheels are denoted with the intel platform tag • As opposed to i686 or x86_64 • python.org pythons are universal • Libraries you link against must also be universal • lipo -info /path/to/lib
51. UCS2/UCS4 Symbol not found: _PyUnicodeUCS2_AsASCIIString
52. UCS2/UCS4 • Python has two ways to compile unicode support in 2.x (and < 3.3, but ignore those releases) • UCS2 and UCS4. • The resulting Python has a different ABI so we need different wheels. • pip/wheel did not handle this until recently (wheel 0.27.0 and pip 8.1) • Use a pyenv 2.7 to handle this case.
53. What’s Required (OS X) • Pythons from python.org • One pyenv Python 2.7 • Patience to understand all the permutations
54. What’s Required (Windows) • Make life simple by building a separate VM for 32- bit vs 64-bit Python. • This way Python can just live in C:\Python35, C:\Python27, et cetera • Install Microsoft Visual C++ Compiler for Python 2.7 • Install Visual Studio 2015 Express for Python 3.5 • Install Visual Studio 2010 for Python 3.3 and 3.4 • You’re on your own for libraries L
55. What’s Required (manylinux1) docker run -it --rm -v /path/to/python/code:/io quay.io/pypa/manylinux1_x86_64 /bin/bash • https://github.com/pypa/manylinux
56. What’s Required Automation
57. Caveats • Users need the right versions of pip • 8.1+ if you want to avoid quite a few issues
58. Caveats • Users need the right versions of pip/wheel • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it
59. Caveats • Users need the right versions of pip/wheel • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it • Unusual architectures aren’t supported by wheels on PyPI
60. Caveats • Users need the right versions of pip/wheel • 8.1+ if you want to avoid quite a few issues • If the library you're using isn't guaranteed available you'll have to bundle it • Unusual architectures aren’t supported by wheels on PyPI • Download size and memory footprint increase
61. Bonus Content! • Let’s talk about manylinux1 in the wild…
62. Bonus Content! • Let’s talk about manylinux1 in the wild… • What’s a good test case for manylinux1?
63. Bonus Content! • Let’s talk about manylinux1 in the wild • What’s a good test case for manylinux1? • bcrypt
64. Bonus Content! • Let’s talk about manylinux1 in the wild • What’s a good test case for manylinux1? • bcrypt
65. But wait there’s more • What if…?
66. But wait there’s more • What if…? • cffi!
67. But wait there’s more • What if…? • cffi! • https://github.com/reaperhulk/cffi-manylinux1
68. But wait there’s more • What if…? • cffi! • https://github.com/reaperhulk/cffi-manylinux1
69. manylinux1 works!
70. Good luck Go forth and build wheels!
71. Thank you
72. Resources • manylinux1: https://github.com/pypa/manylinux • Wheel builders mailing list: https://mail.python.org/mailman/listinfo/wheel- builders • Blog: https://langui.sh • Twitter: https://twitter.com/reaperhulk • Simpsons related technology: https://frinkiac.com • Futurama related tech: https://morbotron.com