$30 off During Our Annual Pro Sale. View Details »

Paul Kehrer - Reliably Distributing Compiled Modules

Paul Kehrer - Reliably Distributing Compiled Modules

Shipping Python libraries is easy! ...until you want to use a C library. How do you easily and reliably deliver software to users when they may not have the libraries you depend on, or even a compiler? How do you handle the significant differences between linux, OS X, Windows, FreeBSD, and other platforms Python runs on?

https://us.pycon.org/2016/schedule/presentation/2145/

PyCon 2016

May 29, 2016
Tweet

More Decks by PyCon 2016

Other Decks in Programming

Transcript

  1. Reliably Distributing
    Compiled Modules
    Paul Kehrer (reaperhulk) – PyCon 2016

    View Slide

  2. Who am I?
    • I’m Paul Kehrer
    • I write software, kiteboard, and bake things
    • I am an inveterate Simpsons quoter.
    • @reaperhulk on Twitter
    • @reaperhulk on Freenode
    • @reaperhulk on GitHub
    • …you get the idea

    View Slide

  3. What I work on
    • Projects for Rackspace Managed Security
    • Python Cryptographic Authority (PyCA)
    • cryptography
    • pyOpenSSL
    • PyNaCl
    • bcrypt
    • Frinkiac (but that’s a separate talk)

    View Slide

  4. What do we want to do?
    pip install cryptography

    View Slide

  5. What is a compiled module?
    • Any module that uses ”binary” code

    View Slide

  6. What is a compiled module?
    • Any module that uses “binary” code
    • Less facetiously, a module that contains some sort of
    build process that results in architecture and operating
    system specific output that can be executed or calls into
    such code even if it does not invoke the compilation
    itself.
    • This is a bit of a simplification, but will do for our
    purposes.

    View Slide

  7. Installing…
    Well, we need a compiler right?
    apt-get install build-essential
    …or
    yum install make automake gcc
    if RHEL/CentOS

    View Slide

  8. pip install cryptography

    View Slide

  9. Installing…
    (.venv)root@pycon2016:~# pip install
    cryptography
    ...snip...
    Installing collected packages: cryptography,
    idna, pyasn1, six, setuptools, enum34,
    ipaddress, cffi, pycparser
    Running setup.py install for cryptography
    c/_cffi_backend.c:2:20: fatal error:
    Python.h: No such file or directory
    #include
    ...snip...

    View Slide

  10. (.venv)root@pycon2016:~# pip install cryptography
    Downloading/unpacking cryptography
    Downloading cryptography-1.3.2.tar.gz (383kB): 383kB downloaded
    Running setup.py (path:/root/.venv/build/cryptography/setup.py) egg_info for package cryptography
    no previously-included directories found matching 'docs/_build'
    warning: no previously-included files matching '*' found under directory 'vectors'
    Downloading/unpacking idna>=2.0 (from cryptography)
    Downloading idna-2.1-py2.py3-none-any.whl (54kB): 54kB downloaded
    Downloading/unpacking pyasn1>=0.1.8 (from cryptography)
    Downloading pyasn1-0.1.9-py2.py3-none-any.whl
    Downloading/unpacking six>=1.4.1 (from cryptography)
    Downloading six-1.10.0-py2.py3-none-any.whl
    Downloading/unpacking setuptools>=11.3 (from cryptography)
    Downloading setuptools-21.2.0-py2.py3-none-any.whl (509kB): 509kB downloaded
    Downloading/unpacking enum34 (from cryptography)
    Downloading enum34-1.1.6-py2-none-any.whl
    Downloading/unpacking ipaddress (from cryptography)
    Downloading ipaddress-1.0.16-py27-none-any.whl
    Downloading/unpacking cffi>=1.4.1 (from cryptography)
    Downloading cffi-1.6.0.tar.gz (397kB): 397kB downloaded
    Running setup.py (path:/root/.venv/build/cffi/setup.py) egg_info for package cffi
    Downloading/unpacking pycparser (from cffi>=1.4.1->cryptography)
    Downloading pycparser-2.14.tar.gz (223kB): 223kB downloaded
    Running setup.py (path:/root/.venv/build/pycparser/setup.py) egg_info for package pycparser
    warning: no previously-included files matching 'yacctab.*' found under directory 'tests'
    warning: no previously-included files matching 'lextab.*' found under directory 'tests'
    warning: no previously-included files matching 'yacctab.*' found under directory 'examples'
    warning: no previously-included files matching 'lextab.*' found under directory 'examples'
    Installing collected packages: cryptography, idna, pyasn1, six, setuptools, enum34, ipaddress, cffi,
    pycparser
    Running setup.py install for cryptography

    View Slide

  11. c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
    #include
    ^
    compilation terminated.
    Traceback (most recent call last):
    File "", line 1, in
    File "/root/.venv/build/cryptography/setup.py", line 335, in
    **keywords_with_side_effects(sys.argv)
    File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in
    __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in
    fetch_build_eggs
    replace_conflicting=True
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in
    resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in
    best_match
    return self.obtain(req, installer) # try and download/install
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in
    obtain
    return installer(requirement)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in
    fetch_build_egg
    return cmd.easy_install(req)
    File "/root/.venv/local/lib/python2.7/site-
    packages/setuptools/command/easy_install.py", line 593, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
    File "/root/.venv/local/lib/python2.7/site-
    packages/setuptools/command/easy_install.py", line 623, in install_item
    dists = self.install_eggs(spec, download, tmpdir)

    View Slide

  12. File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py",
    line 809, in install_eggs
    return self.build_and_install(setup_script, setup_base)
    File "/root/.venv/local/lib/python2.7/site-
    packages/setuptools/command/easy_install.py", line 1015, in build_and_install
    self.run_setup(setup_script, setup_base, args)
    File "/root/.venv/local/lib/python2.7/site-
    packages/setuptools/command/easy_install.py", line 1003, in run_setup
    raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-
    gnu-gcc' failed with exit status 1
    Complete output from command /root/.venv/bin/python -c "import setuptools,
    tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize,
    'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record
    /tmp/pip-ag4Mxm-record/install-record.txt --single-version-externally-managed --compile --
    install-headers /root/.venv/include/site/python2.7:
    c/_cffi_backend.c:2:20: fatal error: Python.h: No such file or directory
    #include
    ^
    compilation terminated.
    Traceback (most recent call last):
    File "", line 1, in
    File "/root/.venv/build/cryptography/setup.py", line 335, in
    **keywords_with_side_effects(sys.argv)
    File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 239, in
    __init__
    self.fetch_build_eggs(attrs.pop('setup_requires'))
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 264, in
    fetch_build_eggs

    View Slide

  13. replace_conflicting=True
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 580, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 818, in best_match
    return self.obtain(req, installer) # try and download/install
    File "/root/.venv/local/lib/python2.7/site-packages/pkg_resources.py", line 830, in obtain
    return installer(requirement)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/dist.py", line 314, in
    fetch_build_egg
    return cmd.easy_install(req)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 593,
    in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 623,
    in install_item
    dists = self.install_eggs(spec, download, tmpdir)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 809,
    in install_eggs
    return self.build_and_install(setup_script, setup_base)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1015,
    in build_and_install
    self.run_setup(setup_script, setup_base, args)
    File "/root/.venv/local/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 1003,
    in run_setup
    raise DistutilsError("Setup script exited with %s" % (v.args[0],))
    distutils.errors.DistutilsError: Setup script exited with error: command 'x86_64-linux-gnu-gcc'
    failed with exit status 1
    ----------------------------------------
    Cleaning up...
    Command /root/.venv/bin/python -c "import setuptools,
    tokenize;__file__='/root/.venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open',
    open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-ag4Mxm-
    record/install-record.txt --single-version-externally-managed --compile --install-headers
    /root/.venv/include/site/python2.7 failed with error code 1 in /root/.venv/build/cryptography
    Storing debug log for failure in /root/.pip/pip.log

    View Slide

  14. What went wrong?

    View Slide

  15. What went wrong?
    • No Python.h

    View Slide

  16. What went wrong?
    • No Python.h

    View Slide

  17. What went wrong?
    • No Python.h

    View Slide

  18. What went wrong?
    • No Python.h
    • apt-get install python-dev

    View Slide

  19. How about now?
    (.venv)root@pycon2016:~# pip install
    cryptography
    ...snip...
    Running setup.py install for cryptography
    c/_cffi_backend.c:15:17: fatal error:
    ffi.h: No such file or directory
    #include
    ...snip...

    View Slide

  20. yum install libffi-devel
    (Wait, weren’t we using apt before?)

    View Slide

  21. This time…
    ...snip...
    generating cffi module 'build/temp.linux-x86_64-
    2.7/_openssl.c'
    building '_openssl' extension
    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing
    -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC
    -I/usr/include/python2.7 -c build/temp.linux-x86_64-
    2.7/_openssl.c -o build/temp.linux-x86_64-
    2.7/build/temp.linux-x86_64-2.7/_openssl.o
    build/temp.linux-x86_64-2.7/_openssl.c:423:30:
    fatal error: openssl/opensslv.h: No such file or
    directory
    #include
    ^
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with
    exit status 1
    ...snip...

    View Slide

  22. brew install openssl

    View Slide

  23. brew install openssl
    apt-get install libssl-dev

    View Slide

  24. brew install openssl
    apt-get install libssl-dev
    yum install openssl-devel

    View Slide

  25. brew install openssl
    apt-get install libssl-dev
    yum install openssl-devel
    dnf install openssl-devel

    View Slide

  26. One final time…
    ...snip...
    Successfully installed cryptography idna
    pyasn1 six setuptools enum34 ipaddress cffi
    pycparser

    View Slide

  27. The real prerequisites
    • The abstract requirements consist of
    • compiler - Typically, but not guaranteed to be, gcc, clang,
    or msvc
    • library - If you're linking against one
    • headers - So the code can know what function
    signatures are available in the library
    *Implicit in the following is an assumption that the user has root/admin privileges. If they do
    not then compiling a module may become extremely difficult.

    View Slide

  28. Compiling on OS X
    • xcode-select --install
    • Unless the user’s OS X is older than 10.9…

    View Slide

  29. Compiling on OS X
    • xcode-select --install
    • Unless the user’s OS X is older than 10.9…
    • If your software depends on libraries outside the
    OS X default
    • homebrew
    • macports

    View Slide

  30. Compiling on Windows
    • Download Visual Studio (on occasion someone who
    wants to make your life hard might choose mingw)
    • For Python 2.7 the user will need "Microsoft Visual C++
    Compiler for Python 2.7"
    • Python 3.3 and 3.4 use Visual Studio 2010
    • Python 3.5 uses Visual Studio 2015

    View Slide

  31. Compiling on Windows
    • Architecture matters
    • 32-bit and 64-bit Pythons
    • VS2010 64-bit compilers aren’t free
    • No package manager for getting third party libraries

    View Slide

  32. Compiling on Linux
    • On Linux each distribution has its own view of what
    packages provide the basic compiler tools.
    • apt-get install build-essential
    • yum install make automake gcc
    • …and its own opinion of what packages provide the
    required headers/libraries.

    View Slide

  33. Common Problems
    • C toolchain lookup paths

    View Slide

  34. Common Problems
    • C toolchain lookup paths
    • Unsupported library versions

    View Slide

  35. Common Problems
    • C toolchain lookup paths
    • Unsupported library versions
    • Windows CRT/compiler restrictions

    View Slide

  36. Common Problems
    • C toolchain lookup paths
    • Unsupported library versions
    • Windows CRT/compiler restrictions
    • People do terrible things to their computers
    • Install software from random tarballs or installers that end up in
    /usr/local, but leave conflicting versions from elsewhere in their
    paths.
    • Edit /etc/ld.so.confand screw up library path searches.
    • Add LD_LIBRARY_PATH or DYLD_LIBRARY_PATH env variables to
    their .bash_profile without understanding the ramifications

    View Slide

  37. This is not reasonable

    View Slide

  38. Let’s avoid this

    View Slide

  39. Let’s avoid this
    (Apparently 130x better)

    View Slide

  40. Wheels

    View Slide

  41. Wheels
    • PEP 427/491/513 (and probably more)
    • Provides a way to distribute Python artifacts.
    • Superior in a variety of ways to eggs and sdists
    • Install packages with binary dependencies sans root
    • pip can both produce and install them
    • Tags are used to denote what a given wheel is
    compatible with:
    • e.g. cryptography-1.3.2-cp27-cp27mu-
    macosx_10_10_x86_64.whl

    View Slide

  42. Look deep within
    yourself

    View Slide

  43. Questions for your project
    • What platforms do you care about?

    View Slide

  44. Questions for your project
    • What platforms do you care about?
    • What versions of Python do you care about?

    View Slide

  45. Questions for your project
    • What platforms do you care about?
    • What versions of Python do you care about?
    • Can you rely on the library you need to be
    present on all platforms?

    View Slide

  46. What’s Required (OS X)
    • An OS X machine
    • Any recent release will do (but why aren’t you
    on El Capitan?)

    View Slide

  47. What’s Required (OS X)
    • An OS X machine
    • Any recent release will do, but why aren’t you
    on El Capitan?
    • Challenges
    • OS X SDK versions
    • Universal wheels
    • UCS2/UCS4

    View Slide

  48. Handling OS X SDK versions
    • Build against python.org Pythons
    • Not pyenvor system python
    • pip’s handling of OS X SDK restrictions has changed
    in the recent past

    View Slide

  49. Universal Support
    • Universal wheels are denoted with the intel
    platform tag
    • As opposed to i686 or x86_64

    View Slide

  50. Universal Support
    • Universal wheels are denoted with the intel
    platform tag
    • As opposed to i686 or x86_64
    • python.org pythons are universal

    View Slide

  51. Universal Support
    • Universal wheels are denoted with the intel
    platform tag
    • As opposed to i686 or x86_64
    • python.org pythons are universal
    • Libraries you link against must also be universal
    • lipo -info /path/to/lib

    View Slide

  52. UCS2/UCS4
    Symbol not found:
    _PyUnicodeUCS2_AsASCIIString

    View Slide

  53. UCS2/UCS4
    • Python has two ways to compile unicode support in
    2.x (and < 3.3, but ignore those releases)
    • UCS2 and UCS4.
    • The resulting Python has a different ABI so we need
    different wheels.
    • pip/wheel did not handle this until recently (wheel
    0.27.0 and pip 8.1)
    • Use a pyenv 2.7 to handle this case.

    View Slide

  54. What’s Required (OS X)
    • Pythons from python.org
    • One pyenv Python 2.7
    • Patience to understand all the permutations

    View Slide

  55. What’s Required (Windows)
    • Make life simple by building a separate VM for 32-
    bit vs 64-bit Python.
    • This way Python can just live in C:\Python35, C:\Python27, et cetera
    • Install Microsoft Visual C++ Compiler for Python 2.7
    • Install Visual Studio 2015 Express for Python 3.5
    • Install Visual Studio 2010 for Python 3.3 and 3.4
    • You’re on your own for libraries L

    View Slide

  56. What’s Required (manylinux1)
    docker run -it --rm -v
    /path/to/python/code:/io
    quay.io/pypa/manylinux1_x86_64
    /bin/bash
    • https://github.com/pypa/manylinux

    View Slide

  57. What’s Required
    Automation

    View Slide

  58. Caveats
    • Users need the right versions of pip
    • 8.1+ if you want to avoid quite a few issues

    View Slide

  59. Caveats
    • Users need the right versions of pip
    • 8.1+ if you want to avoid quite a few issues
    • If the library you're using isn't guaranteed available
    you'll have to bundle it

    View Slide

  60. Caveats
    • Users need the right versions of pip
    • 8.1+ if you want to avoid quite a few issues
    • If the library you're using isn't guaranteed available
    you'll have to bundle it
    • Unusual architectures aren’t supported by wheels
    on PyPI

    View Slide

  61. Caveats
    • Users need the right versions of pip/wheel
    • 8.1+ if you want to avoid quite a few issues
    • If the library you're using isn't guaranteed available
    you'll have to bundle it
    • Unusual architectures aren’t supported by wheels
    on PyPI
    • Download size and memory footprint increase

    View Slide

  62. Good luck
    Go forth and build wheels!

    View Slide

  63. Thank you

    View Slide

  64. Resources
    • manylinux1: https://github.com/pypa/manylinux
    • Wheel builders mailing list:
    https://mail.python.org/mailman/listinfo/wheel-
    builders
    • Blog: https://langui.sh
    • Twitter: https://twitter.com/reaperhulk
    • Simpsons related technology: https://frinkiac.com

    View Slide