Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python packaging in 2017

February 12, 2017

Python packaging in 2017

A keynote talk I gave at PyCon Colombia.

Describe the state of python packaging in 2017, with some history, and things that I think need to improve.


February 12, 2017

More Decks by cournape

Other Decks in Programming


  1. The narcissist slide • Started using python in 2004 •

    Became heavy user of NumPy in 2006, core contributor since 2007 • Used python for speech processing, scripting, web applications • Has a unhealthy interest in packaging
  2. A bit of history 1/2 • 199*: Setup (see Lib/Setup*

    in cpython sources) • 1999-2000: distutils (setup.py) by Greg Ward. An official method to distribute 3rd party packages • early 2000: pypi aka the cheeseshop (Richard Jones) • 2004-2005: setuptools, easy_install, eggs (Phillip J. Eby, Bob Ippolito). Automatically install packages from pypi/other sources, binary format (egg)
  3. A bit of history 2/2 • 2008: pip. Always install

    from sources (no modification of sys.path), requirements file • 2012/2013: pip and setuptools managed by the community (Donald Stufft, Jason R Coombs) • 2012/2013: wheel, a new binary format (PEP 427), and python tags (PEP 425)
  4. Where are we in 2017 ? • For python packages:

    use pip, wheels and setuptools, (distribute, eggs and and easy_install are obsolete) • To push to pypi, use twine • more on https://packaging.python.org/
  5. Packages with C extensions • pure, cross platform packages: wheels

    are simple to build • C extensions: different wheels for different platforms • on windows, each python version needs different compiler • on OS X, backward compatibility can be tricky • on Linux: use many linux effort (PEP 513)
  6. Manylinux wheels • Binary wheels that work on many linux

    distributions • see https://github.com/pypa/manylinux for tools to make it relatively simple
  7. Packaging and languages • Tools from different, unfamiliar ecosystems =

    terrible experience • Your enduser do not care about python • (Some of) python packaging needs to forget about python
  8. Go-like packaging 1/3 • A go program: often a single

    binary • Common approach: pyinstaller/pyfreeze/etc. • hard to use • often arcane, use “magic” to figure out imports • Wouldn’t it be great if we could do like go in python ?
  9. go-like packaging 2/3 • It is possible: http://www.egenix.com/products/ python/PyRun/ •

    basic idea: • build python statically • use freeze module from Tools • example
  10. go-like packaging (3/3) • Many limitations (static builds not well

    supported, C extensions, package data) • Opportunities to make this simpler
  11. Decoupling packaging stack • setup.py: defines both metadata and “build”

    actions • build and packaging are coupled • some packages need a “real” build system
  12. Automatic wheel building • Service to build wheels from tarballs

    • See e.g. conda forge for anaconda, R build service
  13. This is only the beginning • You can be part

    of it ! • if you have OSS packages: • build binary wheels for pypi • follow suggestions from https:// packaging.python.org/ • try to help pypa: https://www.pypa.io/en/latest/
  14. Thank you ! • Thanks to the PyCon Colombia team

    for organizing the conference • Thanks to AxiaCore for the warm welcome in Colombia