Python packaging in 2017

A keynote talk I gave at PyCon Colombia.

Describe the state of python packaging in 2017, with some history, and things that I think need to improve.

Transcript

1. Python packaging where are we in 2017 (and how did we get there) ?
2. The narcissist slide • Started using python in 2004 • Became heavy user of NumPy in 2006, core contributor since 2007 • Used python for speech processing, scripting, web applications • Has a unhealthy interest in packaging
3. What is packaging ?
4. “whatever needs to happen for your code to reach your users”
5. Can be a source tarball
6. Can be platform native package (msi, deb, rpm, etc…)
7. Some binary artifact to deploy to a bunch of servers
8. Without packaging, your software is useless to most people !
9. Different audiences = different needs
10. A bit of history
11. A bit of history 1/2 • 199*: Setup (see Lib/Setup* in cpython sources) • 1999-2000: distutils (setup.py) by Greg Ward. An official method to distribute 3rd party packages • early 2000: pypi aka the cheeseshop (Richard Jones) • 2004-2005: setuptools, easy_install, eggs (Phillip J. Eby, Bob Ippolito). Automatically install packages from pypi/other sources, binary format (egg)
12. A bit of history 2/2 • 2008: pip. Always install from sources (no modification of sys.path), requirements file • 2012/2013: pip and setuptools managed by the community (Donald Stufft, Jason R Coombs) • 2012/2013: wheel, a new binary format (PEP 427), and python tags (PEP 425)
13. “parallel efforts” • Binary distributions: ActiveState, (ana)conda by Continuum Analytics, Canopy by Enthought (my employer) • numpy.distutils
14. The present
15. Where are we in 2017 ? • For python packages: use pip, wheels and setuptools, (distribute, eggs and and easy_install are obsolete) • To push to pypi, use twine • more on https://packaging.python.org/
16. Packages with C extensions • pure, cross platform packages: wheels are simple to build • C extensions: different wheels for different platforms • on windows, each python version needs different compiler • on OS X, backward compatibility can be tricky • on Linux: use many linux effort (PEP 513)
17. Manylinux wheels • Binary wheels that work on many linux distributions • see https://github.com/pypa/manylinux for tools to make it relatively simple
18. The road ahead
19. Have you ever used a ruby, php or node.js application ?
20. Packaging and languages • Tools from different, unfamiliar ecosystems = terrible experience • Your enduser do not care about python • (Some of) python packaging needs to forget about python
21. Go-like packaging 1/3 • A go program: often a single binary • Common approach: pyinstaller/pyfreeze/etc. • hard to use • often arcane, use “magic” to figure out imports • Wouldn’t it be great if we could do like go in python ?
22. go-like packaging 2/3 • It is possible: http://www.egenix.com/products/ python/PyRun/ • basic idea: • build python statically • use freeze module from Tools • example
23. go-like packaging (3/3) • Many limitations (static builds not well supported, C extensions, package data) • Opportunities to make this simpler
24. Decoupling packaging stack • setup.py: defines both metadata and “build” actions • build and packaging are coupled • some packages need a “real” build system
25. Automatic wheel building • Service to build wheels from tarballs • See e.g. conda forge for anaconda, R build service
26. This is only the beginning • You can be part of it ! • if you have OSS packages: • build binary wheels for pypi • follow suggestions from https:// packaging.python.org/ • try to help pypa: https://www.pypa.io/en/latest/
27. Thank you ! • Thanks to the PyCon Colombia team for organizing the conference • Thanks to AxiaCore for the warm welcome in Colombia