Python packaging where are we in 2017 (and how did we get there) ? 

The narcissist slide • Started using python in 2004 • Became heavy user of NumPy in 2006, core contributor since 2007 • Used python for speech processing, scripting, web applications • Has a unhealthy interest in packaging 

What is packaging ? 

“whatever needs to happen for your code to reach your users” 

Can be a source tarball 

Can be platform native package (msi, deb, rpm, etc…) 

Some binary artifact to deploy to a bunch of servers 

Without packaging, your software is useless to most people ! 

Different audiences = different needs 

A bit of history 

A bit of history 1/2 • 199*: Setup (see Lib/Setup* in cpython sources) • 1999-2000: distutils (setup.py) by Greg Ward. An official method to distribute 3rd party packages • early 2000: pypi aka the cheeseshop (Richard Jones) • 2004-2005: setuptools, easy_install, eggs (Phillip J. Eby, Bob Ippolito). Automatically install packages from pypi/other sources, binary format (egg) 

A bit of history 2/2 • 2008: pip. Always install from sources (no modification of sys.path), requirements file • 2012/2013: pip and setuptools managed by the community (Donald Stufft, Jason R Coombs) • 2012/2013: wheel, a new binary format (PEP 427), and python tags (PEP 425) 

“parallel efforts” • Binary distributions: ActiveState, (ana)conda by Continuum Analytics, Canopy by Enthought (my employer) • numpy.distutils 

The present 

Where are we in 2017 ? • For python packages: use pip, wheels and setuptools, (distribute, eggs and and easy_install are obsolete) • To push to pypi, use twine • more on https://packaging.python.org/ 

Packages with C extensions • pure, cross platform packages: wheels are simple to build • C extensions: different wheels for different platforms • on windows, each python version needs different compiler • on OS X, backward compatibility can be tricky • on Linux: use many linux effort (PEP 513) 

Manylinux wheels • Binary wheels that work on many linux distributions • see https://github.com/pypa/manylinux for tools to make it relatively simple 

The road ahead 

Have you ever used a ruby, php or node.js application ? 

Packaging and languages • Tools from different, unfamiliar ecosystems = terrible experience • Your enduser do not care about python • (Some of) python packaging needs to forget about python 

Go-like packaging 1/3 • A go program: often a single binary • Common approach: pyinstaller/pyfreeze/etc. • hard to use • often arcane, use “magic” to figure out imports • Wouldn’t it be great if we could do like go in python ? 

go-like packaging 2/3 • It is possible: http://www.egenix.com/products/ python/PyRun/ • basic idea: • build python statically • use freeze module from Tools • example 

go-like packaging (3/3) • Many limitations (static builds not well supported, C extensions, package data) • Opportunities to make this simpler 

Decoupling packaging stack • setup.py: defines both metadata and “build” actions • build and packaging are coupled • some packages need a “real” build system 

Automatic wheel building • Service to build wheels from tarballs • See e.g. conda forge for anaconda, R build service 

This is only the beginning • You can be part of it ! • if you have OSS packages: • build binary wheels for pypi • follow suggestions from https:// packaging.python.org/ • try to help pypa: https://www.pypa.io/en/latest/ 

Thank you ! • Thanks to the PyCon Colombia team for organizing the conference • Thanks to AxiaCore for the warm welcome in Colombia