Python packaging in 2017

A keynote talk I gave at PyCon Colombia.

Describe the state of python packaging in 2017, with some history, and things that I think need to improve.

Transcript

1. Python packaging
   where are we in 2017 (and how did we get there) ?
   

   View Slide

2. The narcissist slide
   • Started using python in 2004
   • Became heavy user of NumPy
   in 2006, core contributor since
   2007
   • Used python for speech
   processing, scripting, web
   applications
   • Has a unhealthy interest in
   packaging
   

   View Slide

3. What is packaging ?
   

   View Slide

4. “whatever needs to happen for
   your code to reach your users”
   

   View Slide

5. Can be a source
   tarball
   

   View Slide

6. Can be platform native
   package (msi, deb, rpm,
   etc…)
   

   View Slide

7. Some binary artifact to
   deploy to a bunch of
   servers
   

   View Slide

8. Without packaging, your
   software is useless to
   most people !
   

   View Slide

9. Different audiences =
   different needs
   

   View Slide

10. A bit of history
    

    View Slide

11. A bit of history 1/2
    • 199*: Setup (see Lib/Setup* in cpython sources)
    • 1999-2000: distutils (setup.py) by Greg Ward. An
    official method to distribute 3rd party packages
    • early 2000: pypi aka the cheeseshop (Richard
    Jones)
    • 2004-2005: setuptools, easy_install, eggs (Phillip J.
    Eby, Bob Ippolito). Automatically install packages
    from pypi/other sources, binary format (egg)
    

    View Slide

12. A bit of history 2/2
    • 2008: pip. Always install from sources (no
    modification of sys.path), requirements file
    • 2012/2013: pip and setuptools managed by the
    community (Donald Stufft, Jason R Coombs)
    • 2012/2013: wheel, a new binary format (PEP
    427), and python tags (PEP 425)
    

    View Slide

13. “parallel efforts”
    • Binary distributions: ActiveState, (ana)conda by
    Continuum Analytics, Canopy by Enthought (my
    employer)
    • numpy.distutils
    

    View Slide

14. The present
    

    View Slide

15. Where are we in 2017 ?
    • For python packages: use pip, wheels and
    setuptools, (distribute, eggs and and
    easy_install are obsolete)
    • To push to pypi, use twine
    • more on https://packaging.python.org/
    

    View Slide

16. Packages with C extensions
    • pure, cross platform packages: wheels are simple
    to build
    • C extensions: different wheels for different platforms
    • on windows, each python version needs different
    compiler
    • on OS X, backward compatibility can be tricky
    • on Linux: use many linux effort (PEP 513)
    

    View Slide

17. Manylinux wheels
    • Binary wheels that work on many linux
    distributions
    • see https://github.com/pypa/manylinux for tools
    to make it relatively simple
    

    View Slide

18. The road ahead
    

    View Slide

19. Have you ever used a ruby,
    php or node.js application ?
    

    View Slide

20. Packaging and languages
    • Tools from different, unfamiliar ecosystems =
    terrible experience
    • Your enduser do not care about python
    • (Some of) python packaging needs to forget
    about python
    

    View Slide

21. Go-like packaging 1/3
    • A go program: often a single binary
    • Common approach: pyinstaller/pyfreeze/etc.
    • hard to use
    • often arcane, use “magic” to figure out imports
    • Wouldn’t it be great if we could do like go in
    python ?
    

    View Slide

22. go-like packaging 2/3
    • It is possible: http://www.egenix.com/products/
    python/PyRun/
    • basic idea:
    • build python statically
    • use freeze module from Tools
    • example
    

    View Slide

23. go-like packaging (3/3)
    • Many limitations (static builds not well
    supported, C extensions, package data)
    • Opportunities to make this simpler
    

    View Slide

24. Decoupling packaging stack
    • setup.py: defines both metadata and “build”
    actions
    • build and packaging are coupled
    • some packages need a “real” build system
    

    View Slide

25. Automatic wheel building
    • Service to build wheels from tarballs
    • See e.g. conda forge for anaconda, R build
    service
    

    View Slide

26. This is only the beginning
    • You can be part of it !
    • if you have OSS packages:
    • build binary wheels for pypi
    • follow suggestions from https://
    packaging.python.org/
    • try to help pypa: https://www.pypa.io/en/latest/
    

    View Slide

27. Thank you !
    • Thanks to the PyCon Colombia team for
    organizing the conference
    • Thanks to AxiaCore for the warm welcome in
    Colombia
    

    View Slide