Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python packaging in 2017

cournape
February 12, 2017

Python packaging in 2017

A keynote talk I gave at PyCon Colombia.

Describe the state of python packaging in 2017, with some history, and things that I think need to improve.

cournape

February 12, 2017
Tweet

More Decks by cournape

Other Decks in Programming

Transcript

  1. Python packaging
    where are we in 2017 (and how did we get there) ?

    View full-size slide

  2. The narcissist slide
    • Started using python in 2004
    • Became heavy user of NumPy
    in 2006, core contributor since
    2007
    • Used python for speech
    processing, scripting, web
    applications
    • Has a unhealthy interest in
    packaging

    View full-size slide

  3. What is packaging ?

    View full-size slide

  4. “whatever needs to happen for
    your code to reach your users”

    View full-size slide

  5. Can be a source
    tarball

    View full-size slide

  6. Can be platform native
    package (msi, deb, rpm,
    etc…)

    View full-size slide

  7. Some binary artifact to
    deploy to a bunch of
    servers

    View full-size slide

  8. Without packaging, your
    software is useless to
    most people !

    View full-size slide

  9. Different audiences =
    different needs

    View full-size slide

  10. A bit of history

    View full-size slide

  11. A bit of history 1/2
    • 199*: Setup (see Lib/Setup* in cpython sources)
    • 1999-2000: distutils (setup.py) by Greg Ward. An
    official method to distribute 3rd party packages
    • early 2000: pypi aka the cheeseshop (Richard
    Jones)
    • 2004-2005: setuptools, easy_install, eggs (Phillip J.
    Eby, Bob Ippolito). Automatically install packages
    from pypi/other sources, binary format (egg)

    View full-size slide

  12. A bit of history 2/2
    • 2008: pip. Always install from sources (no
    modification of sys.path), requirements file
    • 2012/2013: pip and setuptools managed by the
    community (Donald Stufft, Jason R Coombs)
    • 2012/2013: wheel, a new binary format (PEP
    427), and python tags (PEP 425)

    View full-size slide

  13. “parallel efforts”
    • Binary distributions: ActiveState, (ana)conda by
    Continuum Analytics, Canopy by Enthought (my
    employer)
    • numpy.distutils

    View full-size slide

  14. Where are we in 2017 ?
    • For python packages: use pip, wheels and
    setuptools, (distribute, eggs and and
    easy_install are obsolete)
    • To push to pypi, use twine
    • more on https://packaging.python.org/

    View full-size slide

  15. Packages with C extensions
    • pure, cross platform packages: wheels are simple
    to build
    • C extensions: different wheels for different platforms
    • on windows, each python version needs different
    compiler
    • on OS X, backward compatibility can be tricky
    • on Linux: use many linux effort (PEP 513)

    View full-size slide

  16. Manylinux wheels
    • Binary wheels that work on many linux
    distributions
    • see https://github.com/pypa/manylinux for tools
    to make it relatively simple

    View full-size slide

  17. The road ahead

    View full-size slide

  18. Have you ever used a ruby,
    php or node.js application ?

    View full-size slide

  19. Packaging and languages
    • Tools from different, unfamiliar ecosystems =
    terrible experience
    • Your enduser do not care about python
    • (Some of) python packaging needs to forget
    about python

    View full-size slide

  20. Go-like packaging 1/3
    • A go program: often a single binary
    • Common approach: pyinstaller/pyfreeze/etc.
    • hard to use
    • often arcane, use “magic” to figure out imports
    • Wouldn’t it be great if we could do like go in
    python ?

    View full-size slide

  21. go-like packaging 2/3
    • It is possible: http://www.egenix.com/products/
    python/PyRun/
    • basic idea:
    • build python statically
    • use freeze module from Tools
    • example

    View full-size slide

  22. go-like packaging (3/3)
    • Many limitations (static builds not well
    supported, C extensions, package data)
    • Opportunities to make this simpler

    View full-size slide

  23. Decoupling packaging stack
    • setup.py: defines both metadata and “build”
    actions
    • build and packaging are coupled
    • some packages need a “real” build system

    View full-size slide

  24. Automatic wheel building
    • Service to build wheels from tarballs
    • See e.g. conda forge for anaconda, R build
    service

    View full-size slide

  25. This is only the beginning
    • You can be part of it !
    • if you have OSS packages:
    • build binary wheels for pypi
    • follow suggestions from https://
    packaging.python.org/
    • try to help pypa: https://www.pypa.io/en/latest/

    View full-size slide

  26. Thank you !
    • Thanks to the PyCon Colombia team for
    organizing the conference
    • Thanks to AxiaCore for the warm welcome in
    Colombia

    View full-size slide