Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python Packaging: Why Don’t You Just…?

Python Packaging: Why Don’t You Just…?

PackagingCon 2021

Schedule: https://pretalx.com/packagingcon-2021/talk/X8N8ME/
Recording: https://www.youtube.com/watch?v=jpP07nZiicE

Every packaging system has its specific way of doing things, but to an outsider. Python’s seems to have a knack of finding the most non-straightforward and weird solution for every choice. This talk attempts to trace some of the peculiarities to find out the reasoning behind the decisions, and how they stand in the modern packaging landscape.


Python packaging has a long and winding history. When it startd out, few communities have good packaging offerings, and Python’s was praised for being easy to be picked up and develop with. As time went on, many tried to bring in new ideas from relatively new packaging systems to “modernise” Python packaging, but those suggestions are not always received favourably, and Python’s packaging community is sometimes categorised as overly conservative, or even user-hostile due to this.

In most of such situations, however, the main problem is not that Python packaging is unwilling to change, but there are certain design decisions that affect how problems should be approached. While those characteristics may seem alien to newcomers, they are made to accomdate special needs in the Python community, and any changes to the packaging system need to take them into consideration. This sometimes require contributors to be more creative when putting together a new feature, but the Python community is only as valuable as its users, and new progress should avoid throwing away what enable Python to grow to what it is.

Tzu-ping Chung

November 11, 2021

More Decks by Tzu-ping Chung

Other Decks in Programming


  1. Me Call me TP (uraunsjr.com) Taipei, Taiwan Apache Airflow @

    Astronomer Python packaging (PyPA & others) 2
  2. Python Packaging LOL So weird we must split it into

    topics How did this thing happen? What does the solution achieve? How do we improve the situation? 3
  3. Why O Why A ton of packages downloaded for no

    reason! PyPI should automatically build wheels! I can import my package but you can’t find it? 4
  4. A ton of packages downloaded for no reason! Why O

    Why PyPI should automatically build wheels! I can import my package but you can’t find it? 5
  5. Per-artifact Metadata — History PyPI is just a bunch of

    tarballs Well, also zipballs So simple it’s called the Simple API Conditionals in setup.py Wheel contains prebuilt metadata 6
  6. They may all have different metadata <a href="...">tensorflow-2.6.1-cp36-cp36m-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp36-cp36m-manylinux2010_x86_64.whl</a>

    <a href="...">tensorflow-2.6.1-cp36-cp36m-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-win_amd64.whl</a> 7
  7. Per-artifact Metadata — Advantages Every file server is a Python

    package index Just put artifacts in a directory! Install a package with a “direct URL” 8
  8. Per-artifact Metadata — Improvements A way to say I don’t

    have dynamic metadata Serve metadata alongside with each artifact Detect and degrade gracefully 9
  9. PyPI should automatically build wheels! Why O Why A ton

    of packages downloaded for no reason! I can import my package but you can’t find it? 10
  10. Infinite Wheelhouse Paradox How many platforms does CPython support? Trick

    question! As long as it builds, it’s good There is a very long tail of platforms You can never build all wheels, theoratically 11
  11. Pull-based Platform Declaration Basically feature detection Declare needs for platforms

    to satisfy Platform PEPs are Informational As long as your binaries work for your users 12
  12. Sufficiently Large Enough Make platform reflection easy sysconfig Create a

    “common enough” template cibuildwheel Simplify and externalise wheel-building 13
  13. I can import my package but you can’t find it?

    Why O Why A ton of packages downloaded for no reason! PyPI should automatically build wheels! 14
  14. How did tools know what to uninstall? Trick question! They

    didn’t LOL Setuptools invented eggs (kind of like JAR) Turns out many packages don’t work inside a ZIP Let’s not ZIP them? Metadata become plain files. 15
  15. Plain files are pretty nice Metadata as package resources (data

    files) Reflection is automatic via import search path Disabling user-site, PYTHONPATH, etc. Are there any real technical downsides? 16
  16. Third System Effect First effort (distutils) went OK Second, not

    so much You probably never heard of it More difficult to gather volunteers No central driving force 18
  17. From pip to PEPs Set the ground rules for people

    to innovate Small, reusable libraries for standards Invite discussions to combine the parts 19
  18. Python Packaging Platypus Evolutions make sense when they happen Special

    advantages for particular use cases Remember the lineage 20