Python Packaging: Why Don’t You Just…?

PackagingCon 2021

Schedule: https://pretalx.com/packagingcon-2021/talk/X8N8ME/
Recording: https://www.youtube.com/watch?v=jpP07nZiicE

Every packaging system has its specific way of doing things, but to an outsider. Python’s seems to have a knack of finding the most non-straightforward and weird solution for every choice. This talk attempts to trace some of the peculiarities to find out the reasoning behind the decisions, and how they stand in the modern packaging landscape.

———

Python packaging has a long and winding history. When it startd out, few communities have good packaging offerings, and Python’s was praised for being easy to be picked up and develop with. As time went on, many tried to bring in new ideas from relatively new packaging systems to “modernise” Python packaging, but those suggestions are not always received favourably, and Python’s packaging community is sometimes categorised as overly conservative, or even user-hostile due to this.

In most of such situations, however, the main problem is not that Python packaging is unwilling to change, but there are certain design decisions that affect how problems should be approached. While those characteristics may seem alien to newcomers, they are made to accomdate special needs in the Python community, and any changes to the packaging system need to take them into consideration. This sometimes require contributors to be more creative when putting together a new feature, but the Python community is only as valuable as its users, and new progress should avoid throwing away what enable Python to grow to what it is.

Transcript

1. Python Packaging: Why Don’t You Just? Tzu-ping Chung @ PackagingCon 2021 1
2. Me Call me TP (uraunsjr.com) Taipei, Taiwan Apache Airflow @ Astronomer Python packaging (PyPA & others) 2
3. Python Packaging LOL So weird we must split it into topics How did this thing happen? What does the solution achieve? How do we improve the situation? 3
4. Why O Why A ton of packages downloaded for no reason! PyPI should automatically build wheels! I can import my package but you can’t find it? 4
5. A ton of packages downloaded for no reason! Why O Why PyPI should automatically build wheels! I can import my package but you can’t find it? 5
6. Per-artifact Metadata — History PyPI is just a bunch of tarballs Well, also zipballs So simple it’s called the Simple API Conditionals in setup.py Wheel contains prebuilt metadata 6
7. They may all have different metadata <a href="...">tensorflow-2.6.1-cp36-cp36m-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp36-cp36m-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp36-cp36m-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp37-cp37m-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp38-cp38-win_amd64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-macosx_10_14_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-manylinux2010_x86_64.whl</a> <a href="...">tensorflow-2.6.1-cp39-cp39-win_amd64.whl</a> 7
8. Per-artifact Metadata — Advantages Every file server is a Python package index Just put artifacts in a directory! Install a package with a “direct URL” 8
9. Per-artifact Metadata — Improvements A way to say I don’t have dynamic metadata Serve metadata alongside with each artifact Detect and degrade gracefully 9
10. PyPI should automatically build wheels! Why O Why A ton of packages downloaded for no reason! I can import my package but you can’t find it? 10
11. Infinite Wheelhouse Paradox How many platforms does CPython support? Trick question! As long as it builds, it’s good There is a very long tail of platforms You can never build all wheels, theoratically 11
12. Pull-based Platform Declaration Basically feature detection Declare needs for platforms to satisfy Platform PEPs are Informational As long as your binaries work for your users 12
13. Sufficiently Large Enough Make platform reflection easy sysconfig Create a “common enough” template cibuildwheel Simplify and externalise wheel-building 13
14. I can import my package but you can’t find it? Why O Why A ton of packages downloaded for no reason! PyPI should automatically build wheels! 14
15. How did tools know what to uninstall? Trick question! They didn’t LOL Setuptools invented eggs (kind of like JAR) Turns out many packages don’t work inside a ZIP Let’s not ZIP them? Metadata become plain files. 15
16. Plain files are pretty nice Metadata as package resources (data files) Reflection is automatic via import search path Disabling user-site, PYTHONPATH, etc. Are there any real technical downsides? 16
17. None of these makes sense! 17
18. Third System Effect First effort (distutils) went OK Second, not so much You probably never heard of it More difficult to gather volunteers No central driving force 18
19. From pip to PEPs Set the ground rules for people to innovate Small, reusable libraries for standards Invite discussions to combine the parts 19
20. Python Packaging Platypus Evolutions make sense when they happen Special advantages for particular use cases Remember the lineage 20
21. Thank You! https://monotreme.club 21