$30 off During Our Annual Pro Sale. View Details »

Python Packaging: Why Don’t You Just…?

Python Packaging: Why Don’t You Just…?

PackagingCon 2021

Schedule: https://pretalx.com/packagingcon-2021/talk/X8N8ME/
Recording: https://www.youtube.com/watch?v=jpP07nZiicE

Every packaging system has its specific way of doing things, but to an outsider. Python’s seems to have a knack of finding the most non-straightforward and weird solution for every choice. This talk attempts to trace some of the peculiarities to find out the reasoning behind the decisions, and how they stand in the modern packaging landscape.

———

Python packaging has a long and winding history. When it startd out, few communities have good packaging offerings, and Python’s was praised for being easy to be picked up and develop with. As time went on, many tried to bring in new ideas from relatively new packaging systems to “modernise” Python packaging, but those suggestions are not always received favourably, and Python’s packaging community is sometimes categorised as overly conservative, or even user-hostile due to this.

In most of such situations, however, the main problem is not that Python packaging is unwilling to change, but there are certain design decisions that affect how problems should be approached. While those characteristics may seem alien to newcomers, they are made to accomdate special needs in the Python community, and any changes to the packaging system need to take them into consideration. This sometimes require contributors to be more creative when putting together a new feature, but the Python community is only as valuable as its users, and new progress should avoid throwing away what enable Python to grow to what it is.

Tzu-ping Chung

November 11, 2021
Tweet

More Decks by Tzu-ping Chung

Other Decks in Programming

Transcript

  1. Python Packaging:

    Why Don’t You Just?
    Tzu-ping Chung @ PackagingCon 2021
    1

    View Slide

  2. Me
    Call me TP (uraunsjr.com)
    Taipei, Taiwan
    Apache Airflow @ Astronomer
    Python packaging (PyPA & others)
    2

    View Slide

  3. Python Packaging LOL
    So weird we must split it into topics
    How did this thing happen?
    What does the solution achieve?
    How do we improve the situation?
    3

    View Slide

  4. Why O Why
    A ton of packages downloaded for no reason!
    PyPI should automatically build wheels!
    I can import my package but you can’t find it?
    4

    View Slide

  5. A ton of packages downloaded for no reason!
    Why O Why
    PyPI should automatically build wheels!
    I can import my package but you can’t find it?
    5

    View Slide

  6. Per-artifact Metadata — History
    PyPI is just a bunch of tarballs
    Well, also zipballs
    So simple it’s called the Simple API
    Conditionals in setup.py
    Wheel contains prebuilt metadata
    6

    View Slide

  7. They may all have different metadata
    tensorflow-2.6.1-cp36-cp36m-macosx_10_14_x86_64.whl
    tensorflow-2.6.1-cp36-cp36m-manylinux2010_x86_64.whl
    tensorflow-2.6.1-cp36-cp36m-win_amd64.whl
    tensorflow-2.6.1-cp37-cp37m-macosx_10_14_x86_64.whl
    tensorflow-2.6.1-cp37-cp37m-manylinux2010_x86_64.whl
    tensorflow-2.6.1-cp37-cp37m-win_amd64.whl
    tensorflow-2.6.1-cp38-cp38-macosx_10_14_x86_64.whl
    tensorflow-2.6.1-cp38-cp38-manylinux2010_x86_64.whl
    tensorflow-2.6.1-cp38-cp38-win_amd64.whl
    tensorflow-2.6.1-cp39-cp39-macosx_10_14_x86_64.whl
    tensorflow-2.6.1-cp39-cp39-manylinux2010_x86_64.whl
    tensorflow-2.6.1-cp39-cp39-win_amd64.whl
    7

    View Slide

  8. Per-artifact Metadata — Advantages
    Every file server is a Python package index
    Just put artifacts in a directory!
    Install a package with a “direct URL”
    8

    View Slide

  9. Per-artifact Metadata — Improvements
    A way to say I don’t have dynamic metadata
    Serve metadata alongside with each artifact
    Detect and degrade gracefully
    9

    View Slide

  10. PyPI should automatically build wheels!
    Why O Why
    A ton of packages downloaded for no reason!
    I can import my package but you can’t find it?
    10

    View Slide

  11. Infinite Wheelhouse Paradox
    How many platforms does CPython support?
    Trick question! As long as it builds, it’s good
    There is a very long tail of platforms
    You can never build all wheels, theoratically
    11

    View Slide

  12. Pull-based Platform Declaration
    Basically feature detection
    Declare needs for platforms to satisfy
    Platform PEPs are Informational
    As long as your binaries work for your users
    12

    View Slide

  13. Sufficiently Large Enough
    Make platform reflection easy
    sysconfig
    Create a “common enough” template
    cibuildwheel
    Simplify and externalise wheel-building
    13

    View Slide

  14. I can import my package but you can’t find it?
    Why O Why
    A ton of packages downloaded for no reason!
    PyPI should automatically build wheels!
    14

    View Slide

  15. How did tools know what to uninstall?
    Trick question! They didn’t LOL
    Setuptools invented eggs (kind of like JAR)
    Turns out many packages don’t work inside a ZIP
    Let’s not ZIP them? Metadata become plain files.
    15

    View Slide

  16. Plain files are pretty nice
    Metadata as package resources (data files)
    Reflection is automatic via import search path
    Disabling user-site, PYTHONPATH, etc.
    Are there any real technical downsides?
    16

    View Slide

  17. None of these makes sense!
    17

    View Slide

  18. Third System Effect
    First effort (distutils) went OK
    Second, not so much
    You probably never heard of it
    More difficult to gather volunteers
    No central driving force
    18

    View Slide

  19. From pip to PEPs
    Set the ground rules for people to innovate
    Small, reusable libraries for standards
    Invite discussions to combine the parts
    19

    View Slide

  20. Python Packaging Platypus
    Evolutions make sense when they happen
    Special advantages for particular use cases
    Remember the lineage
    20

    View Slide

  21. Thank You!
    https://monotreme.club
    21

    View Slide