Slide 1

Slide 1 text

Python Packaging: Why Don’t You Just? Tzu-ping Chung @ PackagingCon 2021 1

Slide 2

Slide 2 text

Me Call me TP (uraunsjr.com) Taipei, Taiwan Apache Airflow @ Astronomer Python packaging (PyPA & others) 2

Slide 3

Slide 3 text

Python Packaging LOL So weird we must split it into topics How did this thing happen? What does the solution achieve? How do we improve the situation? 3

Slide 4

Slide 4 text

Why O Why A ton of packages downloaded for no reason! PyPI should automatically build wheels! I can import my package but you can’t find it? 4

Slide 5

Slide 5 text

A ton of packages downloaded for no reason! Why O Why PyPI should automatically build wheels! I can import my package but you can’t find it? 5

Slide 6

Slide 6 text

Per-artifact Metadata — History PyPI is just a bunch of tarballs Well, also zipballs So simple it’s called the Simple API Conditionals in setup.py Wheel contains prebuilt metadata 6

Slide 8

Slide 8 text

Per-artifact Metadata — Advantages Every file server is a Python package index Just put artifacts in a directory! Install a package with a “direct URL” 8

Slide 9

Slide 9 text

Per-artifact Metadata — Improvements A way to say I don’t have dynamic metadata Serve metadata alongside with each artifact Detect and degrade gracefully 9

Slide 10

Slide 10 text

PyPI should automatically build wheels! Why O Why A ton of packages downloaded for no reason! I can import my package but you can’t find it? 10

Slide 11

Slide 11 text

Infinite Wheelhouse Paradox How many platforms does CPython support? Trick question! As long as it builds, it’s good There is a very long tail of platforms You can never build all wheels, theoratically 11

Slide 12

Slide 12 text

Pull-based Platform Declaration Basically feature detection Declare needs for platforms to satisfy Platform PEPs are Informational As long as your binaries work for your users 12

Slide 13

Slide 13 text

Sufficiently Large Enough Make platform reflection easy sysconfig Create a “common enough” template cibuildwheel Simplify and externalise wheel-building 13

Slide 14

Slide 14 text

I can import my package but you can’t find it? Why O Why A ton of packages downloaded for no reason! PyPI should automatically build wheels! 14

Slide 15

Slide 15 text

How did tools know what to uninstall? Trick question! They didn’t LOL Setuptools invented eggs (kind of like JAR) Turns out many packages don’t work inside a ZIP Let’s not ZIP them? Metadata become plain files. 15

Slide 16

Slide 16 text

Plain files are pretty nice Metadata as package resources (data files) Reflection is automatic via import search path Disabling user-site, PYTHONPATH, etc. Are there any real technical downsides? 16

Slide 17

Slide 17 text

None of these makes sense! 17

Slide 18

Slide 18 text

Third System Effect First effort (distutils) went OK Second, not so much You probably never heard of it More difficult to gather volunteers No central driving force 18

Slide 19

Slide 19 text

From pip to PEPs Set the ground rules for people to innovate Small, reusable libraries for standards Invite discussions to combine the parts 19

Slide 20

Slide 20 text

Python Packaging Platypus Evolutions make sense when they happen Special advantages for particular use cases Remember the lineage 20

Slide 21

Slide 21 text

Thank You! https://monotreme.club 21