Me
Call me TP (uraunsjr.com)
Taipei, Taiwan
Apache Airflow @ Astronomer
Python packaging (PyPA & others)
2
Slide 3
Slide 3 text
Python Packaging LOL
So weird we must split it into topics
How did this thing happen?
What does the solution achieve?
How do we improve the situation?
3
Slide 4
Slide 4 text
Why O Why
A ton of packages downloaded for no reason!
PyPI should automatically build wheels!
I can import my package but you can’t find it?
4
Slide 5
Slide 5 text
A ton of packages downloaded for no reason!
Why O Why
PyPI should automatically build wheels!
I can import my package but you can’t find it?
5
Slide 6
Slide 6 text
Per-artifact Metadata — History
PyPI is just a bunch of tarballs
Well, also zipballs
So simple it’s called the Simple API
Conditionals in setup.py
Wheel contains prebuilt metadata
6
Per-artifact Metadata — Advantages
Every file server is a Python package index
Just put artifacts in a directory!
Install a package with a “direct URL”
8
Slide 9
Slide 9 text
Per-artifact Metadata — Improvements
A way to say I don’t have dynamic metadata
Serve metadata alongside with each artifact
Detect and degrade gracefully
9
Slide 10
Slide 10 text
PyPI should automatically build wheels!
Why O Why
A ton of packages downloaded for no reason!
I can import my package but you can’t find it?
10
Slide 11
Slide 11 text
Infinite Wheelhouse Paradox
How many platforms does CPython support?
Trick question! As long as it builds, it’s good
There is a very long tail of platforms
You can never build all wheels, theoratically
11
Slide 12
Slide 12 text
Pull-based Platform Declaration
Basically feature detection
Declare needs for platforms to satisfy
Platform PEPs are Informational
As long as your binaries work for your users
12
Slide 13
Slide 13 text
Sufficiently Large Enough
Make platform reflection easy
sysconfig
Create a “common enough” template
cibuildwheel
Simplify and externalise wheel-building
13
Slide 14
Slide 14 text
I can import my package but you can’t find it?
Why O Why
A ton of packages downloaded for no reason!
PyPI should automatically build wheels!
14
Slide 15
Slide 15 text
How did tools know what to uninstall?
Trick question! They didn’t LOL
Setuptools invented eggs (kind of like JAR)
Turns out many packages don’t work inside a ZIP
Let’s not ZIP them? Metadata become plain files.
15
Slide 16
Slide 16 text
Plain files are pretty nice
Metadata as package resources (data files)
Reflection is automatic via import search path
Disabling user-site, PYTHONPATH, etc.
Are there any real technical downsides?
16
Slide 17
Slide 17 text
None of these makes sense!
17
Slide 18
Slide 18 text
Third System Effect
First effort (distutils) went OK
Second, not so much
You probably never heard of it
More difficult to gather volunteers
No central driving force
18
Slide 19
Slide 19 text
From pip to PEPs
Set the ground rules for people to innovate
Small, reusable libraries for standards
Invite discussions to combine the parts
19
Slide 20
Slide 20 text
Python Packaging Platypus
Evolutions make sense when they happen
Special advantages for particular use cases
Remember the lineage
20