Why I run my own Python package repository

Lightning talk given at PyCon UK about piwheels.org

Transcript

1. Why I run my own Python package repository Ben Nuttall
2. pip install numpy • Browse PyPI • Look for wheel with a matching platform tag • If one is found, download and install • If no match is found, download and build from source
3. What kind of x86 are you?
4. ...I have them all
5. Actually I’m linux_armv7l
6. What? I’ve never heard of it. Here’s the source. Build it yourself! • pip install numpy • ~20 mins on Pi 3 • ~2.5 hours on Pi 1
7. Fine! I’ll build my own package repository... • cd Projects • mkdir piwheels • cd piwheels • git init
8. pip wheel <package> • pip wheel numpy • numpy-1.15.1-cp35-cp35m-linux_armv7l.whl • That, but for all packages • 175,000 packages • That, but for all versions of all packages • 1,200,000 package versions • That, but for Python 3.4 and 3.5...
9. I assume you cross-compile? Nope
10. Mythic Beasts Pi cloud
11. The stack • Master • Distributes jobs, hosts wheel fles and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
12. The stack • Master • Distributes jobs, hosts wheel fles and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
13. Using piwheels • Raspbian Stretch is pre-confgured to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
14. Tensorfow • Ofcial Tensorfow support for Raspberry Pi • Build not automated (no sdist provided) • Wheels provided by Google, imported into piwheels
15. piwheels usage stats • 940,000 wheels • 400-500k downloads per month • Total of 23 years of build time saved in 10 months • See piwheels.org for more info