piwheels: a Python package repository for Raspberry Pi

Talk given at remote python pizza April 2020

Transcript

1. piwheels: a Python package repository for Raspberry Pi Ben Nuttall twitter.com/ben_nuttall
2. @ben_nuttall Ben Nuttall • Software engineer at BBC News Labs • Formerly at Raspberry Pi Foundation • Creator of gpiozero, piwheels and pyjokes • Opensource.com correspondent • twitter.com/ben_nuttall • github.com/bennuttall
3. @ben_nuttall Why piwheels? • Before piwheels, “pip install” on Raspberry Pi would take ages • People don’t generally build Arm platform wheels • Just x86/x86_64 for Linux, Mac & Windows • I wondered if it would be possible to build a repository to host Arm wheels • It is • So I did
4. @ben_nuttall Uploading Pure Python package to PyPI Python version 2/3 Universal Python wheel Source (sdist)
5. @ben_nuttall Uploading Python package with C extensions to PyPI Python version 2.7 3.5 3.6 3.7 3.8 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist)
6. @ben_nuttall What is piwheels? • Open source project maintained by: • Ben Nuttall • Dave Jones • Tooling to automate building wheels of everything on PyPI • Wheels are pre-compiled binary distributions of Python packages • Tailored towards the Arm platform • linux_armv6l and linux_armv7l platform tags
7. @ben_nuttall What is piwheels.org? • Python package repository (like PyPI) • Hosts Arm wheels of all packages on PyPI • Website comprises: • Simple index • Project pages • Blog & info pages
8. @ben_nuttall What is piwheels.org? Python version 3.4 3.5 3.7 linux_armv6l linux_armv7l
9. @ben_nuttall I assume you cross-compile? Nope
10. @ben_nuttall Mythic Beasts Pi cloud
11. @ben_nuttall The stack • Master • Distributes jobs, web server hosts index and wheel files • Postgres database • Monitors PyPI, generates build queue • Logs build attempts, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • 2 × Raspbian Buster (Python 3.7) • Easy to scale up/down
12. @ben_nuttall The stack • Master • Distributes jobs, web server hosts index and wheel files • Postgres database • Monitors PyPI, generates build queue • Logs build attempts, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • 2 × Raspbian Buster (Python 3.7) • Easy to scale up/down
13. @ben_nuttall piwheels builds
14. @ben_nuttall Using piwheels • Raspbian is pre-configured to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
15. @ben_nuttall Using piwheels • Raspbian is pre-configured to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
16. @ben_nuttall piwheels.org/simple/scipy/
17. @ben_nuttall piwheels.org/project/scipy/
18. @ben_nuttall How many downloads? • 2018: 5 million • 2019: 9 million • 2020 so far: 4 million • To date: 19 million • March 2020: 1.2 million • Yesterday: 40,000
19. @ben_nuttall How much time saved? • 2018: 42 years • 2019: 128 years • 2020 so far: 76 years • To date: 250 years • March 2020: 22 years • Yesterday: 262 days
20. @ben_nuttall Dependency calculation
21. @ben_nuttall Dependency calculation
22. @ben_nuttall What’s next for piwheels? • Implement new PEPs • Requires-Python • Yank • Deploy JSON API • Package information at piwheels.org/project/<project>/json • Keep fixing broken builds • Get ready for Debian Bullseye (Python ~3.9) in 2021
23. @ben_nuttall piwheels • piwheels.org • github.com/piwheels • twitter.com/piwheels • blog.piwheels.org • Ben: twitter.com/ben_nuttall • Dave: twitter.com/waveform80