@ben_nuttall Running a Python Package Index for Raspberry Pi 

@ben_nuttall Ben Nuttall ● Senior Software Engineer, BBC News Labs ● Former Community Manager at Raspberry Pi ● Based in Cambridgeshire, UK ● bennuttall.com ● twitter.com/ben_nuttall ● github.com/bennuttall 

@ben_nuttall Pure Python package Binary distribution (bdist) .whl Source distribution (sdist) .zip .tar.gz 

@ben_nuttall Compiled Python package Python version 3.5 3.6 3.7 3.8 3.9 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist) 

@ben_nuttall pip 

@ben_nuttall pip 

@ben_nuttall What is piwheels? ● Open source project maintained by: – Ben Nuttall – Dave Jones ● Tooling to automate building wheels of everything on PyPI – Wheels are pre-compiled binary distributions of Python packages ● Pip-compatible index (PEP-503) – Can be used by pip as an "extra index" alongside PyPI ● Tailored towards the Arm platform – linux_armv6l and linux_armv7l platform tags 

@ben_nuttall piwheels Stretch (Python 3.5) Buster (Python 3.7) Bullseye (Python 3.9) linux_armv6l linux_armv7l 

@ben_nuttall piwheels.org ● Website comprises: ● Simple index (for pip) ● Project pages (for people) ● Blog & info pages (for people) 

@ben_nuttall Mythic Beasts Pi Cloud 

@ben_nuttall piwheels stack ● Master – Web server hosts index and wheel files – Monitors PyPI events – Logs build attempts & downloads in DB ● Postgres database – Generates build queue ● Builders – Get given build jobs – 5 × Pi 3 (Stretch / Python 3.5) – 5 × Pi 4 (Buster / Python 3.7) – 5 × Pi 4 (Bullseye / Python 3.9) – Easy to scale up/down 

@ben_nuttall Following PyPI’s event log 36195 scipy 2007-10-15 20:47:37 remove Owner edschofield 36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield 36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman 2416 scipy 0.6.0 2007-10-17 08:53:30 new release 36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden 26010 scipy 0.6.0 2007-10-17 08:57:56 update description 

@ben_nuttall Following PyPI’s event log ● We care about: – Package creation/deletion – Version creation/deletion – Source distribution upload/removal – Yank/Unyank version ● Update database ● Add builds to queue ● Remove files ● Rewrite web pages as necessary 

@ben_nuttall piwheels build queue 

@ben_nuttall Using piwheels ● Raspberry Pi OS is pre- configured to use piwheels.org as an additional index to PyPI ● Users get platform wheels without knowing it’s there 

@ben_nuttall Using piwheels ● Raspberry Pi OS is pre- configured to use piwheels.org as an additional index to PyPI ● Users get platform wheels without knowing it’s there 

@ben_nuttall Stats 

@ben_nuttall Static site build ● Once/periodically: – Homepage & single pages ● When changed: – Simple index – Package simple index (create, update and delete) – Package project page (create, update and delete) – Wheel files (create and delete) 

@ben_nuttall piwheels.org/simple/scipy/ 

@ben_nuttall piwheels.org/project/scipy/ 

@ben_nuttall apt dependencies 

@ben_nuttall apt dependencies 

@ben_nuttall apt dependencies ● ldd cv2.cpython-35m-arm-linux-gnueabihf.so – libhdf5_serial.so.100 => not found ● apt-file search libhdf5_serial.so.100 – libhdf5-100 ● sudo apt install libhdf5-100 https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/ 

@ben_nuttall apt dependencies 

@ben_nuttall JSON API ● piwheels.org/project//json – Info about a specific package (versions, builds, files, etc) ● piwheels.org/packages.json – List of all packages, with per-package download stats 

@ben_nuttall Package issues 

@ben_nuttall Package issues 

@ben_nuttall Package issues 

@ben_nuttall Running a Python Package Index for Raspberry Pi