Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Running a Python Package Index for Raspberry Pi

Running a Python Package Index for Raspberry Pi

Talk given at PackagingCon 2021

Ben Nuttall

November 10, 2021
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. @ben_nuttall Ben Nuttall • Senior Software Engineer, BBC News Labs

    • Former Community Manager at Raspberry Pi • Based in Cambridgeshire, UK • bennuttall.com • twitter.com/ben_nuttall • github.com/bennuttall
  2. @ben_nuttall Compiled Python package Python version 3.5 3.6 3.7 3.8

    3.9 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist)
  3. @ben_nuttall What is piwheels? • Open source project maintained by:

    – Ben Nuttall – Dave Jones • Tooling to automate building wheels of everything on PyPI – Wheels are pre-compiled binary distributions of Python packages • Pip-compatible index (PEP-503) – Can be used by pip as an "extra index" alongside PyPI • Tailored towards the Arm platform – linux_armv6l and linux_armv7l platform tags
  4. @ben_nuttall piwheels.org • Website comprises: • Simple index (for pip)

    • Project pages (for people) • Blog & info pages (for people)
  5. @ben_nuttall piwheels stack • Master – Web server hosts index

    and wheel files – Monitors PyPI events – Logs build attempts & downloads in DB • Postgres database – Generates build queue • Builders – Get given build jobs – 5 × Pi 3 (Stretch / Python 3.5) – 5 × Pi 4 (Buster / Python 3.7) – 5 × Pi 4 (Bullseye / Python 3.9) – Easy to scale up/down
  6. @ben_nuttall Following PyPI’s event log 36195 scipy 2007-10-15 20:47:37 remove

    Owner edschofield 36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield 36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman 2416 scipy 0.6.0 2007-10-17 08:53:30 new release 36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden 26010 scipy 0.6.0 2007-10-17 08:57:56 update description
  7. @ben_nuttall Following PyPI’s event log • We care about: –

    Package creation/deletion – Version creation/deletion – Source distribution upload/removal – Yank/Unyank version • Update database • Add builds to queue • Remove files • Rewrite web pages as necessary
  8. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured

    to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
  9. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured

    to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
  10. @ben_nuttall Static site build • Once/periodically: – Homepage & single

    pages • When changed: – Simple index – Package simple index (create, update and delete) – Package project page (create, update and delete) – Wheel files (create and delete)
  11. @ben_nuttall apt dependencies • ldd cv2.cpython-35m-arm-linux-gnueabihf.so – libhdf5_serial.so.100 => not

    found • apt-file search libhdf5_serial.so.100 – libhdf5-100 • sudo apt install libhdf5-100 https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/
  12. @ben_nuttall JSON API • piwheels.org/project/<package>/json – Info about a specific

    package (versions, builds, files, etc) • piwheels.org/packages.json – List of all packages, with per-package download stats