Save 37% off PRO during our Black Friday Sale! »

Running a Python Package Index for Raspberry Pi

Running a Python Package Index for Raspberry Pi

Talk given at PackagingCon 2021

A1995c9abe48450ce2f82d93ca0b863f?s=128

Ben Nuttall

November 10, 2021
Tweet

Transcript

  1. @ben_nuttall Running a Python Package Index for Raspberry Pi

  2. @ben_nuttall Ben Nuttall • Senior Software Engineer, BBC News Labs

    • Former Community Manager at Raspberry Pi • Based in Cambridgeshire, UK • bennuttall.com • twitter.com/ben_nuttall • github.com/bennuttall
  3. @ben_nuttall Pure Python package Binary distribution (bdist) .whl Source distribution

    (sdist) .zip .tar.gz
  4. @ben_nuttall Compiled Python package Python version 3.5 3.6 3.7 3.8

    3.9 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist)
  5. @ben_nuttall pip

  6. @ben_nuttall pip

  7. @ben_nuttall What is piwheels? • Open source project maintained by:

    – Ben Nuttall – Dave Jones • Tooling to automate building wheels of everything on PyPI – Wheels are pre-compiled binary distributions of Python packages • Pip-compatible index (PEP-503) – Can be used by pip as an "extra index" alongside PyPI • Tailored towards the Arm platform – linux_armv6l and linux_armv7l platform tags
  8. @ben_nuttall piwheels Stretch (Python 3.5) Buster (Python 3.7) Bullseye (Python

    3.9) linux_armv6l linux_armv7l
  9. @ben_nuttall piwheels.org • Website comprises: • Simple index (for pip)

    • Project pages (for people) • Blog & info pages (for people)
  10. @ben_nuttall Mythic Beasts Pi Cloud

  11. @ben_nuttall piwheels stack • Master – Web server hosts index

    and wheel files – Monitors PyPI events – Logs build attempts & downloads in DB • Postgres database – Generates build queue • Builders – Get given build jobs – 5 × Pi 3 (Stretch / Python 3.5) – 5 × Pi 4 (Buster / Python 3.7) – 5 × Pi 4 (Bullseye / Python 3.9) – Easy to scale up/down
  12. @ben_nuttall Following PyPI’s event log 36195 scipy 2007-10-15 20:47:37 remove

    Owner edschofield 36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield 36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman 2416 scipy 0.6.0 2007-10-17 08:53:30 new release 36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden 26010 scipy 0.6.0 2007-10-17 08:57:56 update description
  13. @ben_nuttall Following PyPI’s event log • We care about: –

    Package creation/deletion – Version creation/deletion – Source distribution upload/removal – Yank/Unyank version • Update database • Add builds to queue • Remove files • Rewrite web pages as necessary
  14. @ben_nuttall piwheels build queue

  15. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured

    to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
  16. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured

    to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
  17. @ben_nuttall Stats

  18. @ben_nuttall Static site build • Once/periodically: – Homepage & single

    pages • When changed: – Simple index – Package simple index (create, update and delete) – Package project page (create, update and delete) – Wheel files (create and delete)
  19. @ben_nuttall piwheels.org/simple/scipy/

  20. @ben_nuttall piwheels.org/project/scipy/

  21. @ben_nuttall apt dependencies

  22. @ben_nuttall apt dependencies

  23. @ben_nuttall apt dependencies • ldd cv2.cpython-35m-arm-linux-gnueabihf.so – libhdf5_serial.so.100 => not

    found • apt-file search libhdf5_serial.so.100 – libhdf5-100 • sudo apt install libhdf5-100 https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/
  24. @ben_nuttall apt dependencies

  25. @ben_nuttall JSON API • piwheels.org/project/<package>/json – Info about a specific

    package (versions, builds, files, etc) • piwheels.org/packages.json – List of all packages, with per-package download stats
  26. @ben_nuttall Package issues

  27. @ben_nuttall Package issues

  28. @ben_nuttall Package issues

  29. @ben_nuttall Running a Python Package Index for Raspberry Pi