Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Running a Python Package Index for Raspberry Pi

Running a Python Package Index for Raspberry Pi

Talk given at PackagingCon 2021

Ben Nuttall

November 10, 2021
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. @ben_nuttall
    Running a Python Package Index for
    Raspberry Pi

    View Slide

  2. @ben_nuttall
    Ben Nuttall

    Senior Software Engineer, BBC
    News Labs

    Former Community Manager at
    Raspberry Pi

    Based in Cambridgeshire, UK

    bennuttall.com

    twitter.com/ben_nuttall

    github.com/bennuttall

    View Slide

  3. @ben_nuttall
    Pure Python package
    Binary distribution (bdist) .whl
    Source distribution (sdist)
    .zip
    .tar.gz

    View Slide

  4. @ben_nuttall
    Compiled Python package
    Python
    version
    3.5 3.6 3.7 3.8 3.9
    Windows (win32)
    Windows (amd64)
    Mac OS (x86_64)
    Linux (i686)
    Linux (x86_64)
    Source (sdist)

    View Slide

  5. @ben_nuttall
    pip

    View Slide

  6. @ben_nuttall
    pip

    View Slide

  7. @ben_nuttall
    What is piwheels?

    Open source project maintained by:

    Ben Nuttall

    Dave Jones

    Tooling to automate building wheels of everything on PyPI

    Wheels are pre-compiled binary distributions of Python
    packages

    Pip-compatible index (PEP-503)

    Can be used by pip as an "extra index" alongside PyPI

    Tailored towards the Arm platform

    linux_armv6l and linux_armv7l platform tags

    View Slide

  8. @ben_nuttall
    piwheels
    Stretch
    (Python 3.5)
    Buster
    (Python 3.7)
    Bullseye
    (Python 3.9)
    linux_armv6l
    linux_armv7l

    View Slide

  9. @ben_nuttall
    piwheels.org

    Website comprises:

    Simple index (for pip)

    Project pages (for people)

    Blog & info pages (for people)

    View Slide

  10. @ben_nuttall
    Mythic Beasts Pi Cloud

    View Slide

  11. @ben_nuttall
    piwheels stack

    Master

    Web server hosts index and wheel files

    Monitors PyPI events

    Logs build attempts & downloads in DB

    Postgres database

    Generates build queue

    Builders

    Get given build jobs

    5 × Pi 3 (Stretch / Python 3.5)

    5 × Pi 4 (Buster / Python 3.7)

    5 × Pi 4 (Bullseye / Python 3.9)

    Easy to scale up/down

    View Slide

  12. @ben_nuttall
    Following PyPI’s event log
    36195 scipy 2007-10-15 20:47:37 remove Owner edschofield
    36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield
    36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman
    2416 scipy 0.6.0 2007-10-17 08:53:30 new release
    36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden
    26010 scipy 0.6.0 2007-10-17 08:57:56 update description

    View Slide

  13. @ben_nuttall
    Following PyPI’s event log

    We care about:

    Package creation/deletion

    Version creation/deletion

    Source distribution upload/removal

    Yank/Unyank version

    Update database

    Add builds to queue

    Remove files

    Rewrite web pages as necessary

    View Slide

  14. @ben_nuttall
    piwheels build queue

    View Slide

  15. @ben_nuttall
    Using piwheels

    Raspberry Pi OS is pre-
    configured to use piwheels.org
    as an additional index to PyPI

    Users get platform wheels
    without knowing it’s there

    View Slide

  16. @ben_nuttall
    Using piwheels

    Raspberry Pi OS is pre-
    configured to use piwheels.org
    as an additional index to PyPI

    Users get platform wheels
    without knowing it’s there

    View Slide

  17. @ben_nuttall
    Stats

    View Slide

  18. @ben_nuttall
    Static site build

    Once/periodically:

    Homepage & single pages

    When changed:

    Simple index

    Package simple index (create, update and delete)

    Package project page (create, update and delete)

    Wheel files (create and delete)

    View Slide

  19. @ben_nuttall
    piwheels.org/simple/scipy/

    View Slide

  20. @ben_nuttall
    piwheels.org/project/scipy/

    View Slide

  21. @ben_nuttall
    apt dependencies

    View Slide

  22. @ben_nuttall
    apt dependencies

    View Slide

  23. @ben_nuttall
    apt dependencies
    ● ldd cv2.cpython-35m-arm-linux-gnueabihf.so
    – libhdf5_serial.so.100 => not found
    ● apt-file search libhdf5_serial.so.100
    – libhdf5-100
    ● sudo apt install libhdf5-100
    https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/

    View Slide

  24. @ben_nuttall
    apt dependencies

    View Slide

  25. @ben_nuttall
    JSON API

    piwheels.org/project//json

    Info about a specific package (versions, builds, files, etc)

    piwheels.org/packages.json

    List of all packages, with per-package download stats

    View Slide

  26. @ben_nuttall
    Package issues

    View Slide

  27. @ben_nuttall
    Package issues

    View Slide

  28. @ben_nuttall
    Package issues

    View Slide

  29. @ben_nuttall
    Running a Python Package Index for
    Raspberry Pi

    View Slide