Running a Python Package Index for Raspberry Pi

Talk given at PackagingCon 2021

Transcript

1. @ben_nuttall
   Running a Python Package Index for
   Raspberry Pi
   

   View Slide

2. @ben_nuttall
   Ben Nuttall
   ●
   Senior Software Engineer, BBC
   News Labs
   ●
   Former Community Manager at
   Raspberry Pi
   ●
   Based in Cambridgeshire, UK
   ●
   bennuttall.com
   ●
   twitter.com/ben_nuttall
   ●
   github.com/bennuttall
   

   View Slide

3. @ben_nuttall
   Pure Python package
   Binary distribution (bdist) .whl
   Source distribution (sdist)
   .zip
   .tar.gz
   

   View Slide

4. @ben_nuttall
   Compiled Python package
   Python
   version
   3.5 3.6 3.7 3.8 3.9
   Windows (win32)
   Windows (amd64)
   Mac OS (x86_64)
   Linux (i686)
   Linux (x86_64)
   Source (sdist)
   

   View Slide

5. @ben_nuttall
   pip
   

   View Slide

6. @ben_nuttall
   pip
   

   View Slide

7. @ben_nuttall
   What is piwheels?
   ●
   Open source project maintained by:
   –
   Ben Nuttall
   –
   Dave Jones
   ●
   Tooling to automate building wheels of everything on PyPI
   –
   Wheels are pre-compiled binary distributions of Python
   packages
   ●
   Pip-compatible index (PEP-503)
   –
   Can be used by pip as an "extra index" alongside PyPI
   ●
   Tailored towards the Arm platform
   –
   linux_armv6l and linux_armv7l platform tags
   

   View Slide

8. @ben_nuttall
   piwheels
   Stretch
   (Python 3.5)
   Buster
   (Python 3.7)
   Bullseye
   (Python 3.9)
   linux_armv6l
   linux_armv7l
   

   View Slide

9. @ben_nuttall
   piwheels.org
   ●
   Website comprises:
   ●
   Simple index (for pip)
   ●
   Project pages (for people)
   ●
   Blog & info pages (for people)
   

   View Slide

10. @ben_nuttall
    Mythic Beasts Pi Cloud
    

    View Slide

11. @ben_nuttall
    piwheels stack
    ●
    Master
    –
    Web server hosts index and wheel files
    –
    Monitors PyPI events
    –
    Logs build attempts & downloads in DB
    ●
    Postgres database
    –
    Generates build queue
    ●
    Builders
    –
    Get given build jobs
    –
    5 × Pi 3 (Stretch / Python 3.5)
    –
    5 × Pi 4 (Buster / Python 3.7)
    –
    5 × Pi 4 (Bullseye / Python 3.9)
    –
    Easy to scale up/down
    

    View Slide

12. @ben_nuttall
    Following PyPI’s event log
    36195 scipy 2007-10-15 20:47:37 remove Owner edschofield
    36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield
    36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman
    2416 scipy 0.6.0 2007-10-17 08:53:30 new release
    36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden
    26010 scipy 0.6.0 2007-10-17 08:57:56 update description
    

    View Slide

13. @ben_nuttall
    Following PyPI’s event log
    ●
    We care about:
    –
    Package creation/deletion
    –
    Version creation/deletion
    –
    Source distribution upload/removal
    –
    Yank/Unyank version
    ●
    Update database
    ●
    Add builds to queue
    ●
    Remove files
    ●
    Rewrite web pages as necessary
    

    View Slide

14. @ben_nuttall
    piwheels build queue
    

    View Slide

15. @ben_nuttall
    Using piwheels
    ●
    Raspberry Pi OS is pre-
    configured to use piwheels.org
    as an additional index to PyPI
    ●
    Users get platform wheels
    without knowing it’s there
    

    View Slide

16. @ben_nuttall
    Using piwheels
    ●
    Raspberry Pi OS is pre-
    configured to use piwheels.org
    as an additional index to PyPI
    ●
    Users get platform wheels
    without knowing it’s there
    

    View Slide

17. @ben_nuttall
    Stats
    

    View Slide

18. @ben_nuttall
    Static site build
    ●
    Once/periodically:
    –
    Homepage & single pages
    ●
    When changed:
    –
    Simple index
    –
    Package simple index (create, update and delete)
    –
    Package project page (create, update and delete)
    –
    Wheel files (create and delete)
    

    View Slide

19. @ben_nuttall
    piwheels.org/simple/scipy/
    

    View Slide

20. @ben_nuttall
    piwheels.org/project/scipy/
    

    View Slide

21. @ben_nuttall
    apt dependencies
    

    View Slide

22. @ben_nuttall
    apt dependencies
    

    View Slide

23. @ben_nuttall
    apt dependencies
    ● ldd cv2.cpython-35m-arm-linux-gnueabihf.so
    – libhdf5_serial.so.100 => not found
    ● apt-file search libhdf5_serial.so.100
    – libhdf5-100
    ● sudo apt install libhdf5-100
    https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/
    

    View Slide

24. @ben_nuttall
    apt dependencies
    

    View Slide

25. @ben_nuttall
    JSON API
    ●
    piwheels.org/project//json
    –
    Info about a specific package (versions, builds, files, etc)
    ●
    piwheels.org/packages.json
    –
    List of all packages, with per-package download stats
    

    View Slide

26. @ben_nuttall
    Package issues
    

    View Slide

27. @ben_nuttall
    Package issues
    

    View Slide

28. @ben_nuttall
    Package issues
    

    View Slide

29. @ben_nuttall
    Running a Python Package Index for
    Raspberry Pi
    

    View Slide