piwheels: a Python package repository for Raspberry Pi

piwheels: a Python package repository for Raspberry Pi

Talk given at remote python pizza April 2020

A1995c9abe48450ce2f82d93ca0b863f?s=128

Ben Nuttall

April 25, 2020
Tweet

Transcript

  1. piwheels: a Python package repository for Raspberry Pi Ben Nuttall

    twitter.com/ben_nuttall
  2. @ben_nuttall Ben Nuttall • Software engineer at BBC News Labs

    • Formerly at Raspberry Pi Foundation • Creator of gpiozero, piwheels and pyjokes • Opensource.com correspondent • twitter.com/ben_nuttall • github.com/bennuttall
  3. @ben_nuttall Why piwheels? • Before piwheels, “pip install” on Raspberry

    Pi would take ages • People don’t generally build Arm platform wheels • Just x86/x86_64 for Linux, Mac & Windows • I wondered if it would be possible to build a repository to host Arm wheels • It is • So I did
  4. @ben_nuttall Uploading Pure Python package to PyPI Python version 2/3

    Universal Python wheel Source (sdist)
  5. @ben_nuttall Uploading Python package with C extensions to PyPI Python

    version 2.7 3.5 3.6 3.7 3.8 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist)
  6. @ben_nuttall What is piwheels? • Open source project maintained by:

    • Ben Nuttall • Dave Jones • Tooling to automate building wheels of everything on PyPI • Wheels are pre-compiled binary distributions of Python packages • Tailored towards the Arm platform • linux_armv6l and linux_armv7l platform tags
  7. @ben_nuttall What is piwheels.org? • Python package repository (like PyPI)

    • Hosts Arm wheels of all packages on PyPI • Website comprises: • Simple index • Project pages • Blog & info pages
  8. @ben_nuttall What is piwheels.org? Python version 3.4 3.5 3.7 linux_armv6l

    linux_armv7l
  9. @ben_nuttall I assume you cross-compile? Nope

  10. @ben_nuttall Mythic Beasts Pi cloud

  11. @ben_nuttall The stack • Master • Distributes jobs, web server

    hosts index and wheel files • Postgres database • Monitors PyPI, generates build queue • Logs build attempts, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • 2 × Raspbian Buster (Python 3.7) • Easy to scale up/down
  12. @ben_nuttall The stack • Master • Distributes jobs, web server

    hosts index and wheel files • Postgres database • Monitors PyPI, generates build queue • Logs build attempts, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • 2 × Raspbian Buster (Python 3.7) • Easy to scale up/down
  13. @ben_nuttall piwheels builds

  14. @ben_nuttall Using piwheels • Raspbian is pre-configured to use piwheels.org

    as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
  15. @ben_nuttall Using piwheels • Raspbian is pre-configured to use piwheels.org

    as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
  16. @ben_nuttall piwheels.org/simple/scipy/

  17. @ben_nuttall piwheels.org/project/scipy/

  18. @ben_nuttall How many downloads? • 2018: 5 million • 2019:

    9 million • 2020 so far: 4 million • To date: 19 million • March 2020: 1.2 million • Yesterday: 40,000
  19. @ben_nuttall How much time saved? • 2018: 42 years •

    2019: 128 years • 2020 so far: 76 years • To date: 250 years • March 2020: 22 years • Yesterday: 262 days
  20. @ben_nuttall Dependency calculation

  21. @ben_nuttall Dependency calculation

  22. @ben_nuttall What’s next for piwheels? • Implement new PEPs •

    Requires-Python • Yank • Deploy JSON API • Package information at piwheels.org/project/<project>/json • Keep fixing broken builds • Get ready for Debian Bullseye (Python ~3.9) in 2021
  23. @ben_nuttall piwheels • piwheels.org • github.com/piwheels • twitter.com/piwheels • blog.piwheels.org

    • Ben: twitter.com/ben_nuttall • Dave: twitter.com/waveform80