Upgrade to Pro — share decks privately, control downloads, hide ads and more …

piwheels - campug

piwheels - campug

piwheels: building a faster Python package repository for Raspberry Pi users

Talk given at campug on 1 August 2017

Ben Nuttall

August 01, 2017

More Decks by Ben Nuttall

Other Decks in Programming


  1. piwheels: building a faster Python package repository for Raspberry Pi

    users Ben Nuttall Raspberry Pi Foundation UK Charity 1129409
  2. Ben Nuttall • Raspberry Pi Community Manager • Columnist on

    opensource.com • github.com/bennuttall • twitter.com/ben_nuttall • [email protected]
  3. Dave • Dave: I need your help. Can we install

    these libraries in space? • Me: I think so, if we build wheels of them.
  4. PyPI • Python package repository hosted at pypi.python.org • Install

    packages with “pip install” • Packages can be implemented in Python or C • Packages implemented in C require building – This can take a long time
  5. Python wheels • Wheels can be uploaded to PyPI alongside

    source distributions to save users from building themselves • Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 • A recent addition allowed “manylinux” wheels to be uploaded • Most maintainers don't bother uploading wheels, but popular build-intensive packages tend to
  6. Python wheels • When you build a wheel, the filename

    is made up of a number of parts: – The package name – The package version – The Python tag (e.g. cp34) – The ABI tag (e.g. cp34m) – The platform tag (e.g. linux_armv7l) • e.g: numpy-1.13.1-cp34-cp34m-linux_armv7l.whl
  7. pip install numpy • Browse http://pypi.python.org/simple/numpy • Look for a

    wheel file with a platform tag matching the current platform...
  8. What? I’ve never heard of it. Here’s the source, build

    it yourself! • PyPI doesn’t support uploading ARM wheels :( • “pip install numpy” takes: – ~20 mins on Pi 3 (1.2GHz quad-core) – ~2.5 hours on Pi 1 (700MHz single-core)
  9. Fine! I’ll build my own package repository... • cd Projects

    • mkdir piwheels • cd piwheels • git init
  10. piwheels • I could build everything on PyPI and host

    my own repository – “pip wheel numpy” works on a Pi – you can distribute the wheel • I want to log builds and store output in a database – I’d better write it in Python, not bash – Stack overflow says “import pip” is a thing • I need a list of all packages in PyPI – Stack overflow says PyPI provides an xmlrpclib interface (whatever that is) and gives a Python example • Can I host a package repository? – At minimum, an Apache directory listing will do the trick
  11. piwheels v1 • Pi 3 in my living room •

    Build the latest version of every package (106k packages) • Log output into postgres database • Host a package repository on the same Pi • On GitHub but not really reproducable
  12. piwheels v1: the results • It took 10 days to

    complete the build run • 76% build success rate • Repository live at piwheels.bennuttall.com • “pip install numpy ­i http://piwheels.bennuttall.com” works and takes 6 seconds :) • Still running on a Pi 3 in my living room • Proof of concept: it works, it’s probably useful
  13. Warehouse • Next-gen PyPI project at pypi.org • A work

    in progress – it mostly works now • Changes made on pypi.org are made on pypi.python.org
  14. Warehouse • A Google developer working on a Raspberry Pi

    related Python package came across piwheels • Asked if I’m going to try to get maintainers to upload my wheels to PyPI • I said “they can’t” • He filed an issue with warehouse, suggesting they allow ARM platform wheels • They said “ok”
  15. Warehouse • Maintainers can now upload ARMv6 and ARMv7 wheels

    to pypi.io and they appear on pypi.python.org!
  16. Planning piwheels v2 • Build every version of every package

    • Keep up with new releases automatically • Host a package repository as before • Test suite • Provide installation instructions & developer documentation so people can contribute
  17. 750,000 releases • Now 113k packages • 750k package versions

    to build • At the previous rate, this will take 70 days • Left it running for a couple of weeks: – Actually running slower than before due to network filesystem – New estimate: 100 days
  18. Why don’t you just cross-compile? • It’s not all about

    speed • Reliability • Compatability • Familiarity • Ease of use • I can scale up Pis easily • Eating my own dog food
  19. Adding more Pis • Provisioned a second Pi – No

    web server or database – Connected to the database on first Pi – rsync files to first Pi • Provisioned a third Pi • Internalled “terminator” • Provisioned Pis 4 and 5 • This is easy. I’ll be done in no time!
  20. It didn’t scale • Output on 3 Pis was about

    the same as the output on 1 Pi • The database was getting hammered
  21. Dave Jones • Author of picamera • Co-author of GPIO

    Zero • Self-professed SQL know-it- all • We’ve worked together on open source projects a lot
  22. Make it scale! • Pull request: – Query optimisations –

    Queuing system with zeromq • Re-deployed the code • Original Pi is now “master” running database and web server only • Other Pis are now “builders” using master’s database and rsync- ing files to master
  23. 20 Raspberry Pis • ~6k packages per hour • ~120k

    per day (15%) • Now also logging which Pi built each package • Pis seem to be holding up • Dropped rsync in favour of sshfs • It’s going well! I’ll be done in no time!
  24. It’s done! • Scaled down to 5 Pis to keep

    up with new releases • Thanks, Pete! You can have them back now.
  25. The results • Total packages: 113, 649 • Package versions:

    752, 817 • Build success rate: 76% • Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: • 16 days with 1 Pi building • 10 days with up to 19 Pis building • Total disk usage from wheels: 250GB
  26. pypi.org • Next generation PyPI project • A Google developer

    came across piwheels and filed an issue with the pypi.org project (warehouse) – github.com/pypa/warehouse/issues/2003 • pypi.org now supports uploading ARM wheels :) – Thanks @kpayson64 and @dstufft • Package maintainers can upload wheels built by piwheels to pypi.org and they appear on pypi.python.org – \o/
  27. Tags • ABI tag: – none: 715, 081 – cp34m:

    15, 548 – noabi: 1 (oddly, this was the wheel package) • Platform tag: – any: 715, 062 – linux_armv7l: 15, 561 – manylinux_armv7l: 6 – noarch: 1 (again, wheel)
  28. People do stupid things • Random files created in my

    home directory • Random stuff appended to my .bashrc • Some people run “git clone” in their setup.py • Inadvertently importing numpy
  29. In the future • Continue to build all new releases

    on a small number of buider Pis • Add SSL to web domain • Create individual package pages with build output • Install key dependencies and try to fix failed builds • Rebuild Python 3.4 wheels for 3.5 & 3.6 • Ensure ARMv6 wheels are available too – You can rename ARMv7 wheels and they work on ARMv6... • Add the piwheels server to pip config in Raspbian (our distro) as an additional index – Users get wheels for free without needing to know about it