piwheels - pyconuk

piwheels - pyconuk

piwheels talk given at PyCon UK 2017

A1995c9abe48450ce2f82d93ca0b863f?s=128

Ben Nuttall

October 29, 2017
Tweet

Transcript

  1. piwheels: building a faster Python package repository for Raspberry Pi

    users Ben Nuttall Raspberry Pi Foundation UK Charity 1129409
  2. Ben Nuttall • Raspberry Pi Community Manager • Columnist for

    opensource.com • github.com/bennuttall • twitter.com/ben_nuttall • ben@raspberrypi.org
  3. pip install numpy

  4. Python wheels • Wheels are built distributions, and can be

    uploaded to PyPI alongside source distributions • This saves users from building themselves • Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 • A recent addition allowed “manylinux” wheels to be uploaded • Raspberry Pi is not the “manylinux” architecture, it’s ARM
  5. ARM wheels • Technically... – Pi 3 is ARMv8 –

    Pi 2 is ARMv7 – Pi 1 / Zero are ARMv6 • But... – Wheels built on Pi 2/3 are tagged “armv7l” – Wheels built on Pi 1 / Zero are tagged “armv6l” • And… – They are actually ARMv6 wheels, and they’re all the same • So… – A wheel built on a Pi 3 will work on a Pi 2, as is – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”)
  6. pip install <package>

  7. Which type of x86/x86_64 are you?

  8. ...I have them all

  9. Actually I’m linux_armv7l

  10. What? I’ve never heard of it. Here’s the source, build

    it yourself! • ~20 mins on Pi 3 (1.2GHz quad-core) • ~2.5 hours on Pi 1 (700MHz single-core)
  11. Fine! I’ll build my own package repository... • cd Projects

    • mkdir piwheels • cd piwheels • git init
  12. piwheels • I could build everything on PyPI and host

    my own repository – “pip wheel numpy” works on a Pi – You can distribute the wheel and it’s a super fast install • Can I host my own package repository? – Apparently, yes! – At minimum, an Apache directory listing will do the trick
  13. piwheels v1 • Pi 3 in my living room •

    Build the latest version of every package (106k packages at the time) • Log output into postgres database • Host a package repository on the same Pi
  14. piwheels v1: the results • It took 10 days to

    complete the build run • 76% build success rate • Repository (was) live at piwheels.bennuttall.com • “pip install numpy ­i http://piwheels.bennuttall.com” works and takes 6 seconds :) • Proof of concept: it works, it’s probably useful
  15. Planning piwheels v2 • Build every version of every package

    • Keep up with new releases automatically • Host a package repository as before • Create a test suite • Provide installation instructions & developer documentation so people can contribute
  16. Planning piwheels v2 • Now 113k packages on PyPI •

    But now I’m building every version of every package • 750k package versions to build • At the previous rate, this will take 70 days on one Pi… – Maybe I could use … more than one Pi?
  17. Mythic Beasts: Pi in the cloud

  18. There is no Raspberry Pi cloud...

  19. Pete

  20. None
  21. piwheels v2 • One Pi to start • Left it

    running for a couple of weeks: – Actually running slower than before due to NFS – New estimate: 100 days
  22. Lightning talk at EuroPython

  23. Why don’t you just cross-compile? • It’s not all about

    speed • Reliability • Compatability • Familiarity • Ease of use • I can scale up Pis easily • Eating my own dog food
  24. None
  25. None
  26. ALL the cloud Pis

  27. Adding more Pis • Provisioned a second Pi – No

    web server or database – Connected to the database on first Pi – rsync files to first Pi • Provisioned a third Pi • Installed “terminator” • Provisioned Pis 4 and 5 • This is easy. I’ll be done in no time!
  28. None
  29. It didn’t scale • Output on 3 Pis was about

    the same as the output on 1 Pi • The database was getting hammered
  30. Dave Jones • Author of picamera • Co-author of GPIO

    Zero • Self-professed SQL know-it- all • We’ve worked together on open source projects a lot
  31. Make it scale! • Pull request: – Query optimisations –

    Queuing system with zeromq • Re-deployed the code • Original Pi is now “master” running database and web server only • Other Pis are now “builders” using master’s database and rsync- ing files to master
  32. It worked! Keep going!

  33. 20 Raspberry Pis • ~3k packages per hour • ~72k

    per day (10%) • Now also logging which Pi built each package • Pis seem to be holding up • Dropped rsync in favour of sshfs • It’s going well! I’ll be done in no time!
  34. People do stupid things • Random files created in my

    home directory • Random stuff appended to my .bashrc • Some people run “git clone” in their setup.py • Inadvertently importing numpy
  35. The results • Total packages processed: 113, 649 • Package

    versions built: 570, 648 / 752, 817 (76%) • Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: • 16 days with 1 Pi building • 10 days with up to 19 Pis building • 250GB disk space used by wheels
  36. Reasons for failure

  37. The results (round 2) • Discount “no release” versions •

    Install key missing dependencies • Total packages: 100, 802 / 117, 444 (86%) • Package versions: 586, 266 / 703, 571 (83%)
  38. Reasons for failure (round 2)

  39. It’s done! • Scaled down to 5 Pis to keep

    up with new releases • Thanks, Pete! You can have them back now.
  40. Supporting multiple Python versions • We built everything on Raspbian

    Jessie (Python 3.4) • Raspbian Stretch is now released (Python 3.5) • We also want to support Python 3.6 (and 2.7) • Pure Python packages are built for a whole major Python version (i.e. any Python 3.x) • Compiled packages are built for a specific minor Python version (e.g. Python 3.4) and need rebuilding for other ABIs
  41. Build procedure • Keep package list and version list up-to-date

    in database • Form a build queue for unattempted package versions • Attempt to build everything in the queue • If a wheel is tagged with “linux_armv7l”, create a symlink from “linux_armv6l” for Raspberry Pi 1/Zero users • If a wheel is tagged with an ABI other than “none”, trigger it to be rebuilt for other ABIs • Stretch builder will pick this up and attempt to build for Python 3.5 • Same principle for Python 3.6 and 2.7
  42. How to get it • Raspbian Jessie (old stable): –

    Manually configure pip to use piwheels (/etc/pip.conf) • Raspbian Stretch (new stable): – pip now pre-configured to use piwheels as an additional index – “sudo apt upgrade” will bring in the config :)
  43. www.piwheels.hostedpi.com

  44. github.com/bennuttall/piwheels

  45. None
  46. piwheels: building a faster Python package repository for Raspberry Pi

    users Ben Nuttall Raspberry Pi Foundation UK Charity 1129409