Upgrade to Pro — share decks privately, control downloads, hide ads and more …

piwheels - pyconuk

piwheels - pyconuk

piwheels talk given at PyCon UK 2017

Ben Nuttall

October 29, 2017
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View Slide

  2. Ben Nuttall

    Raspberry Pi Community Manager

    Columnist for opensource.com

    github.com/bennuttall

    twitter.com/ben_nuttall

    [email protected]

    View Slide

  3. pip install numpy

    View Slide

  4. Python wheels

    Wheels are built distributions, and can be uploaded to PyPI
    alongside source distributions

    This saves users from building themselves

    Wheels are architecture-specific
    – e.g. win32, win64, macosx, linux32, linux64

    A recent addition allowed “manylinux” wheels to be uploaded

    Raspberry Pi is not the “manylinux” architecture, it’s ARM

    View Slide

  5. ARM wheels

    Technically...
    – Pi 3 is ARMv8
    – Pi 2 is ARMv7
    – Pi 1 / Zero are ARMv6

    But...
    – Wheels built on Pi 2/3 are tagged “armv7l”
    – Wheels built on Pi 1 / Zero are tagged “armv6l”

    And…
    – They are actually ARMv6 wheels, and they’re all the same

    So…
    – A wheel built on a Pi 3 will work on a Pi 2, as is
    – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”)

    View Slide

  6. pip install

    View Slide

  7. Which type of x86/x86_64 are you?

    View Slide

  8. ...I have them all

    View Slide

  9. Actually I’m linux_armv7l

    View Slide

  10. What? I’ve never heard of it.
    Here’s the source, build it yourself!

    ~20 mins on Pi 3 (1.2GHz quad-core)

    ~2.5 hours on Pi 1 (700MHz single-core)

    View Slide

  11. Fine! I’ll build my own package
    repository...

    cd Projects

    mkdir piwheels

    cd piwheels

    git init

    View Slide

  12. piwheels

    I could build everything on PyPI and host my own repository
    – “pip wheel numpy” works on a Pi
    – You can distribute the wheel and it’s a super fast install

    Can I host my own package repository?
    – Apparently, yes!
    – At minimum, an Apache directory listing will do the trick

    View Slide

  13. piwheels v1

    Pi 3 in my living room

    Build the latest version of every package (106k packages at
    the time)

    Log output into postgres database

    Host a package repository on the same Pi

    View Slide

  14. piwheels v1: the results

    It took 10 days to complete the build run

    76% build success rate

    Repository (was) live at piwheels.bennuttall.com

    “pip install numpy ­i http://piwheels.bennuttall.com” works
    and takes 6 seconds :)

    Proof of concept: it works, it’s probably useful

    View Slide

  15. Planning piwheels v2

    Build every version of every package

    Keep up with new releases automatically

    Host a package repository as before

    Create a test suite

    Provide installation instructions & developer documentation
    so people can contribute

    View Slide

  16. Planning piwheels v2

    Now 113k packages on PyPI

    But now I’m building every version of every package

    750k package versions to build

    At the previous rate, this will take 70 days on one Pi…
    – Maybe I could use … more than one Pi?

    View Slide

  17. Mythic Beasts: Pi in the cloud

    View Slide

  18. There is no Raspberry Pi cloud...

    View Slide

  19. Pete

    View Slide

  20. View Slide

  21. piwheels v2

    One Pi to start

    Left it running for a couple of weeks:
    – Actually running slower than before due to NFS
    – New estimate: 100 days

    View Slide

  22. Lightning talk at EuroPython

    View Slide

  23. Why don’t you just cross-compile?

    It’s not all about speed

    Reliability

    Compatability

    Familiarity

    Ease of use

    I can scale up Pis easily

    Eating my own dog food

    View Slide

  24. View Slide

  25. View Slide

  26. ALL the cloud Pis

    View Slide

  27. Adding more Pis

    Provisioned a second Pi
    – No web server or database
    – Connected to the database on first Pi
    – rsync files to first Pi

    Provisioned a third Pi

    Installed “terminator”

    Provisioned Pis 4 and 5

    This is easy. I’ll be done in no time!

    View Slide

  28. View Slide

  29. It didn’t scale

    Output on 3 Pis was about the same as the output on 1 Pi

    The database was getting hammered

    View Slide

  30. Dave Jones

    Author of picamera

    Co-author of GPIO Zero

    Self-professed SQL know-it-
    all

    We’ve worked together on
    open source projects a lot

    View Slide

  31. Make it scale!

    Pull request:
    – Query optimisations
    – Queuing system with zeromq

    Re-deployed the code

    Original Pi is now “master” running database and web server only

    Other Pis are now “builders” using master’s database and rsync-
    ing files to master

    View Slide

  32. It worked! Keep going!

    View Slide

  33. 20 Raspberry Pis

    ~3k packages per hour

    ~72k per day (10%)

    Now also logging which Pi built each package

    Pis seem to be holding up

    Dropped rsync in favour of sshfs

    It’s going well! I’ll be done in no time!

    View Slide

  34. People do stupid things

    Random files created in my home directory

    Random stuff appended to my .bashrc

    Some people run “git clone” in their setup.py

    Inadvertently importing numpy

    View Slide

  35. The results

    Total packages processed: 113, 649

    Package versions built: 570, 648 / 752, 817 (76%)

    Total cumulative time spent building: 156 days, 18 hours
    (including duplicates)
    – In real time this was 26 days:

    16 days with 1 Pi building

    10 days with up to 19 Pis building

    250GB disk space used by wheels

    View Slide

  36. Reasons for failure

    View Slide

  37. The results (round 2)

    Discount “no release” versions

    Install key missing dependencies

    Total packages: 100, 802 / 117, 444 (86%)

    Package versions: 586, 266 / 703, 571 (83%)

    View Slide

  38. Reasons for failure (round 2)

    View Slide

  39. It’s done!

    Scaled down to 5 Pis to keep up with new releases

    Thanks, Pete! You can have them back now.

    View Slide

  40. Supporting multiple Python versions

    We built everything on Raspbian Jessie (Python 3.4)

    Raspbian Stretch is now released (Python 3.5)

    We also want to support Python 3.6 (and 2.7)

    Pure Python packages are built for a whole major Python
    version (i.e. any Python 3.x)

    Compiled packages are built for a specific minor Python
    version (e.g. Python 3.4) and need rebuilding for other ABIs

    View Slide

  41. Build procedure

    Keep package list and version list up-to-date in database

    Form a build queue for unattempted package versions

    Attempt to build everything in the queue

    If a wheel is tagged with “linux_armv7l”, create a symlink from
    “linux_armv6l” for Raspberry Pi 1/Zero users

    If a wheel is tagged with an ABI other than “none”, trigger it to be
    rebuilt for other ABIs

    Stretch builder will pick this up and attempt to build for Python 3.5

    Same principle for Python 3.6 and 2.7

    View Slide

  42. How to get it

    Raspbian Jessie (old stable):
    – Manually configure pip to use piwheels (/etc/pip.conf)

    Raspbian Stretch (new stable):
    – pip now pre-configured to use piwheels as an additional index
    – “sudo apt upgrade” will bring in the config :)

    View Slide

  43. www.piwheels.hostedpi.com

    View Slide

  44. github.com/bennuttall/piwheels

    View Slide

  45. View Slide

  46. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View Slide