Upgrade to Pro — share decks privately, control downloads, hide ads and more …

piwheels - pyconuk

piwheels - pyconuk

piwheels talk given at PyCon UK 2017

Ben Nuttall

October 29, 2017
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View full-size slide

  2. Ben Nuttall

    Raspberry Pi Community Manager

    Columnist for opensource.com

    github.com/bennuttall

    twitter.com/ben_nuttall

    [email protected]

    View full-size slide

  3. pip install numpy

    View full-size slide

  4. Python wheels

    Wheels are built distributions, and can be uploaded to PyPI
    alongside source distributions

    This saves users from building themselves

    Wheels are architecture-specific
    – e.g. win32, win64, macosx, linux32, linux64

    A recent addition allowed “manylinux” wheels to be uploaded

    Raspberry Pi is not the “manylinux” architecture, it’s ARM

    View full-size slide

  5. ARM wheels

    Technically...
    – Pi 3 is ARMv8
    – Pi 2 is ARMv7
    – Pi 1 / Zero are ARMv6

    But...
    – Wheels built on Pi 2/3 are tagged “armv7l”
    – Wheels built on Pi 1 / Zero are tagged “armv6l”

    And…
    – They are actually ARMv6 wheels, and they’re all the same

    So…
    – A wheel built on a Pi 3 will work on a Pi 2, as is
    – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”)

    View full-size slide

  6. Which type of x86/x86_64 are you?

    View full-size slide

  7. ...I have them all

    View full-size slide

  8. Actually I’m linux_armv7l

    View full-size slide

  9. What? I’ve never heard of it.
    Here’s the source, build it yourself!

    ~20 mins on Pi 3 (1.2GHz quad-core)

    ~2.5 hours on Pi 1 (700MHz single-core)

    View full-size slide

  10. Fine! I’ll build my own package
    repository...

    cd Projects

    mkdir piwheels

    cd piwheels

    git init

    View full-size slide

  11. piwheels

    I could build everything on PyPI and host my own repository
    – “pip wheel numpy” works on a Pi
    – You can distribute the wheel and it’s a super fast install

    Can I host my own package repository?
    – Apparently, yes!
    – At minimum, an Apache directory listing will do the trick

    View full-size slide

  12. piwheels v1

    Pi 3 in my living room

    Build the latest version of every package (106k packages at
    the time)

    Log output into postgres database

    Host a package repository on the same Pi

    View full-size slide

  13. piwheels v1: the results

    It took 10 days to complete the build run

    76% build success rate

    Repository (was) live at piwheels.bennuttall.com

    “pip install numpy ­i http://piwheels.bennuttall.com” works
    and takes 6 seconds :)

    Proof of concept: it works, it’s probably useful

    View full-size slide

  14. Planning piwheels v2

    Build every version of every package

    Keep up with new releases automatically

    Host a package repository as before

    Create a test suite

    Provide installation instructions & developer documentation
    so people can contribute

    View full-size slide

  15. Planning piwheels v2

    Now 113k packages on PyPI

    But now I’m building every version of every package

    750k package versions to build

    At the previous rate, this will take 70 days on one Pi…
    – Maybe I could use … more than one Pi?

    View full-size slide

  16. Mythic Beasts: Pi in the cloud

    View full-size slide

  17. There is no Raspberry Pi cloud...

    View full-size slide

  18. piwheels v2

    One Pi to start

    Left it running for a couple of weeks:
    – Actually running slower than before due to NFS
    – New estimate: 100 days

    View full-size slide

  19. Lightning talk at EuroPython

    View full-size slide

  20. Why don’t you just cross-compile?

    It’s not all about speed

    Reliability

    Compatability

    Familiarity

    Ease of use

    I can scale up Pis easily

    Eating my own dog food

    View full-size slide

  21. ALL the cloud Pis

    View full-size slide

  22. Adding more Pis

    Provisioned a second Pi
    – No web server or database
    – Connected to the database on first Pi
    – rsync files to first Pi

    Provisioned a third Pi

    Installed “terminator”

    Provisioned Pis 4 and 5

    This is easy. I’ll be done in no time!

    View full-size slide

  23. It didn’t scale

    Output on 3 Pis was about the same as the output on 1 Pi

    The database was getting hammered

    View full-size slide

  24. Dave Jones

    Author of picamera

    Co-author of GPIO Zero

    Self-professed SQL know-it-
    all

    We’ve worked together on
    open source projects a lot

    View full-size slide

  25. Make it scale!

    Pull request:
    – Query optimisations
    – Queuing system with zeromq

    Re-deployed the code

    Original Pi is now “master” running database and web server only

    Other Pis are now “builders” using master’s database and rsync-
    ing files to master

    View full-size slide

  26. It worked! Keep going!

    View full-size slide

  27. 20 Raspberry Pis

    ~3k packages per hour

    ~72k per day (10%)

    Now also logging which Pi built each package

    Pis seem to be holding up

    Dropped rsync in favour of sshfs

    It’s going well! I’ll be done in no time!

    View full-size slide

  28. People do stupid things

    Random files created in my home directory

    Random stuff appended to my .bashrc

    Some people run “git clone” in their setup.py

    Inadvertently importing numpy

    View full-size slide

  29. The results

    Total packages processed: 113, 649

    Package versions built: 570, 648 / 752, 817 (76%)

    Total cumulative time spent building: 156 days, 18 hours
    (including duplicates)
    – In real time this was 26 days:

    16 days with 1 Pi building

    10 days with up to 19 Pis building

    250GB disk space used by wheels

    View full-size slide

  30. Reasons for failure

    View full-size slide

  31. The results (round 2)

    Discount “no release” versions

    Install key missing dependencies

    Total packages: 100, 802 / 117, 444 (86%)

    Package versions: 586, 266 / 703, 571 (83%)

    View full-size slide

  32. Reasons for failure (round 2)

    View full-size slide

  33. It’s done!

    Scaled down to 5 Pis to keep up with new releases

    Thanks, Pete! You can have them back now.

    View full-size slide

  34. Supporting multiple Python versions

    We built everything on Raspbian Jessie (Python 3.4)

    Raspbian Stretch is now released (Python 3.5)

    We also want to support Python 3.6 (and 2.7)

    Pure Python packages are built for a whole major Python
    version (i.e. any Python 3.x)

    Compiled packages are built for a specific minor Python
    version (e.g. Python 3.4) and need rebuilding for other ABIs

    View full-size slide

  35. Build procedure

    Keep package list and version list up-to-date in database

    Form a build queue for unattempted package versions

    Attempt to build everything in the queue

    If a wheel is tagged with “linux_armv7l”, create a symlink from
    “linux_armv6l” for Raspberry Pi 1/Zero users

    If a wheel is tagged with an ABI other than “none”, trigger it to be
    rebuilt for other ABIs

    Stretch builder will pick this up and attempt to build for Python 3.5

    Same principle for Python 3.6 and 2.7

    View full-size slide

  36. How to get it

    Raspbian Jessie (old stable):
    – Manually configure pip to use piwheels (/etc/pip.conf)

    Raspbian Stretch (new stable):
    – pip now pre-configured to use piwheels as an additional index
    – “sudo apt upgrade” will bring in the config :)

    View full-size slide

  37. www.piwheels.hostedpi.com

    View full-size slide

  38. github.com/bennuttall/piwheels

    View full-size slide

  39. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View full-size slide