$30 off During Our Annual Pro Sale. View Details »

piwheels - camjam

piwheels - camjam

piwheels talk given at camjam October 2017

Ben Nuttall

October 14, 2017
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View Slide

  2. Ben Nuttall

    Raspberry Pi Community Manager

    Columnist on opensource.com

    github.com/bennuttall

    twitter.com/ben_nuttall

    [email protected]

    View Slide

  3. Space

    View Slide

  4. Astro Pi

    View Slide

  5. Dave

    Dave: I need your help. Can we install
    these libraries in space?

    Me: I think so, if we build wheels of
    them.

    View Slide

  6. pip wheel numpy

    View Slide

  7. PyPI

    Python package repository hosted at pypi.python.org

    Install packages with “pip install”

    Packages can be implemented in Python or C

    Packages implemented in C require building
    – This can take a long time

    View Slide

  8. Python wheels

    Wheels are built distributions, and can be uploaded to PyPI
    alongside source distributions

    This saves users from building themselves

    Wheels are architecture-specific
    – e.g. win32, win64, macosx, linux32, linux64

    A recent addition allowed “manylinux” wheels to be uploaded

    Raspberry Pi is not the “manylinux” architecture, it’s ARM

    View Slide

  9. ARM wheels

    Technically...
    – Pi 3 is ARMv8
    – Pi 2 is ARMv7
    – Pi 1 / Zero are ARMv6

    But...
    – Wheels built on Pi 2/3 are tagged “armv7l”
    – Wheels built on Pi 1 / Zero are tagged “armv6l”

    And…
    – They are actually ARMv6 wheels, and they’re all the same

    So…
    – A wheel built on a Pi 3 will work on a Pi 2, as is
    – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”)

    View Slide

  10. pip install

    View Slide

  11. Which type of x86/x86_64 are you?

    View Slide

  12. ...I have them all

    View Slide

  13. Actually I’m linux_armv7l

    View Slide

  14. What? I’ve never heard of it.
    Here’s the source, build it yourself!

    PyPI doesn’t support uploading ARM wheels :(

    “pip install numpy” takes:
    – ~20 mins on Pi 3 (1.2GHz quad-core)
    – ~2.5 hours on Pi 1 (700MHz single-core)

    View Slide

  15. Fine! I’ll build my own package
    repository...

    cd Projects

    mkdir piwheels

    cd piwheels

    git init

    View Slide

  16. piwheels

    I could build everything on PyPI and host my own repository
    – “pip wheel numpy” works on a Pi
    – You can distribute the wheel and it’s a super fast install

    Can I host my own package repository?
    – Apparently, yes!
    – At minimum, an Apache directory listing will do the trick

    View Slide

  17. piwheels v1

    Pi 3 in my living room

    Build the latest version of every package (106k packages at
    the time)

    Log output into postgres database

    Host a package repository on the same Pi

    View Slide

  18. piwheels v1: the results

    It took 10 days to complete the build run

    76% build success rate

    Repository (was) live at piwheels.bennuttall.com

    “pip install numpy ­i http://piwheels.bennuttall.com” works
    and takes 6 seconds :)

    Proof of concept: it works, it’s probably useful

    View Slide

  19. Planning piwheels v2

    Build every version of every package

    Keep up with new releases automatically

    Host a package repository as before

    Create a test suite

    Provide installation instructions & developer documentation
    so people can contribute

    View Slide

  20. Planning piwheels v2

    Now 113k packages on PyPI

    But now I’m building every version of every package

    750k package versions to build

    At the previous rate, this will take 70 days on one Pi…
    – Maybe I could use … more than one Pi?

    View Slide

  21. Mythic Beasts: Pi in the cloud

    View Slide

  22. There is no Raspberry Pi cloud...

    View Slide

  23. Pete

    View Slide

  24. View Slide

  25. piwheels v2

    One Pi to start

    Left it running for a couple of weeks:
    – Actually running slower than before due to NFS
    – New estimate: 100 days

    View Slide

  26. Lightning talk at EuroPython

    View Slide

  27. Why don’t you just cross-compile?

    It’s not all about speed

    Reliability

    Compatability

    Familiarity

    Ease of use

    I can scale up Pis easily

    Eating my own dog food

    View Slide

  28. View Slide

  29. View Slide

  30. ALL the cloud Pis

    View Slide

  31. Adding more Pis

    Provisioned a second Pi
    – No web server or database
    – Connected to the database on first Pi
    – rsync files to first Pi

    Provisioned a third Pi

    Installed “terminator”

    Provisioned Pis 4 and 5

    This is easy. I’ll be done in no time!

    View Slide

  32. View Slide

  33. It didn’t scale

    Output on 3 Pis was about the same as the output on 1 Pi

    The database was getting hammered

    View Slide

  34. Dave Jones

    Author of picamera

    Co-author of GPIO Zero

    Self-professed SQL know-it-
    all

    We’ve worked together on
    open source projects a lot

    View Slide

  35. Make it scale!

    Pull request:
    – Query optimisations
    – Queuing system with zeromq

    Re-deployed the code

    Original Pi is now “master” running database and web server only

    Other Pis are now “builders” using master’s database and rsync-
    ing files to master

    View Slide

  36. It worked! Keep going!

    View Slide

  37. 20 Raspberry Pis

    ~3k packages per hour

    ~72k per day (10%)

    Now also logging which Pi built each package

    Pis seem to be holding up

    Dropped rsync in favour of sshfs

    It’s going well! I’ll be done in no time!

    View Slide

  38. People do stupid things

    Random files created in my home directory

    Random stuff appended to my .bashrc

    Some people run “git clone” in their setup.py

    Inadvertently importing numpy

    View Slide

  39. The results

    Total packages processed: 113, 649

    Package versions built: 570, 648 / 752, 817 (76%)

    Total cumulative time spent building: 156 days, 18 hours
    (including duplicates)
    – In real time this was 26 days:

    16 days with 1 Pi building

    10 days with up to 19 Pis building

    250GB disk space used by wheels

    View Slide

  40. Reasons for failure

    View Slide

  41. The results (round 2)

    Discount “no release” versions

    Install key missing dependencies

    Total packages: 100, 802 / 117, 444 (86%)

    Package versions: 586, 266 / 703, 571 (83%)

    View Slide

  42. Reasons for failure (round 2)

    View Slide

  43. It’s done!

    Scaled down to 5 Pis to keep up with new releases

    Thanks, Pete! You can have them back now.

    View Slide

  44. Running piwheels as a real service

    Continue to build all new releases on a small number of
    buider Pis

    Added SSL to web domain

    Provide ARMv6 wheels for Pi 1 / Zero

    Build mechanism for triggering builds for multiple Python
    versions (3.4, 3.5, 3.6, 2.7)

    Added the piwheels server to pip config in Raspbian
    – Users get wheels for free without needing to know about it

    View Slide

  45. Does it work?

    Raspbian Stretch:
    – pip configured to use piwheels
    – But… we haven’t rebuilt packages for Stretch yet (Python 3.5)
    – Pure Python packages are available, but big C ones like numpy
    aren’t (yet)

    Raspbian Jessie:
    – Manually configure pip to use piwheels
    – All packages are built for Jessie (Python 3.4)

    View Slide

  46. www.piwheels.hostedpi.com

    View Slide

  47. piwheels:
    building a faster Python package
    repository for Raspberry Pi users
    Ben Nuttall
    Raspberry Pi Foundation
    UK Charity 1129409

    View Slide