piwheels: building a faster Python package repository for Raspberry Pi users Ben Nuttall Raspberry Pi Foundation UK Charity 1129409 

Ben Nuttall ● Raspberry Pi Community Manager ● Columnist for opensource.com ● github.com/bennuttall ● twitter.com/ben_nuttall ● [email protected] 

pip install numpy 

Python wheels ● Wheels are built distributions, and can be uploaded to PyPI alongside source distributions ● This saves users from building themselves ● Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 ● A recent addition allowed “manylinux” wheels to be uploaded ● Raspberry Pi is not the “manylinux” architecture, it’s ARM 

ARM wheels ● Technically... – Pi 3 is ARMv8 – Pi 2 is ARMv7 – Pi 1 / Zero are ARMv6 ● But... – Wheels built on Pi 2/3 are tagged “armv7l” – Wheels built on Pi 1 / Zero are tagged “armv6l” ● And… – They are actually ARMv6 wheels, and they’re all the same ● So… – A wheel built on a Pi 3 will work on a Pi 2, as is – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”) 

pip install 

Which type of x86/x86_64 are you? 

...I have them all 

Actually I’m linux_armv7l 

What? I’ve never heard of it. Here’s the source, build it yourself! ● ~20 mins on Pi 3 (1.2GHz quad-core) ● ~2.5 hours on Pi 1 (700MHz single-core) 

Fine! I’ll build my own package repository... ● cd Projects ● mkdir piwheels ● cd piwheels ● git init 

piwheels ● I could build everything on PyPI and host my own repository – “pip wheel numpy” works on a Pi – You can distribute the wheel and it’s a super fast install ● Can I host my own package repository? – Apparently, yes! – At minimum, an Apache directory listing will do the trick 

piwheels v1 ● Pi 3 in my living room ● Build the latest version of every package (106k packages at the time) ● Log output into postgres database ● Host a package repository on the same Pi 

piwheels v1: the results ● It took 10 days to complete the build run ● 76% build success rate ● Repository (was) live at piwheels.bennuttall.com ● “pip install numpy ­i http://piwheels.bennuttall.com” works and takes 6 seconds :) ● Proof of concept: it works, it’s probably useful 

Planning piwheels v2 ● Build every version of every package ● Keep up with new releases automatically ● Host a package repository as before ● Create a test suite ● Provide installation instructions & developer documentation so people can contribute 

Planning piwheels v2 ● Now 113k packages on PyPI ● But now I’m building every version of every package ● 750k package versions to build ● At the previous rate, this will take 70 days on one Pi… – Maybe I could use … more than one Pi? 

Mythic Beasts: Pi in the cloud 

There is no Raspberry Pi cloud... 

Pete 

No content

piwheels v2 ● One Pi to start ● Left it running for a couple of weeks: – Actually running slower than before due to NFS – New estimate: 100 days 

Lightning talk at EuroPython 

Why don’t you just cross-compile? ● It’s not all about speed ● Reliability ● Compatability ● Familiarity ● Ease of use ● I can scale up Pis easily ● Eating my own dog food 

No content

No content

ALL the cloud Pis 

Adding more Pis ● Provisioned a second Pi – No web server or database – Connected to the database on first Pi – rsync files to first Pi ● Provisioned a third Pi ● Installed “terminator” ● Provisioned Pis 4 and 5 ● This is easy. I’ll be done in no time! 

No content

It didn’t scale ● Output on 3 Pis was about the same as the output on 1 Pi ● The database was getting hammered 

Dave Jones ● Author of picamera ● Co-author of GPIO Zero ● Self-professed SQL know-it- all ● We’ve worked together on open source projects a lot 

Make it scale! ● Pull request: – Query optimisations – Queuing system with zeromq ● Re-deployed the code ● Original Pi is now “master” running database and web server only ● Other Pis are now “builders” using master’s database and rsync- ing files to master 

It worked! Keep going! 

20 Raspberry Pis ● ~3k packages per hour ● ~72k per day (10%) ● Now also logging which Pi built each package ● Pis seem to be holding up ● Dropped rsync in favour of sshfs ● It’s going well! I’ll be done in no time! 

People do stupid things ● Random files created in my home directory ● Random stuff appended to my .bashrc ● Some people run “git clone” in their setup.py ● Inadvertently importing numpy 

The results ● Total packages processed: 113, 649 ● Package versions built: 570, 648 / 752, 817 (76%) ● Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: ● 16 days with 1 Pi building ● 10 days with up to 19 Pis building ● 250GB disk space used by wheels 

Reasons for failure 

The results (round 2) ● Discount “no release” versions ● Install key missing dependencies ● Total packages: 100, 802 / 117, 444 (86%) ● Package versions: 586, 266 / 703, 571 (83%) 

Reasons for failure (round 2) 

It’s done! ● Scaled down to 5 Pis to keep up with new releases ● Thanks, Pete! You can have them back now. 

Supporting multiple Python versions ● We built everything on Raspbian Jessie (Python 3.4) ● Raspbian Stretch is now released (Python 3.5) ● We also want to support Python 3.6 (and 2.7) ● Pure Python packages are built for a whole major Python version (i.e. any Python 3.x) ● Compiled packages are built for a specific minor Python version (e.g. Python 3.4) and need rebuilding for other ABIs 

Build procedure ● Keep package list and version list up-to-date in database ● Form a build queue for unattempted package versions ● Attempt to build everything in the queue ● If a wheel is tagged with “linux_armv7l”, create a symlink from “linux_armv6l” for Raspberry Pi 1/Zero users ● If a wheel is tagged with an ABI other than “none”, trigger it to be rebuilt for other ABIs ● Stretch builder will pick this up and attempt to build for Python 3.5 ● Same principle for Python 3.6 and 2.7 

How to get it ● Raspbian Jessie (old stable): – Manually configure pip to use piwheels (/etc/pip.conf) ● Raspbian Stretch (new stable): – pip now pre-configured to use piwheels as an additional index – “sudo apt upgrade” will bring in the config :) 

www.piwheels.hostedpi.com 

github.com/bennuttall/piwheels 

No content

piwheels: building a faster Python package repository for Raspberry Pi users Ben Nuttall Raspberry Pi Foundation UK Charity 1129409