PyPI ● Python package repository hosted at pypi.python.org ● Install packages with “pip install” ● Packages can be implemented in Python or C ● Packages implemented in C require building – This can take a long time
Python wheels ● Wheels are built distributions, and can be uploaded to PyPI alongside source distributions ● This saves users from building themselves ● Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 ● A recent addition allowed “manylinux” wheels to be uploaded ● Raspberry Pi is not the “manylinux” architecture, it’s ARM
ARM wheels ● Technically... – Pi 3 is ARMv8 – Pi 2 is ARMv7 – Pi 1 / Zero are ARMv6 ● But... – Wheels built on Pi 2/3 are tagged “armv7l” – Wheels built on Pi 1 / Zero are tagged “armv6l” ● And… – They are actually ARMv6 wheels, and they’re all the same ● So… – A wheel built on a Pi 3 will work on a Pi 2, as is – A wheel built on a Pi 3 will work on a Pi 1 / Zero (if renamed to “armv6l”)
What? I’ve never heard of it. Here’s the source, build it yourself! ● PyPI doesn’t support uploading ARM wheels :( ● “pip install numpy” takes: – ~20 mins on Pi 3 (1.2GHz quad-core) – ~2.5 hours on Pi 1 (700MHz single-core)
piwheels ● I could build everything on PyPI and host my own repository – “pip wheel numpy” works on a Pi – You can distribute the wheel and it’s a super fast install ● Can I host my own package repository? – Apparently, yes! – At minimum, an Apache directory listing will do the trick
piwheels v1 ● Pi 3 in my living room ● Build the latest version of every package (106k packages at the time) ● Log output into postgres database ● Host a package repository on the same Pi
piwheels v1: the results ● It took 10 days to complete the build run ● 76% build success rate ● Repository (was) live at piwheels.bennuttall.com ● “pip install numpy i http://piwheels.bennuttall.com” works and takes 6 seconds :) ● Proof of concept: it works, it’s probably useful
Planning piwheels v2 ● Build every version of every package ● Keep up with new releases automatically ● Host a package repository as before ● Create a test suite ● Provide installation instructions & developer documentation so people can contribute
Planning piwheels v2 ● Now 113k packages on PyPI ● But now I’m building every version of every package ● 750k package versions to build ● At the previous rate, this will take 70 days on one Pi… – Maybe I could use … more than one Pi?
Why don’t you just cross-compile? ● It’s not all about speed ● Reliability ● Compatability ● Familiarity ● Ease of use ● I can scale up Pis easily ● Eating my own dog food
Adding more Pis ● Provisioned a second Pi – No web server or database – Connected to the database on first Pi – rsync files to first Pi ● Provisioned a third Pi ● Installed “terminator” ● Provisioned Pis 4 and 5 ● This is easy. I’ll be done in no time!
Make it scale! ● Pull request: – Query optimisations – Queuing system with zeromq ● Re-deployed the code ● Original Pi is now “master” running database and web server only ● Other Pis are now “builders” using master’s database and rsync- ing files to master
20 Raspberry Pis ● ~3k packages per hour ● ~72k per day (10%) ● Now also logging which Pi built each package ● Pis seem to be holding up ● Dropped rsync in favour of sshfs ● It’s going well! I’ll be done in no time!
People do stupid things ● Random files created in my home directory ● Random stuff appended to my .bashrc ● Some people run “git clone” in their setup.py ● Inadvertently importing numpy
The results ● Total packages processed: 113, 649 ● Package versions built: 570, 648 / 752, 817 (76%) ● Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: ● 16 days with 1 Pi building ● 10 days with up to 19 Pis building ● 250GB disk space used by wheels
Running piwheels as a real service ● Continue to build all new releases on a small number of buider Pis ● Added SSL to web domain ● Provide ARMv6 wheels for Pi 1 / Zero ● Build mechanism for triggering builds for multiple Python versions (3.4, 3.5, 3.6, 2.7) ● Added the piwheels server to pip config in Raspbian – Users get wheels for free without needing to know about it
Does it work? ● Raspbian Stretch: – pip configured to use piwheels – But… we haven’t rebuilt packages for Stretch yet (Python 3.5) – Pure Python packages are available, but big C ones like numpy aren’t (yet) ● Raspbian Jessie: – Manually configure pip to use piwheels – All packages are built for Jessie (Python 3.4)
bennuttall
fadis
niftycorp
tomokusaba
ohakutsu
os1ma
syukai
okuzawats
deepu105
kasacchiful
ivargrimstad
aridlehoover
doggy
pedronauck
colly
lauravandoore
chriscoyier
maggiecrowley
philhawksworth
thekraken
malarkey
bkeepers
philnash
reverentgeek
geeforr