PyPI ● Python package repository hosted at pypi.python.org ● Install packages with “pip install” ● Packages can be implemented in Python or C ● Packages implemented in C require building – This can take a long time
Python wheels ● Wheels can be uploaded to PyPI alongside source distributions to save users from building themselves ● Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 ● A recent addition allowed “manylinux” wheels to be uploaded ● Most maintainers don't bother uploading wheels, but popular build-intensive packages tend to
Python wheels ● When you build a wheel, the filename is made up of a number of parts: – The package name – The package version – The Python tag (e.g. cp34) – The ABI tag (e.g. cp34m) – The platform tag (e.g. linux_armv7l) ● e.g: numpy-1.13.1-cp34-cp34m-linux_armv7l.whl
What? I’ve never heard of it. Here’s the source, build it yourself! ● PyPI doesn’t support uploading ARM wheels :( ● “pip install numpy” takes: – ~20 mins on Pi 3 (1.2GHz quad-core) – ~2.5 hours on Pi 1 (700MHz single-core)
piwheels ● I could build everything on PyPI and host my own repository – “pip wheel numpy” works on a Pi – you can distribute the wheel ● I want to log builds and store output in a database – I’d better write it in Python, not bash – Stack overflow says “import pip” is a thing ● I need a list of all packages in PyPI – Stack overflow says PyPI provides an xmlrpclib interface (whatever that is) and gives a Python example ● Can I host a package repository? – At minimum, an Apache directory listing will do the trick
piwheels v1 ● Pi 3 in my living room ● Build the latest version of every package (106k packages) ● Log output into postgres database ● Host a package repository on the same Pi ● On GitHub but not really reproducable
piwheels v1: the results ● It took 10 days to complete the build run ● 76% build success rate ● Repository live at piwheels.bennuttall.com ● “pip install numpy i http://piwheels.bennuttall.com” works and takes 6 seconds :) ● Still running on a Pi 3 in my living room ● Proof of concept: it works, it’s probably useful
Warehouse ● A Google developer working on a Raspberry Pi related Python package came across piwheels ● Asked if I’m going to try to get maintainers to upload my wheels to PyPI ● I said “they can’t” ● He filed an issue with warehouse, suggesting they allow ARM platform wheels ● They said “ok”
Planning piwheels v2 ● Build every version of every package ● Keep up with new releases automatically ● Host a package repository as before ● Test suite ● Provide installation instructions & developer documentation so people can contribute
750,000 releases ● Now 113k packages ● 750k package versions to build ● At the previous rate, this will take 70 days ● Left it running for a couple of weeks: – Actually running slower than before due to network filesystem – New estimate: 100 days
Why don’t you just cross-compile? ● It’s not all about speed ● Reliability ● Compatability ● Familiarity ● Ease of use ● I can scale up Pis easily ● Eating my own dog food
Adding more Pis ● Provisioned a second Pi – No web server or database – Connected to the database on first Pi – rsync files to first Pi ● Provisioned a third Pi ● Internalled “terminator” ● Provisioned Pis 4 and 5 ● This is easy. I’ll be done in no time!
Make it scale! ● Pull request: – Query optimisations – Queuing system with zeromq ● Re-deployed the code ● Original Pi is now “master” running database and web server only ● Other Pis are now “builders” using master’s database and rsync- ing files to master
20 Raspberry Pis ● ~6k packages per hour ● ~120k per day (15%) ● Now also logging which Pi built each package ● Pis seem to be holding up ● Dropped rsync in favour of sshfs ● It’s going well! I’ll be done in no time!
The results ● Total packages: 113, 649 ● Package versions: 752, 817 ● Build success rate: 76% ● Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: ● 16 days with 1 Pi building ● 10 days with up to 19 Pis building ● Total disk usage from wheels: 250GB
pypi.org ● Next generation PyPI project ● A Google developer came across piwheels and filed an issue with the pypi.org project (warehouse) – github.com/pypa/warehouse/issues/2003 ● pypi.org now supports uploading ARM wheels :) – Thanks @kpayson64 and @dstufft ● Package maintainers can upload wheels built by piwheels to pypi.org and they appear on pypi.python.org – \o/
People do stupid things ● Random files created in my home directory ● Random stuff appended to my .bashrc ● Some people run “git clone” in their setup.py ● Inadvertently importing numpy
In the future ● Continue to build all new releases on a small number of buider Pis ● Add SSL to web domain ● Create individual package pages with build output ● Install key dependencies and try to fix failed builds ● Rebuild Python 3.4 wheels for 3.5 & 3.6 ● Ensure ARMv6 wheels are available too – You can rename ARMv7 wheels and they work on ARMv6... ● Add the piwheels server to pip config in Raspbian (our distro) as an additional index – Users get wheels for free without needing to know about it