PyPI ● Python package repository hosted at pypi.python.org ● Install packages with “pip install” ● Packages can be implemented in Python or C ● Packages implemented in C require building – This can take a long time
Python wheels ● Wheels can be uploaded to PyPI alongside source distributions to save users from building themselves ● Wheels are architecture-specific – e.g. win32, win64, macosx, linux32, linux64 ● A recent addition allowed “manylinux” wheels to be uploaded ● Most maintainers don't bother uploading wheels, but popular build-intensive packages tend to
Python wheels ● When you build a wheel, the filename is made up of a number of parts: – The package name – The package version – The Python tag (e.g. cp34) – The ABI tag (e.g. cp34m) – The platform tag (e.g. linux_armv7l) ● e.g: numpy-1.13.1-cp34-cp34m-linux_armv7l.whl
What? I’ve never heard of it. Here’s the source, build it yourself! ● PyPI doesn’t support uploading ARM wheels :( ● “pip install numpy” takes: – ~20 mins on Pi 3 (1.2GHz quad-core) – ~2.5 hours on Pi 1 (700MHz single-core)
piwheels ● I could build everything on PyPI and host my own repository – “pip wheel numpy” works on a Pi – you can distribute the wheel ● I want to log builds and store output in a database – I’d better write it in Python, not bash – Stack overflow says “import pip” is a thing ● I need a list of all packages in PyPI – Stack overflow says PyPI provides an xmlrpclib interface (whatever that is) and gives a Python example ● Can I host a package repository? – At minimum, an Apache directory listing will do the trick
piwheels v1 ● Pi 3 in my living room ● Build the latest version of every package (106k packages) ● Log output into postgres database ● Host a package repository on the same Pi ● On GitHub but not really reproducable
piwheels v1: the results ● It took 10 days to complete the build run ● 76% build success rate ● Repository live at piwheels.bennuttall.com ● “pip install numpy i http://piwheels.bennuttall.com” works and takes 6 seconds :) ● Still running on a Pi 3 in my living room ● Proof of concept: it works, it’s probably useful
Warehouse ● A Google developer working on a Raspberry Pi related Python package came across piwheels ● Asked if I’m going to try to get maintainers to upload my wheels to PyPI ● I said “they can’t” ● He filed an issue with warehouse, suggesting they allow ARM platform wheels ● They said “ok”
Planning piwheels v2 ● Build every version of every package ● Keep up with new releases automatically ● Host a package repository as before ● Test suite ● Provide installation instructions & developer documentation so people can contribute
750,000 releases ● Now 113k packages ● 750k package versions to build ● At the previous rate, this will take 70 days ● Left it running for a couple of weeks: – Actually running slower than before due to network filesystem – New estimate: 100 days
Why don’t you just cross-compile? ● It’s not all about speed ● Reliability ● Compatability ● Familiarity ● Ease of use ● I can scale up Pis easily ● Eating my own dog food
Adding more Pis ● Provisioned a second Pi – No web server or database – Connected to the database on first Pi – rsync files to first Pi ● Provisioned a third Pi ● Internalled “terminator” ● Provisioned Pis 4 and 5 ● This is easy. I’ll be done in no time!
Make it scale! ● Pull request: – Query optimisations – Queuing system with zeromq ● Re-deployed the code ● Original Pi is now “master” running database and web server only ● Other Pis are now “builders” using master’s database and rsync- ing files to master
20 Raspberry Pis ● ~6k packages per hour ● ~120k per day (15%) ● Now also logging which Pi built each package ● Pis seem to be holding up ● Dropped rsync in favour of sshfs ● It’s going well! I’ll be done in no time!
The results ● Total packages: 113, 649 ● Package versions: 752, 817 ● Build success rate: 76% ● Total cumulative time spent building: 156 days, 18 hours (including duplicates) – In real time this was 26 days: ● 16 days with 1 Pi building ● 10 days with up to 19 Pis building ● Total disk usage from wheels: 250GB
pypi.org ● Next generation PyPI project ● A Google developer came across piwheels and filed an issue with the pypi.org project (warehouse) – github.com/pypa/warehouse/issues/2003 ● pypi.org now supports uploading ARM wheels :) – Thanks @kpayson64 and @dstufft ● Package maintainers can upload wheels built by piwheels to pypi.org and they appear on pypi.python.org – \o/
People do stupid things ● Random files created in my home directory ● Random stuff appended to my .bashrc ● Some people run “git clone” in their setup.py ● Inadvertently importing numpy
In the future ● Continue to build all new releases on a small number of buider Pis ● Add SSL to web domain ● Create individual package pages with build output ● Install key dependencies and try to fix failed builds ● Rebuild Python 3.4 wheels for 3.5 & 3.6 ● Ensure ARMv6 wheels are available too – You can rename ARMv7 wheels and they work on ARMv6... ● Add the piwheels server to pip config in Raspbian (our distro) as an additional index – Users get wheels for free without needing to know about it
bennuttall
dicoding
cocoeyes02
codehex
maroon1st
youliangdao
fabpot
kin29
fendo181
penpenpng
alli83
hirobe1999
keithpitt
chriscoyier
afnizarnur
shpigford
zakiwarfel
revolveconf
sachag
panda_program
bcantrill
bkeepers
rocio
michaelherold