piwheels lightning talk

Lightning talk on my project 'piwheels' given at EuroPython 2017

Transcript

1. A title would give away the talk contents • I’m Ben Nuttall • I work for the Raspberry Pi Foundation • I do a lot of Python on Raspberry Pi
2. pip install numpy • Browse http://pypi.python.org/simple/numpy • Look for a wheel file with a platform tag matching the current platform – If one is found, download and install :) – If no match is found, look for source tarball/zip • If source is found, download and build from source :O • If no source is found, fail :(
3. Which type of x86/x86_64 are you?
4. ...I have them all
5. Actually I’m linux_armv7l
6. What? I’ve never heard of it. Here’s the source, build it yourself! • PyPI doesn’t support uploading ARM wheels :( • “pip install numpy” takes: – ~20 mins on Pi 3 (1.2GHz quad-core) – ~2.5 hours on Pi 1 (700MHz single-core)
7. Fine! I’ll build my own package repository... • cd Projects • mkdir piwheels • cd piwheels • git init
8. piwheels v1 • Pi 3 in my living room • Build the latest version of every package (106k packages) • Log output into postgres database • Host a package repository on the same Pi • On GitHub but not reproducable
9. piwheels v1 • It took 10 days to complete the build run • 76% build success rate • Repository live at piwheels.bennuttall.com • “pip install numpy ­i http://piwheels.bennuttall.com” works and takes 6 seconds :)
10. piwheels v2 • Pi 3 in a data centre (Thanks Mythic Beasts) • Build every version of every package (740k releases) • Host a package repository on the same Pi • Repository live at www.piwheels.hostedpi.com • Test suite, installation instructions & developer documentation on GitHub: – github.com/bennuttall/piwheels – Help! Contribute!
11. In the future • Complete the build backlog (in progress) • Continue to build all new releases • Try to fix failed builds (missing dependencies, etc) • Rebuild Python 3.4 wheels for 3.5 & 3.6 • Add the piwheels server to pip config in Raspbian (our distro) as an additional index – Users get wheels for free without needing to know about it
12. pypi.org • Next generation PyPI project • A Google developer came across piwheels and filed an issue with the pypi.org project (warehouse) – github.com/pypa/warehouse/issues/2003 • pypi.org now supports uploading ARM wheels :) – Thanks @kpayson64 and @dstufft • Package maintainers can upload wheels built by piwheels to pypi.org and they appear on pypi.python.org – \o/