piwheels

Talk given at Raspberry Pint London 30 May 2017

Transcript

1. My project
2. PyPI – the Python package index • Install Python libraries and applications with “pip install” • Python packages can be implemented in Python or in C – gpiozero is implemented in Python – numpy is implemented in C • Packages implemented in Python do not require building: – Download source – Install • Packages implemented in C require building: – Download source – Build – Install
3. The problem • Packages implemented in C can take a long time to build • “pip install numpy” takes: – 30 mins on Pi 3 – 2.5 hours on Pi 1
4. The general solution: Python wheels • A built distribution format called “wheels” exists • Wheels can be uploaded to PyPI alongside source distributions • Wheels are architecture-specific – e.g. win32, win_amd64, macosx, linux_i686, linux_x86_64 • A recent addition allowed “manylinux” wheels to be uploaded • Most people don't bother uploading wheels
5. Example: cwiid Source tarball No wheels
6. Example: django Source tarball 1 any platform wheel
7. Example: pygame Source tarball 18 platform wheels
8. Example: numpy Source zip 23 platform wheels
9. Useless! • None of these wheels are of any use to a Pi user • No architecture match • Fallback: build from source!
10. ARM Wheels • Wheels built on a Raspberry Pi are specific to the ARM architecture, tagged with the ARM version: – numpy-1.12.1-cp34-cp34m-linux_armv6l.whl – numpy-1.12.1-cp34-cp34m-linux_armv7l.whl • PyPI does not currently support uploading ARM wheels – The next generation of PyPI does have support for ARM wheels (due to my project)
11. Why use pip not apt? • pip provides newer versions of packages • virtualenv users install dependencies using pip – This includes pre-installed system packages – “pip install sense-hat” inside a virtalenv includes installing numpy and other dependencies
12. The solution: piwheels • There were 106, 589 packages on PyPI (at the time) • I attempted to build them all on a Pi 3 – It took 10 days – 76% built successfully – Build output logged in a database • The same Pi 3 hosts a package repository – “pip install numpy -i http://piwheels.bennuttall.com” works and takes 6 seconds :) – Browse http://piwheels.bennuttall.com/package to see build output
13. Benchmarks Package Pi 1 install time (PyPI) Pi 3 install time (PyPI) Pi 1 install time (piwheels) Pi 3 install time (piwheels) numpy sense-hat gpiozero matplotlib PIL scipy pandas django flask
14. What next? • Make reproducable project to welcome open-source contributions • Move to a Mythic Beasts hosted Pi • Automate rebuilding wheels for new releases • Manually inspect failed builds, try to resolve issues • Get a domain name and SSL cert • Add server URL to Raspbian's pip config • Build for multiple Python versions (2.7, 3.4, 3.5, 3.6) • Build for ARMv6, ARMv7, ARMv8