Upgrade to Pro — share decks privately, control downloads, hide ads and more …

piwheels

 piwheels

Talk given at Raspberry Pint London 30 May 2017

Avatar for Ben Nuttall

Ben Nuttall

May 30, 2017
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. PyPI – the Python package index • Install Python libraries

    and applications with “pip install” • Python packages can be implemented in Python or in C – gpiozero is implemented in Python – numpy is implemented in C • Packages implemented in Python do not require building: – Download source – Install • Packages implemented in C require building: – Download source – Build – Install
  2. The problem • Packages implemented in C can take a

    long time to build • “pip install numpy” takes: – 30 mins on Pi 3 – 2.5 hours on Pi 1
  3. The general solution: Python wheels • A built distribution format

    called “wheels” exists • Wheels can be uploaded to PyPI alongside source distributions • Wheels are architecture-specific – e.g. win32, win_amd64, macosx, linux_i686, linux_x86_64 • A recent addition allowed “manylinux” wheels to be uploaded • Most people don't bother uploading wheels
  4. Useless! • None of these wheels are of any use

    to a Pi user • No architecture match • Fallback: build from source!
  5. ARM Wheels • Wheels built on a Raspberry Pi are

    specific to the ARM architecture, tagged with the ARM version: – numpy-1.12.1-cp34-cp34m-linux_armv6l.whl – numpy-1.12.1-cp34-cp34m-linux_armv7l.whl • PyPI does not currently support uploading ARM wheels – The next generation of PyPI does have support for ARM wheels (due to my project)
  6. Why use pip not apt? • pip provides newer versions

    of packages • virtualenv users install dependencies using pip – This includes pre-installed system packages – “pip install sense-hat” inside a virtalenv includes installing numpy and other dependencies
  7. The solution: piwheels • There were 106, 589 packages on

    PyPI (at the time) • I attempted to build them all on a Pi 3 – It took 10 days – 76% built successfully – Build output logged in a database • The same Pi 3 hosts a package repository – “pip install numpy -i http://piwheels.bennuttall.com” works and takes 6 seconds :) – Browse http://piwheels.bennuttall.com/package to see build output
  8. Benchmarks Package Pi 1 install time (PyPI) Pi 3 install

    time (PyPI) Pi 1 install time (piwheels) Pi 3 install time (piwheels) numpy sense-hat gpiozero matplotlib PIL scipy pandas django flask
  9. What next? • Make reproducable project to welcome open-source contributions

    • Move to a Mythic Beasts hosted Pi • Automate rebuilding wheels for new releases • Manually inspect failed builds, try to resolve issues • Get a domain name and SSL cert • Add server URL to Raspbian's pip config • Build for multiple Python versions (2.7, 3.4, 3.5, 3.6) • Build for ARMv6, ARMv7, ARMv8