$30 off During Our Annual Pro Sale. View Details »

piwheels lightning talk

piwheels lightning talk

Lightning talk on my project 'piwheels' given at EuroPython 2017

Ben Nuttall

July 11, 2017
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. A title would give away the talk contents

    I’m Ben Nuttall

    I work for the Raspberry Pi Foundation

    I do a lot of Python on Raspberry Pi

    View Slide

  2. pip install numpy

    Browse http://pypi.python.org/simple/numpy

    Look for a wheel file with a platform tag matching the current
    platform
    – If one is found, download and install :)
    – If no match is found, look for source tarball/zip

    If source is found, download and build from source :O

    If no source is found, fail :(

    View Slide

  3. Which type of x86/x86_64 are you?

    View Slide

  4. ...I have them all

    View Slide

  5. Actually I’m linux_armv7l

    View Slide

  6. What? I’ve never heard of it.
    Here’s the source, build it yourself!

    PyPI doesn’t support uploading ARM wheels :(

    “pip install numpy” takes:
    – ~20 mins on Pi 3 (1.2GHz quad-core)
    – ~2.5 hours on Pi 1 (700MHz single-core)

    View Slide

  7. Fine! I’ll build my own package repository...

    cd Projects

    mkdir piwheels

    cd piwheels

    git init

    View Slide

  8. piwheels v1

    Pi 3 in my living room

    Build the latest version of every package (106k packages)

    Log output into postgres database

    Host a package repository on the same Pi

    On GitHub but not reproducable

    View Slide

  9. piwheels v1

    It took 10 days to complete the build run

    76% build success rate

    Repository live at piwheels.bennuttall.com

    “pip install numpy ­i http://piwheels.bennuttall.com” works
    and takes 6 seconds :)

    View Slide

  10. piwheels v2

    Pi 3 in a data centre (Thanks Mythic Beasts)

    Build every version of every package (740k releases)

    Host a package repository on the same Pi

    Repository live at www.piwheels.hostedpi.com

    Test suite, installation instructions & developer documentation on
    GitHub:
    – github.com/bennuttall/piwheels
    – Help! Contribute!

    View Slide

  11. In the future

    Complete the build backlog (in progress)

    Continue to build all new releases

    Try to fix failed builds (missing dependencies, etc)

    Rebuild Python 3.4 wheels for 3.5 & 3.6

    Add the piwheels server to pip config in Raspbian (our distro)
    as an additional index
    – Users get wheels for free without needing to know about it

    View Slide

  12. pypi.org

    Next generation PyPI project

    A Google developer came across piwheels and filed an issue with the
    pypi.org project (warehouse)
    – github.com/pypa/warehouse/issues/2003

    pypi.org now supports uploading ARM wheels :)
    – Thanks @kpayson64 and @dstufft

    Package maintainers can upload wheels built by piwheels to pypi.org
    and they appear on pypi.python.org
    – \o/

    View Slide