piwheels lightning talk

Lightning talk on my project 'piwheels' given at EuroPython 2017

Transcript

1. A title would give away the talk contents
   ●
   I’m Ben Nuttall
   ●
   I work for the Raspberry Pi Foundation
   ●
   I do a lot of Python on Raspberry Pi
   

   View Slide

2. pip install numpy
   ●
   Browse http://pypi.python.org/simple/numpy
   ●
   Look for a wheel file with a platform tag matching the current
   platform
   – If one is found, download and install :)
   – If no match is found, look for source tarball/zip
   ●
   If source is found, download and build from source :O
   ●
   If no source is found, fail :(
   

   View Slide

3. Which type of x86/x86_64 are you?
   

   View Slide

4. ...I have them all
   

   View Slide

5. Actually I’m linux_armv7l
   

   View Slide

6. What? I’ve never heard of it.
   Here’s the source, build it yourself!
   ●
   PyPI doesn’t support uploading ARM wheels :(
   ●
   “pip install numpy” takes:
   – ~20 mins on Pi 3 (1.2GHz quad-core)
   – ~2.5 hours on Pi 1 (700MHz single-core)
   

   View Slide

7. Fine! I’ll build my own package repository...
   ●
   cd Projects
   ●
   mkdir piwheels
   ●
   cd piwheels
   ●
   git init
   

   View Slide

8. piwheels v1
   ●
   Pi 3 in my living room
   ●
   Build the latest version of every package (106k packages)
   ●
   Log output into postgres database
   ●
   Host a package repository on the same Pi
   ●
   On GitHub but not reproducable
   

   View Slide

9. piwheels v1
   ●
   It took 10 days to complete the build run
   ●
   76% build success rate
   ●
   Repository live at piwheels.bennuttall.com
   ●
   “pip install numpy ­i http://piwheels.bennuttall.com” works
   and takes 6 seconds :)
   

   View Slide

10. piwheels v2
    ●
    Pi 3 in a data centre (Thanks Mythic Beasts)
    ●
    Build every version of every package (740k releases)
    ●
    Host a package repository on the same Pi
    ●
    Repository live at www.piwheels.hostedpi.com
    ●
    Test suite, installation instructions & developer documentation on
    GitHub:
    – github.com/bennuttall/piwheels
    – Help! Contribute!
    

    View Slide

11. In the future
    ●
    Complete the build backlog (in progress)
    ●
    Continue to build all new releases
    ●
    Try to fix failed builds (missing dependencies, etc)
    ●
    Rebuild Python 3.4 wheels for 3.5 & 3.6
    ●
    Add the piwheels server to pip config in Raspbian (our distro)
    as an additional index
    – Users get wheels for free without needing to know about it
    

    View Slide

12. pypi.org
    ●
    Next generation PyPI project
    ●
    A Google developer came across piwheels and filed an issue with the
    pypi.org project (warehouse)
    – github.com/pypa/warehouse/issues/2003
    ●
    pypi.org now supports uploading ARM wheels :)
    – Thanks @kpayson64 and @dstufft
    ●
    Package maintainers can upload wheels built by piwheels to pypi.org
    and they appear on pypi.python.org
    – \o/
    

    View Slide