Why I run my own Python package repository

Lightning talk given at PyCon UK about piwheels.org

Transcript

1. Why I run my own Python package repository
   Ben Nuttall
   

   View Slide

2. pip install numpy
   ●
   Browse PyPI
   ●
   Look for wheel with a matching platform tag
   ●
   If one is found, download and install
   ●
   If no match is found, download and build from source
   

   View Slide

3. What kind of x86 are you?
   

   View Slide

4. ...I have them all
   

   View Slide

5. Actually I’m linux_armv7l
   

   View Slide

6. What? I’ve never heard of it.
   Here’s the source. Build it yourself!
   ●
   pip install numpy
   ●
   ~20 mins on Pi 3
   ●
   ~2.5 hours on Pi 1
   

   View Slide

7. Fine! I’ll build my own package repository...
   ●
   cd Projects
   ●
   mkdir piwheels
   ●
   cd piwheels
   ●
   git init
   

   View Slide

8. pip wheel
   ●
   pip wheel numpy
   ●
   numpy-1.15.1-cp35-cp35m-linux_armv7l.whl
   ●
   That, but for all packages
   ●
   175,000 packages
   ●
   That, but for all versions of all packages
   ●
   1,200,000 package versions
   ●
   That, but for Python 3.4 and 3.5...
   

   View Slide

9. I assume you cross-compile?
   Nope
   

   View Slide

10. Mythic Beasts Pi cloud
    

    View Slide

11. The stack
    ●
    Master
    ●
    Distributes jobs, hosts wheel fles and web server
    ●
    Postgres database
    ●
    Monitors PyPI, generates build queue
    ●
    Logs build attemps, logs downloads
    ●
    Builders
    ●
    Get given build jobs
    ●
    3 × Raspbian Jessie (Python 3.4)
    ●
    2 × Raspbian Stretch (Python 3.5)
    ●
    Easy to scale up/down
    

    View Slide

12. The stack
    ●
    Master
    ●
    Distributes jobs, hosts wheel fles and web server
    ●
    Postgres database
    ●
    Monitors PyPI, generates build queue
    ●
    Logs build attemps, logs downloads
    ●
    Builders
    ●
    Get given build jobs
    ●
    3 × Raspbian Jessie (Python 3.4)
    ●
    2 × Raspbian Stretch (Python 3.5)
    ●
    Easy to scale up/down
    

    View Slide

13. Using piwheels
    ●
    Raspbian Stretch is pre-confgured to use piwheels.org as an additional
    index to PyPI
    ●
    Users get platform wheels without knowing it’s there
    ●
    Tutorials that say "warning: this step will take ages" are now wrong
    

    View Slide

14. Tensorfow
    ●
    Ofcial Tensorfow support for Raspberry Pi
    ●
    Build not automated (no sdist provided)
    ●
    Wheels provided by Google, imported into piwheels
    

    View Slide

15. piwheels usage stats
    ●
    940,000 wheels
    ●
    400-500k downloads per month
    ●
    Total of 23 years of build time saved in 10 months
    ●
    See piwheels.org for more info
    

    View Slide