$30 off During Our Annual Pro Sale. View Details »

Why I run my own Python package repository

Ben Nuttall
September 16, 2018

Why I run my own Python package repository

Lightning talk given at PyCon UK about piwheels.org

Ben Nuttall

September 16, 2018
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. Why I run my own Python package repository
    Ben Nuttall

    View Slide

  2. pip install numpy

    Browse PyPI

    Look for wheel with a matching platform tag

    If one is found, download and install

    If no match is found, download and build from source

    View Slide

  3. What kind of x86 are you?

    View Slide

  4. ...I have them all

    View Slide

  5. Actually I’m linux_armv7l

    View Slide

  6. What? I’ve never heard of it.
    Here’s the source. Build it yourself!

    pip install numpy

    ~20 mins on Pi 3

    ~2.5 hours on Pi 1

    View Slide

  7. Fine! I’ll build my own package repository...

    cd Projects

    mkdir piwheels

    cd piwheels

    git init

    View Slide

  8. pip wheel

    pip wheel numpy

    numpy-1.15.1-cp35-cp35m-linux_armv7l.whl

    That, but for all packages

    175,000 packages

    That, but for all versions of all packages

    1,200,000 package versions

    That, but for Python 3.4 and 3.5...

    View Slide

  9. I assume you cross-compile?
    Nope

    View Slide

  10. Mythic Beasts Pi cloud

    View Slide

  11. The stack

    Master

    Distributes jobs, hosts wheel fles and web server

    Postgres database

    Monitors PyPI, generates build queue

    Logs build attemps, logs downloads

    Builders

    Get given build jobs

    3 × Raspbian Jessie (Python 3.4)

    2 × Raspbian Stretch (Python 3.5)

    Easy to scale up/down

    View Slide

  12. The stack

    Master

    Distributes jobs, hosts wheel fles and web server

    Postgres database

    Monitors PyPI, generates build queue

    Logs build attemps, logs downloads

    Builders

    Get given build jobs

    3 × Raspbian Jessie (Python 3.4)

    2 × Raspbian Stretch (Python 3.5)

    Easy to scale up/down

    View Slide

  13. Using piwheels

    Raspbian Stretch is pre-confgured to use piwheels.org as an additional
    index to PyPI

    Users get platform wheels without knowing it’s there

    Tutorials that say "warning: this step will take ages" are now wrong

    View Slide

  14. Tensorfow

    Ofcial Tensorfow support for Raspberry Pi

    Build not automated (no sdist provided)

    Wheels provided by Google, imported into piwheels

    View Slide

  15. piwheels usage stats

    940,000 wheels

    400-500k downloads per month

    Total of 23 years of build time saved in 10 months

    See piwheels.org for more info

    View Slide