Why I run my own Python package repository

A1995c9abe48450ce2f82d93ca0b863f?s=47 Ben Nuttall
September 16, 2018

Why I run my own Python package repository

Lightning talk given at PyCon UK about piwheels.org

A1995c9abe48450ce2f82d93ca0b863f?s=128

Ben Nuttall

September 16, 2018
Tweet

Transcript

  1. Why I run my own Python package repository Ben Nuttall

  2. pip install numpy • Browse PyPI • Look for wheel

    with a matching platform tag • If one is found, download and install • If no match is found, download and build from source
  3. What kind of x86 are you?

  4. ...I have them all

  5. Actually I’m linux_armv7l

  6. What? I’ve never heard of it. Here’s the source. Build

    it yourself! • pip install numpy • ~20 mins on Pi 3 • ~2.5 hours on Pi 1
  7. Fine! I’ll build my own package repository... • cd Projects

    • mkdir piwheels • cd piwheels • git init
  8. pip wheel <package> • pip wheel numpy • numpy-1.15.1-cp35-cp35m-linux_armv7l.whl •

    That, but for all packages • 175,000 packages • That, but for all versions of all packages • 1,200,000 package versions • That, but for Python 3.4 and 3.5...
  9. I assume you cross-compile? Nope

  10. Mythic Beasts Pi cloud

  11. The stack • Master • Distributes jobs, hosts wheel fles

    and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
  12. The stack • Master • Distributes jobs, hosts wheel fles

    and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
  13. Using piwheels • Raspbian Stretch is pre-confgured to use piwheels.org

    as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
  14. Tensorfow • Ofcial Tensorfow support for Raspberry Pi • Build

    not automated (no sdist provided) • Wheels provided by Google, imported into piwheels
  15. piwheels usage stats • 940,000 wheels • 400-500k downloads per

    month • Total of 23 years of build time saved in 10 months • See piwheels.org for more info