$30 off During Our Annual Pro Sale. View Details »

Why I run my own Python package repository

Ben Nuttall
September 16, 2018

Why I run my own Python package repository

Lightning talk given at PyCon UK about piwheels.org

Ben Nuttall

September 16, 2018
Tweet

More Decks by Ben Nuttall

Other Decks in Programming

Transcript

  1. Why I run my own Python package repository Ben Nuttall

  2. pip install numpy • Browse PyPI • Look for wheel

    with a matching platform tag • If one is found, download and install • If no match is found, download and build from source
  3. What kind of x86 are you?

  4. ...I have them all

  5. Actually I’m linux_armv7l

  6. What? I’ve never heard of it. Here’s the source. Build

    it yourself! • pip install numpy • ~20 mins on Pi 3 • ~2.5 hours on Pi 1
  7. Fine! I’ll build my own package repository... • cd Projects

    • mkdir piwheels • cd piwheels • git init
  8. pip wheel <package> • pip wheel numpy • numpy-1.15.1-cp35-cp35m-linux_armv7l.whl •

    That, but for all packages • 175,000 packages • That, but for all versions of all packages • 1,200,000 package versions • That, but for Python 3.4 and 3.5...
  9. I assume you cross-compile? Nope

  10. Mythic Beasts Pi cloud

  11. The stack • Master • Distributes jobs, hosts wheel fles

    and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
  12. The stack • Master • Distributes jobs, hosts wheel fles

    and web server • Postgres database • Monitors PyPI, generates build queue • Logs build attemps, logs downloads • Builders • Get given build jobs • 3 × Raspbian Jessie (Python 3.4) • 2 × Raspbian Stretch (Python 3.5) • Easy to scale up/down
  13. Using piwheels • Raspbian Stretch is pre-confgured to use piwheels.org

    as an additional index to PyPI • Users get platform wheels without knowing it’s there • Tutorials that say "warning: this step will take ages" are now wrong
  14. Tensorfow • Ofcial Tensorfow support for Raspberry Pi • Build

    not automated (no sdist provided) • Wheels provided by Google, imported into piwheels
  15. piwheels usage stats • 940,000 wheels • 400-500k downloads per

    month • Total of 23 years of build time saved in 10 months • See piwheels.org for more info