Slide 1

Slide 1 text

Why I run my own Python package repository Ben Nuttall

Slide 2

Slide 2 text

pip install numpy ● Browse PyPI ● Look for wheel with a matching platform tag ● If one is found, download and install ● If no match is found, download and build from source

Slide 3

Slide 3 text

What kind of x86 are you?

Slide 4

Slide 4 text

...I have them all

Slide 5

Slide 5 text

Actually I’m linux_armv7l

Slide 6

Slide 6 text

What? I’ve never heard of it. Here’s the source. Build it yourself! ● pip install numpy ● ~20 mins on Pi 3 ● ~2.5 hours on Pi 1

Slide 7

Slide 7 text

Fine! I’ll build my own package repository... ● cd Projects ● mkdir piwheels ● cd piwheels ● git init

Slide 8

Slide 8 text

pip wheel ● pip wheel numpy ● numpy-1.15.1-cp35-cp35m-linux_armv7l.whl ● That, but for all packages ● 175,000 packages ● That, but for all versions of all packages ● 1,200,000 package versions ● That, but for Python 3.4 and 3.5...

Slide 9

Slide 9 text

I assume you cross-compile? Nope

Slide 10

Slide 10 text

Mythic Beasts Pi cloud

Slide 11

Slide 11 text

The stack ● Master ● Distributes jobs, hosts wheel fles and web server ● Postgres database ● Monitors PyPI, generates build queue ● Logs build attemps, logs downloads ● Builders ● Get given build jobs ● 3 × Raspbian Jessie (Python 3.4) ● 2 × Raspbian Stretch (Python 3.5) ● Easy to scale up/down

Slide 12

Slide 12 text

The stack ● Master ● Distributes jobs, hosts wheel fles and web server ● Postgres database ● Monitors PyPI, generates build queue ● Logs build attemps, logs downloads ● Builders ● Get given build jobs ● 3 × Raspbian Jessie (Python 3.4) ● 2 × Raspbian Stretch (Python 3.5) ● Easy to scale up/down

Slide 13

Slide 13 text

Using piwheels ● Raspbian Stretch is pre-confgured to use piwheels.org as an additional index to PyPI ● Users get platform wheels without knowing it’s there ● Tutorials that say "warning: this step will take ages" are now wrong

Slide 14

Slide 14 text

Tensorfow ● Ofcial Tensorfow support for Raspberry Pi ● Build not automated (no sdist provided) ● Wheels provided by Google, imported into piwheels

Slide 15

Slide 15 text

piwheels usage stats ● 940,000 wheels ● 400-500k downloads per month ● Total of 23 years of build time saved in 10 months ● See piwheels.org for more info