Olivier Grisel - Build and test wheel packages on Linux, OSX & Windows

Slide 1

Slide 1 text

Build and test wheel packages on Linux, OSX & Windows Olivier Grisel - PyCon 2015

Slide 2

Slide 2 text

Why me? What did I ever do to deserve this?

Slide 3

Slide 3 text

About me

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

photo: https://ﬂic.kr/p/7qwCco

Slide 6

Slide 6 text

Outline • The Wheel format • Testing under Linux and OSX with Travis-CI • Building OSX wheels • Testing under Windows with AppVeyor-CI • Building Windows wheels • Automating a release to PyPI • Tricky stuff: compilers on Windows and shared libraries

Slide 7

Slide 7 text

The wheel format

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

What? • A standard distribution format for Python packages: https://www.python.org/dev/peps/pep-0427 • Simple: ZIP archive with Python ﬁles + static metadata + compiled extensions (optional) • Can be installed by pip 1.4+ (current is 6.1) • Can be generated by your usual setup.py or pip

Slide 10

Slide 10 text

Why? • Smaller packages: faster to download, even for pure-Python packages • Faster to install, even for pure-Python packages • Possible to install platform speciﬁc Python packages without a compiler nor header ﬁles for third party libraries

Slide 11

Slide 11 text

$ time pip install numpy scipy scikit-learn >/dev/null real 0m9.034s user 0m6.355s sys 0m2.119s $ python -c "import sklearn; print(sklearn.__version__)" 0.16.0

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

$ time pip install nose-1.3.6.tar.gz Processing ./nose-1.3.6.tar.gz Installing collected packages: nose Running setup.py install for nose Successfully installed nose-1.3.6 real 0m8.818s user 0m8.152s sys 0m0.494s

Slide 14

Slide 14 text

$ time pip install nose-1.3.6-py3-none-any.whl Unpacking ./nose-1.3.6-py3-none-any.whl Installing collected packages: nose Successfully installed nose Cleaning up... real 0m0.568s user 0m0.447s sys 0m0.106s

Slide 15

Slide 15 text

$ unzip nose-1.3.6-py3-none-any.whl Archive: nose-1.3.6-py3-none-any.whl inflating: nose/__init__.py inflating: nose/__main__.py inflating: nose/case.py inflating: nose/commands.py [...] inflating: nose-1.3.6.data/data/man/man1/nosetests.1 inflating: nose-1.3.6.dist-info/DESCRIPTION.rst inflating: nose-1.3.6.dist-info/entry_points.txt inflating: nose-1.3.6.dist-info/metadata.json inflating: nose-1.3.6.dist-info/top_level.txt inflating: nose-1.3.6.dist-info/WHEEL inflating: nose-1.3.6.dist-info/METADATA inflating: nose-1.3.6.dist-info/RECORD

Slide 16

Slide 16 text

Building Universal Wheels

Slide 17

Slide 17 text

# In your setup.cfg: [bdist_wheel] universal = 1 # Then in a console: $ pip install wheel twine $ python setup.py bdist_wheel # At release time: $ python setup.py sdist bdist_wheel $ twine upload dist/* Universal Wheels Python 2 & 3, no compiled extensions

Slide 18

Slide 18 text

Automated testing for Linux & Mac OSX

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

.travis.yml to test on Linux language: python python: - "2.6" - "2.7" - "3.3" - "3.4" - "nightly" install: - "pip install ." script: - "(cd /tmp && nosetests -v mypackage)"

Slide 22

Slide 22 text

.travis.yml to build and test OSX wheels language: objective-c env: matrix: - VERSION=2.7.9 - VERSION=3.3.5 - VERSION=3.4.3 install: - "source terryfy/travis_tools.sh" - "get_python_environment macpython $VERSION venv" - "pip install nose wheel" - "python setup.py bdist_wheel" - "pip install dist/*.whl" script: - "(cd /tmp && nosetests -v mypackage)"

Slide 23

Slide 23 text

OSX and Linux on travis • Multi-OS support: • Use $TRAVIS_OS_NAME to make scripts OS aware • Still need to setup custom Python env under OSX: https://github.com/MacPython/terryfy os: - linux - osx

Slide 24

Slide 24 text

Automated testing for Windows

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

environment: matrix: - PYTHON: "C:\\Python27" - PYTHON: "C:\\Python34" install: - "powershell ./install_python_and_pip.ps1” - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel” - "python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (without compiled extensions)

Slide 29

Slide 29 text

environment: global: COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd" matrix: - PYTHON: "C:\\Python27" PYTHON_ARCH: "2.7.9" PYTHON_ARCH: "32" install: - "powershell ./install_python_and_pip.ps1" - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel" - "%COMPILER_ENV% python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (with compiled extensions)

Slide 30

Slide 30 text

Setting your own AppVeyor conﬁguration • https://packaging.python.org/en/latest/ appveyor.html • https://github.com/ogrisel/python-appveyor-demo

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

Automating releases

Slide 33

Slide 33 text

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for ofﬁcial released wheels export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid export WHEELHOUSE_UPLOADER_SECRET=xxx python -m wheelhouse_uploader upload container_id

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for ofﬁcial released wheels git checkout 0.16.0 python setup.py fetch_artifacts

Slide 38

Slide 38 text

setup.cfg [wheelhouse_uploader] artifact_indexes= # OSX wheels built by travis (only for specific tags): # https://github.com/MacPython/scikit-learn-wheels http://wheels.scipy.org/ # Windows wheels buit by: # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/ http://windows-wheels.scikit-learn.org/

Slide 39

Slide 39 text

travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for ofﬁcial released wheels git checkout 0.16.0 rm -rf dist && python setup.py sdist fetch_artifacts twine upload dist/*

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Windows compilers

Slide 42

Slide 42 text

Installing compilers on Windows 1st option: don’t do it, just use AppVeyor

Slide 43

Slide 43 text

Installing compilers on Windows Install MSVC Express 2008 / 2010 + Windows SDK • Many non-scriptable manual steps • Windows SDK are several GBs in total to download http://scikit-learn.org/dev/install.html#building-on- windows

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

MSVC for Python 2.7 [+] Small .msi ﬁle with a ﬁxed download URL: http://download.microsoft.com/download/ 7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/ VCForPython27.msi [+] Support for 32 and 64 bit Python by default [-] Only C & C++, no fortran compiler (scipy…) [-] No support for Python 3.3 and 3.4 (need MSVC 2010) [+] Python 3.5 might be built with MSVC 2015 (forward compat)

Slide 46

Slide 46 text

Carl Kleffner’s mingwpy • Static MinGW-w64 toolchain for 32 and 64 bit • Links against MSVC runtime libs for binary compat with ofﬁcial Python interpreters from python.org • Provides gcc, g++ and gfortran • Still experimental but already used by WinPython, a wheel-based binary distribution • http://github.com/mingwpy (new, created this morning)  https://bitbucket.org/carlkl/mingw-w64-for-python (old)

Slide 47

Slide 47 text

Dealing with external libraries

Slide 48

Slide 48 text

External libraries • Examples: • lxml: libxml2 / libxslt • numpy & scipy: ATLAS, OpenBLAS or MKL • Under Windows: • Python ext: .pyd, Shared lib: .dll • Under OSX: • Python ext: .so, Shared lib: .dylib

Slide 49

Slide 49 text

Possible solutions • Assume the library is present on the system: • BLAS / LAPACK for numpy & scipy under OSX • Use static linking: big .whl ﬁles • numpy with MKL by Christoph Gohlke • lxml under Windows (and OSX) • Embed the shared lib in the wheel archive

Slide 50

Slide 50 text

Embedding a shared lib under OSX • delocate by Matthew Brett • https://github.com/matthew-brett/delocate

Slide 51

Slide 51 text

$ delocate-listdeps scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

Slide 52

Slide 52 text

$ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

Slide 53

Slide 53 text

$ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Copied to package .dylibs directory: /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

Slide 54

Slide 54 text

Embedding a shared lib under Windows • Put the .dll files next to the .pyd files under the site-packages folder • Put the .dll files in the Scripts folder • Improve Python to have an official way to deploy .dll files shipped with wheels.

Slide 55

Slide 55 text

Thank you! • https://packaging.python.org • http://github.com/ogrisel/wheelhouse-uploader • https://speakerdeck.com/ogrisel • @ogrisel