Build and test wheel packages on Linux, OSX and Windows

Presentation given at PyCon 2015

Transcript

1. Build and test wheel packages on Linux, OSX & Windows Olivier Grisel - PyCon 2015
2. Why me? What did I ever do to deserve this?
3. About me
5. photo: https://flic.kr/p/7qwCco
6. Outline • The Wheel format • Testing under Linux and OSX with Travis-CI • Building OSX wheels • Testing under Windows with AppVeyor-CI • Building Windows wheels • Automating a release to PyPI • Tricky stuff: compilers on Windows and shared libraries
7. The wheel format
9. What? • A standard distribution format for Python packages: https://www.python.org/dev/peps/pep-0427 • Simple: ZIP archive with Python files + static metadata + compiled extensions (optional) • Can be installed by pip 1.4+ (current is 6.1) • Can be generated by your usual setup.py or pip
10. Why? • Smaller packages: faster to download, even for pure-Python packages • Faster to install, even for pure-Python packages • Possible to install platform specific Python packages without a compiler nor header files for third party libraries
11. $ time pip install numpy scipy scikit-learn >/dev/null real 0m9.034s user 0m6.355s sys 0m2.119s $ python -c "import sklearn; print(sklearn.__version__)" 0.16.0
13. $ time pip install nose-1.3.6.tar.gz Processing ./nose-1.3.6.tar.gz Installing collected packages: nose Running setup.py install for nose Successfully installed nose-1.3.6 real 0m8.818s user 0m8.152s sys 0m0.494s
14. $ time pip install nose-1.3.6-py3-none-any.whl Unpacking ./nose-1.3.6-py3-none-any.whl Installing collected packages: nose Successfully installed nose Cleaning up... real 0m0.568s user 0m0.447s sys 0m0.106s
15. $ unzip nose-1.3.6-py3-none-any.whl Archive: nose-1.3.6-py3-none-any.whl inflating: nose/__init__.py inflating: nose/__main__.py inflating: nose/case.py inflating: nose/commands.py [...] inflating: nose-1.3.6.data/data/man/man1/nosetests.1 inflating: nose-1.3.6.dist-info/DESCRIPTION.rst inflating: nose-1.3.6.dist-info/entry_points.txt inflating: nose-1.3.6.dist-info/metadata.json inflating: nose-1.3.6.dist-info/top_level.txt inflating: nose-1.3.6.dist-info/WHEEL inflating: nose-1.3.6.dist-info/METADATA inflating: nose-1.3.6.dist-info/RECORD
16. Building Universal Wheels
17. # In your setup.cfg: [bdist_wheel] universal = 1 # Then in a console: $ pip install wheel twine $ python setup.py bdist_wheel # At release time: $ python setup.py sdist bdist_wheel $ twine upload dist/* Universal Wheels Python 2 & 3, no compiled extensions
18. Automated testing for Linux & Mac OSX
21. .travis.yml to test on Linux language: python python: - "2.6" - "2.7" - "3.3" - "3.4" - "nightly" install: - "pip install ." script: - "(cd /tmp && nosetests -v mypackage)"
22. .travis.yml to build and test OSX wheels language: objective-c env: matrix: - VERSION=2.7.9 - VERSION=3.3.5 - VERSION=3.4.3 install: - "source terryfy/travis_tools.sh" - "get_python_environment macpython $VERSION venv" - "pip install nose wheel" - "python setup.py bdist_wheel" - "pip install dist/*.whl" script: - "(cd /tmp && nosetests -v mypackage)"
23. OSX and Linux on travis • Multi-OS support: • Use $TRAVIS_OS_NAME to make scripts OS aware • Still need to setup custom Python env under OSX: https://github.com/MacPython/terryfy os: - linux - osx
24. Automated testing for Windows
28. environment: matrix: - PYTHON: "C:\\Python27" - PYTHON: "C:\\Python34" install: - "powershell ./install_python_and_pip.ps1” - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel” - "python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (without compiled extensions)
29. environment: global: COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd" matrix: - PYTHON: "C:\\Python27" PYTHON_ARCH: "2.7.9" PYTHON_ARCH: "32" install: - "powershell ./install_python_and_pip.ps1" - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel" - "%COMPILER_ENV% python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (with compiled extensions)
30. Setting your own AppVeyor configuration • https://packaging.python.org/en/latest/ appveyor.html • https://github.com/ogrisel/python-appveyor-demo
32. Automating releases
33. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels
35. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid export WHEELHOUSE_UPLOADER_SECRET=xxx python -m wheelhouse_uploader upload container_id
37. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 python setup.py fetch_artifacts
38. setup.cfg [wheelhouse_uploader] artifact_indexes= # OSX wheels built by travis (only for specific tags): # https://github.com/MacPython/scikit-learn-wheels http://wheels.scipy.org/ # Windows wheels buit by: # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/ http://windows-wheels.scikit-learn.org/
39. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 rm -rf dist && python setup.py sdist fetch_artifacts twine upload dist/*
41. Windows compilers
42. Installing compilers on Windows 1st option: don’t do it, just use AppVeyor
43. Installing compilers on Windows Install MSVC Express 2008 / 2010 + Windows SDK • Many non-scriptable manual steps • Windows SDK are several GBs in total to download http://scikit-learn.org/dev/install.html#building-on- windows
45. MSVC for Python 2.7 [+] Small .msi file with a fixed download URL: http://download.microsoft.com/download/ 7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/ VCForPython27.msi [+] Support for 32 and 64 bit Python by default [-] Only C & C++, no fortran compiler (scipy…) [-] No support for Python 3.3 and 3.4 (need MSVC 2010) [+] Python 3.5 might be built with MSVC 2015 (forward compat)
46. Carl Kleffner’s mingwpy • Static MinGW-w64 toolchain for 32 and 64 bit • Links against MSVC runtime libs for binary compat with official Python interpreters from python.org • Provides gcc, g++ and gfortran • Still experimental but already used by WinPython, a wheel-based binary distribution • http://github.com/mingwpy (new, created this morning)
 https://bitbucket.org/carlkl/mingw-w64-for-python (old)
47. Dealing with external libraries
48. External libraries • Examples: • lxml: libxml2 / libxslt • numpy & scipy: ATLAS, OpenBLAS or MKL • Under Windows: • Python ext: .pyd, Shared lib: .dll • Under OSX: • Python ext: .so, Shared lib: .dylib
49. Possible solutions • Assume the library is present on the system: • BLAS / LAPACK for numpy & scipy under OSX • Use static linking: big .whl files • numpy with MKL by Christoph Gohlke • lxml under Windows (and OSX) • Embed the shared lib in the wheel archive
50. Embedding a shared lib under OSX • delocate by Matthew Brett • https://github.com/matthew-brett/delocate
51. $ delocate-listdeps scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
52. $ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
53. $ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Copied to package .dylibs directory: /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
54. Embedding a shared lib under Windows • Put the .dll files next to the .pyd files under the site-packages folder • Put the .dll files in the Scripts folder • Improve Python to have an official way to deploy .dll files shipped with wheels.
55. Thank you! • https://packaging.python.org • http://github.com/ogrisel/wheelhouse-uploader • https://speakerdeck.com/ogrisel • @ogrisel