Olivier Grisel - Build and test wheel packages on Linux, OSX & Windows

Olivier Grisel - Build and test wheel packages on Linux, OSX & Windows

Practical guide to build and test wheel packages for all platforms using free
Continuous Integration services such as Travis CI (Linux and OSX) and AppVeyor
(Windows).

https://us.pycon.org/2015/schedule/presentation/377/

D5710b3bca38f1233274b4cbc523dc4b?s=128

PyCon 2015

April 18, 2015
Tweet

Transcript

  1. Build and test wheel packages on Linux, OSX & Windows

    Olivier Grisel - PyCon 2015
  2. Why me? What did I ever do to deserve this?

  3. About me

  4. None
  5. photo: https://flic.kr/p/7qwCco

  6. Outline • The Wheel format • Testing under Linux and

    OSX with Travis-CI • Building OSX wheels • Testing under Windows with AppVeyor-CI • Building Windows wheels • Automating a release to PyPI • Tricky stuff: compilers on Windows and shared libraries
  7. The wheel format

  8. None
  9. What? • A standard distribution format for Python packages: https://www.python.org/dev/peps/pep-0427

    • Simple: ZIP archive with Python files + static metadata + compiled extensions (optional) • Can be installed by pip 1.4+ (current is 6.1) • Can be generated by your usual setup.py or pip
  10. Why? • Smaller packages: faster to download, even for pure-Python

    packages • Faster to install, even for pure-Python packages • Possible to install platform specific Python packages without a compiler nor header files for third party libraries
  11. $ time pip install numpy scipy scikit-learn >/dev/null real 0m9.034s

    user 0m6.355s sys 0m2.119s $ python -c "import sklearn; print(sklearn.__version__)" 0.16.0
  12. None
  13. $ time pip install nose-1.3.6.tar.gz Processing ./nose-1.3.6.tar.gz Installing collected packages:

    nose Running setup.py install for nose Successfully installed nose-1.3.6 real 0m8.818s user 0m8.152s sys 0m0.494s
  14. $ time pip install nose-1.3.6-py3-none-any.whl Unpacking ./nose-1.3.6-py3-none-any.whl Installing collected packages:

    nose Successfully installed nose Cleaning up... real 0m0.568s user 0m0.447s sys 0m0.106s
  15. $ unzip nose-1.3.6-py3-none-any.whl Archive: nose-1.3.6-py3-none-any.whl inflating: nose/__init__.py inflating: nose/__main__.py inflating:

    nose/case.py inflating: nose/commands.py [...] inflating: nose-1.3.6.data/data/man/man1/nosetests.1 inflating: nose-1.3.6.dist-info/DESCRIPTION.rst inflating: nose-1.3.6.dist-info/entry_points.txt inflating: nose-1.3.6.dist-info/metadata.json inflating: nose-1.3.6.dist-info/top_level.txt inflating: nose-1.3.6.dist-info/WHEEL inflating: nose-1.3.6.dist-info/METADATA inflating: nose-1.3.6.dist-info/RECORD
  16. Building Universal Wheels

  17. # In your setup.cfg: [bdist_wheel] universal = 1 # Then

    in a console: $ pip install wheel twine $ python setup.py bdist_wheel # At release time: $ python setup.py sdist bdist_wheel $ twine upload dist/* Universal Wheels Python 2 & 3, no compiled extensions
  18. Automated testing for Linux & Mac OSX

  19. None
  20. None
  21. .travis.yml to test on Linux language: python python: - "2.6"

    - "2.7" - "3.3" - "3.4" - "nightly" install: - "pip install ." script: - "(cd /tmp && nosetests -v mypackage)"
  22. .travis.yml to build and test OSX wheels language: objective-c env:

    matrix: - VERSION=2.7.9 - VERSION=3.3.5 - VERSION=3.4.3 install: - "source terryfy/travis_tools.sh" - "get_python_environment macpython $VERSION venv" - "pip install nose wheel" - "python setup.py bdist_wheel" - "pip install dist/*.whl" script: - "(cd /tmp && nosetests -v mypackage)"
  23. OSX and Linux on travis • Multi-OS support: • Use

    $TRAVIS_OS_NAME to make scripts OS aware • Still need to setup custom Python env under OSX: https://github.com/MacPython/terryfy os: - linux - osx
  24. Automated testing for Windows

  25. None
  26. None
  27. None
  28. environment: matrix: - PYTHON: "C:\\Python27" - PYTHON: "C:\\Python34" install: -

    "powershell ./install_python_and_pip.ps1” - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel” - "python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (without compiled extensions)
  29. environment: global: COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd" matrix: -

    PYTHON: "C:\\Python27" PYTHON_ARCH: "2.7.9" PYTHON_ARCH: "32" install: - "powershell ./install_python_and_pip.ps1" - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel" - "%COMPILER_ENV% python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (with compiled extensions)
  30. Setting your own AppVeyor configuration • https://packaging.python.org/en/latest/ appveyor.html • https://github.com/ogrisel/python-appveyor-demo

  31. None
  32. Automating releases

  33. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels
  34. None
  35. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid export WHEELHOUSE_UPLOADER_SECRET=xxx python -m wheelhouse_uploader upload container_id
  36. None
  37. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 python setup.py fetch_artifacts
  38. setup.cfg [wheelhouse_uploader] artifact_indexes= # OSX wheels built by travis (only

    for specific tags): # https://github.com/MacPython/scikit-learn-wheels http://wheels.scipy.org/ # Windows wheels buit by: # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/ http://windows-wheels.scikit-learn.org/
  39. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 rm -rf dist && python setup.py sdist fetch_artifacts twine upload dist/*
  40. None
  41. Windows compilers

  42. Installing compilers on Windows 1st option: don’t do it, just

    use AppVeyor
  43. Installing compilers on Windows Install MSVC Express 2008 / 2010

    + Windows SDK • Many non-scriptable manual steps • Windows SDK are several GBs in total to download http://scikit-learn.org/dev/install.html#building-on- windows
  44. None
  45. MSVC for Python 2.7 [+] Small .msi file with a

    fixed download URL: http://download.microsoft.com/download/ 7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/ VCForPython27.msi [+] Support for 32 and 64 bit Python by default [-] Only C & C++, no fortran compiler (scipy…) [-] No support for Python 3.3 and 3.4 (need MSVC 2010) [+] Python 3.5 might be built with MSVC 2015 (forward compat)
  46. Carl Kleffner’s mingwpy • Static MinGW-w64 toolchain for 32 and

    64 bit • Links against MSVC runtime libs for binary compat with official Python interpreters from python.org • Provides gcc, g++ and gfortran • Still experimental but already used by WinPython, a wheel-based binary distribution • http://github.com/mingwpy (new, created this morning)
 https://bitbucket.org/carlkl/mingw-w64-for-python (old)
  47. Dealing with external libraries

  48. External libraries • Examples: • lxml: libxml2 / libxslt •

    numpy & scipy: ATLAS, OpenBLAS or MKL • Under Windows: • Python ext: .pyd, Shared lib: .dll • Under OSX: • Python ext: .so, Shared lib: .dylib
  49. Possible solutions • Assume the library is present on the

    system: • BLAS / LAPACK for numpy & scipy under OSX • Use static linking: big .whl files • numpy with MKL by Christoph Gohlke • lxml under Windows (and OSX) • Embed the shared lib in the wheel archive
  50. Embedding a shared lib under OSX • delocate by Matthew

    Brett • https://github.com/matthew-brett/delocate
  51. $ delocate-listdeps scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

  52. $ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate /usr/lib/libSystem.B.dylib /usr/lib/libstdc++.6.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

  53. $ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Copied to package .dylibs

    directory: /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
  54. Embedding a shared lib under Windows • Put the .dll

    files next to the .pyd files under the site-packages folder • Put the .dll files in the Scripts folder • Improve Python to have an official way to deploy .dll files shipped with wheels.
  55. Thank you! • https://packaging.python.org • http://github.com/ogrisel/wheelhouse-uploader • https://speakerdeck.com/ogrisel • @ogrisel