Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Build and test wheel packages on Linux, OSX and Windows

Build and test wheel packages on Linux, OSX and Windows

Presentation given at PyCon 2015

Olivier Grisel

April 11, 2015
Tweet

More Decks by Olivier Grisel

Other Decks in Technology

Transcript

  1. Build and test wheel
    packages on Linux, OSX
    & Windows
    Olivier Grisel - PyCon 2015

    View Slide

  2. Why me?
    What did I ever do
    to deserve this?

    View Slide

  3. About me

    View Slide

  4. View Slide

  5. photo: https://flic.kr/p/7qwCco

    View Slide

  6. Outline
    • The Wheel format
    • Testing under Linux and OSX with Travis-CI
    • Building OSX wheels
    • Testing under Windows with AppVeyor-CI
    • Building Windows wheels
    • Automating a release to PyPI
    • Tricky stuff: compilers on Windows and shared libraries

    View Slide

  7. The wheel format

    View Slide

  8. View Slide

  9. What?
    • A standard distribution format for Python packages:
    https://www.python.org/dev/peps/pep-0427
    • Simple: ZIP archive with Python files + static
    metadata + compiled extensions (optional)
    • Can be installed by pip 1.4+ (current is 6.1)
    • Can be generated by your usual setup.py or pip

    View Slide

  10. Why?
    • Smaller packages: faster to download, even for
    pure-Python packages
    • Faster to install, even for pure-Python packages
    • Possible to install platform specific Python
    packages without a compiler nor header files for
    third party libraries

    View Slide

  11. $ time pip install numpy scipy scikit-learn >/dev/null
    real 0m9.034s
    user 0m6.355s
    sys 0m2.119s
    $ python -c "import sklearn; print(sklearn.__version__)"
    0.16.0

    View Slide

  12. View Slide

  13. $ time pip install nose-1.3.6.tar.gz
    Processing ./nose-1.3.6.tar.gz
    Installing collected packages: nose
    Running setup.py install for nose
    Successfully installed nose-1.3.6
    real 0m8.818s
    user 0m8.152s
    sys 0m0.494s

    View Slide

  14. $ time pip install nose-1.3.6-py3-none-any.whl
    Unpacking ./nose-1.3.6-py3-none-any.whl
    Installing collected packages: nose
    Successfully installed nose
    Cleaning up...
    real 0m0.568s
    user 0m0.447s
    sys 0m0.106s

    View Slide

  15. $ unzip nose-1.3.6-py3-none-any.whl
    Archive: nose-1.3.6-py3-none-any.whl
    inflating: nose/__init__.py
    inflating: nose/__main__.py
    inflating: nose/case.py
    inflating: nose/commands.py
    [...]
    inflating: nose-1.3.6.data/data/man/man1/nosetests.1
    inflating: nose-1.3.6.dist-info/DESCRIPTION.rst
    inflating: nose-1.3.6.dist-info/entry_points.txt
    inflating: nose-1.3.6.dist-info/metadata.json
    inflating: nose-1.3.6.dist-info/top_level.txt
    inflating: nose-1.3.6.dist-info/WHEEL
    inflating: nose-1.3.6.dist-info/METADATA
    inflating: nose-1.3.6.dist-info/RECORD

    View Slide

  16. Building Universal
    Wheels

    View Slide

  17. # In your setup.cfg:
    [bdist_wheel]
    universal = 1
    # Then in a console:
    $ pip install wheel twine
    $ python setup.py bdist_wheel
    # At release time:
    $ python setup.py sdist bdist_wheel
    $ twine upload dist/*
    Universal Wheels
    Python 2 & 3, no compiled extensions

    View Slide

  18. Automated testing for
    Linux & Mac OSX

    View Slide

  19. View Slide

  20. View Slide

  21. .travis.yml to test on Linux
    language: python
    python:
    - "2.6"
    - "2.7"
    - "3.3"
    - "3.4"
    - "nightly"
    install:
    - "pip install ."
    script:
    - "(cd /tmp && nosetests -v mypackage)"

    View Slide

  22. .travis.yml to build and test OSX wheels
    language: objective-c
    env:
    matrix:
    - VERSION=2.7.9
    - VERSION=3.3.5
    - VERSION=3.4.3
    install:
    - "source terryfy/travis_tools.sh"
    - "get_python_environment macpython $VERSION venv"
    - "pip install nose wheel"
    - "python setup.py bdist_wheel"
    - "pip install dist/*.whl"
    script:
    - "(cd /tmp && nosetests -v mypackage)"

    View Slide

  23. OSX and Linux on travis
    • Multi-OS support:
    • Use $TRAVIS_OS_NAME to make scripts OS aware
    • Still need to setup custom Python env under OSX:
    https://github.com/MacPython/terryfy
    os:
    - linux
    - osx

    View Slide

  24. Automated testing for
    Windows

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. environment:
    matrix:
    - PYTHON: "C:\\Python27"
    - PYTHON: "C:\\Python34"
    install:
    - "powershell ./install_python_and_pip.ps1”
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - "pip install nose wheel”
    - "python setup.py bdist_wheel"
    - "pip install dist/*"
    build: false
    test_script:
    - "cd C:\\Temp"
    - "nosetests -v mypackage"
    appveyor.yml (without compiled extensions)

    View Slide

  29. environment:
    global:
    COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd"
    matrix:
    - PYTHON: "C:\\Python27"
    PYTHON_ARCH: "2.7.9"
    PYTHON_ARCH: "32"
    install:
    - "powershell ./install_python_and_pip.ps1"
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - "pip install nose wheel"
    - "%COMPILER_ENV% python setup.py bdist_wheel"
    - "pip install dist/*"
    build: false
    test_script:
    - "cd C:\\Temp"
    - "nosetests -v mypackage"
    appveyor.yml (with compiled extensions)

    View Slide

  30. Setting your own AppVeyor
    configuration
    • https://packaging.python.org/en/latest/
    appveyor.html
    • https://github.com/ogrisel/python-appveyor-demo

    View Slide

  31. View Slide

  32. Automating releases

    View Slide

  33. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels

    View Slide

  34. View Slide

  35. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid
    export WHEELHOUSE_UPLOADER_SECRET=xxx
    python -m wheelhouse_uploader upload container_id

    View Slide

  36. View Slide

  37. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    git checkout 0.16.0
    python setup.py fetch_artifacts

    View Slide

  38. setup.cfg
    [wheelhouse_uploader]
    artifact_indexes=
    # OSX wheels built by travis (only for specific tags):
    # https://github.com/MacPython/scikit-learn-wheels
    http://wheels.scipy.org/
    # Windows wheels buit by:
    # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/
    http://windows-wheels.scikit-learn.org/

    View Slide

  39. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    git checkout 0.16.0
    rm -rf dist && python setup.py sdist fetch_artifacts
    twine upload dist/*

    View Slide

  40. View Slide

  41. Windows compilers

    View Slide

  42. Installing compilers on
    Windows
    1st option: don’t do it, just use AppVeyor

    View Slide

  43. Installing compilers on
    Windows
    Install MSVC Express 2008 / 2010 + Windows SDK
    • Many non-scriptable manual steps
    • Windows SDK are several GBs in total to
    download
    http://scikit-learn.org/dev/install.html#building-on-
    windows

    View Slide

  44. View Slide

  45. MSVC for Python 2.7
    [+] Small .msi file with a fixed download URL:
    http://download.microsoft.com/download/
    7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/
    VCForPython27.msi
    [+] Support for 32 and 64 bit Python by default
    [-] Only C & C++, no fortran compiler (scipy…)
    [-] No support for Python 3.3 and 3.4 (need MSVC 2010)
    [+] Python 3.5 might be built with MSVC 2015 (forward compat)

    View Slide

  46. Carl Kleffner’s mingwpy
    • Static MinGW-w64 toolchain for 32 and 64 bit
    • Links against MSVC runtime libs for binary compat with
    official Python interpreters from python.org
    • Provides gcc, g++ and gfortran
    • Still experimental but already used by WinPython, a
    wheel-based binary distribution
    • http://github.com/mingwpy (new, created this morning)

    https://bitbucket.org/carlkl/mingw-w64-for-python (old)

    View Slide

  47. Dealing with
    external libraries

    View Slide

  48. External libraries
    • Examples:
    • lxml: libxml2 / libxslt
    • numpy & scipy: ATLAS, OpenBLAS or MKL
    • Under Windows:
    • Python ext: .pyd, Shared lib: .dll
    • Under OSX:
    • Python ext: .so, Shared lib: .dylib

    View Slide

  49. Possible solutions
    • Assume the library is present on the system:
    • BLAS / LAPACK for numpy & scipy under OSX
    • Use static linking: big .whl files
    • numpy with MKL by Christoph Gohlke
    • lxml under Windows (and OSX)
    • Embed the shared lib in the wheel archive

    View Slide

  50. Embedding a shared lib
    under OSX
    • delocate by Matthew Brett
    • https://github.com/matthew-brett/delocate

    View Slide

  51. $ delocate-listdeps scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

    View Slide

  52. $ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    /usr/lib/libSystem.B.dylib
    /usr/lib/libstdc++.6.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

    View Slide

  53. $ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    Copied to package .dylibs directory:
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib

    View Slide

  54. Embedding a shared lib
    under Windows
    • Put the .dll files next to the .pyd files under the
    site-packages folder
    • Put the .dll files in the Scripts folder
    • Improve Python to have an official way to
    deploy .dll files shipped with wheels.

    View Slide

  55. Thank you!
    • https://packaging.python.org
    • http://github.com/ogrisel/wheelhouse-uploader
    • https://speakerdeck.com/ogrisel
    • @ogrisel

    View Slide