Build and test wheel packages on Linux, OSX and Windows

Presentation given at PyCon 2015

Transcript

1. Build and test wheel
   packages on Linux, OSX
   & Windows
   Olivier Grisel - PyCon 2015
   

   View Slide

2. Why me?
   What did I ever do
   to deserve this?
   

   View Slide

3. About me
   

   View Slide

4. View Slide

5. photo: https://flic.kr/p/7qwCco
   

   View Slide

6. Outline
   • The Wheel format
   • Testing under Linux and OSX with Travis-CI
   • Building OSX wheels
   • Testing under Windows with AppVeyor-CI
   • Building Windows wheels
   • Automating a release to PyPI
   • Tricky stuff: compilers on Windows and shared libraries
   

   View Slide

7. The wheel format
   

   View Slide

8. View Slide

9. What?
   • A standard distribution format for Python packages:
   https://www.python.org/dev/peps/pep-0427
   • Simple: ZIP archive with Python files + static
   metadata + compiled extensions (optional)
   • Can be installed by pip 1.4+ (current is 6.1)
   • Can be generated by your usual setup.py or pip
   

   View Slide

10. Why?
    • Smaller packages: faster to download, even for
    pure-Python packages
    • Faster to install, even for pure-Python packages
    • Possible to install platform specific Python
    packages without a compiler nor header files for
    third party libraries
    

    View Slide

11. $ time pip install numpy scipy scikit-learn >/dev/null
    real 0m9.034s
    user 0m6.355s
    sys 0m2.119s
    $ python -c "import sklearn; print(sklearn.__version__)"
    0.16.0
    

    View Slide

12. View Slide

13. $ time pip install nose-1.3.6.tar.gz
    Processing ./nose-1.3.6.tar.gz
    Installing collected packages: nose
    Running setup.py install for nose
    Successfully installed nose-1.3.6
    real 0m8.818s
    user 0m8.152s
    sys 0m0.494s
    

    View Slide

14. $ time pip install nose-1.3.6-py3-none-any.whl
    Unpacking ./nose-1.3.6-py3-none-any.whl
    Installing collected packages: nose
    Successfully installed nose
    Cleaning up...
    real 0m0.568s
    user 0m0.447s
    sys 0m0.106s
    

    View Slide

15. $ unzip nose-1.3.6-py3-none-any.whl
    Archive: nose-1.3.6-py3-none-any.whl
    inflating: nose/__init__.py
    inflating: nose/__main__.py
    inflating: nose/case.py
    inflating: nose/commands.py
    [...]
    inflating: nose-1.3.6.data/data/man/man1/nosetests.1
    inflating: nose-1.3.6.dist-info/DESCRIPTION.rst
    inflating: nose-1.3.6.dist-info/entry_points.txt
    inflating: nose-1.3.6.dist-info/metadata.json
    inflating: nose-1.3.6.dist-info/top_level.txt
    inflating: nose-1.3.6.dist-info/WHEEL
    inflating: nose-1.3.6.dist-info/METADATA
    inflating: nose-1.3.6.dist-info/RECORD
    

    View Slide

16. Building Universal
    Wheels
    

    View Slide

17. # In your setup.cfg:
    [bdist_wheel]
    universal = 1
    # Then in a console:
    $ pip install wheel twine
    $ python setup.py bdist_wheel
    # At release time:
    $ python setup.py sdist bdist_wheel
    $ twine upload dist/*
    Universal Wheels
    Python 2 & 3, no compiled extensions
    

    View Slide

18. Automated testing for
    Linux & Mac OSX
    

    View Slide

19. View Slide

20. View Slide

21. .travis.yml to test on Linux
    language: python
    python:
    - "2.6"
    - "2.7"
    - "3.3"
    - "3.4"
    - "nightly"
    install:
    - "pip install ."
    script:
    - "(cd /tmp && nosetests -v mypackage)"
    

    View Slide

22. .travis.yml to build and test OSX wheels
    language: objective-c
    env:
    matrix:
    - VERSION=2.7.9
    - VERSION=3.3.5
    - VERSION=3.4.3
    install:
    - "source terryfy/travis_tools.sh"
    - "get_python_environment macpython $VERSION venv"
    - "pip install nose wheel"
    - "python setup.py bdist_wheel"
    - "pip install dist/*.whl"
    script:
    - "(cd /tmp && nosetests -v mypackage)"
    

    View Slide

23. OSX and Linux on travis
    • Multi-OS support:
    • Use $TRAVIS_OS_NAME to make scripts OS aware
    • Still need to setup custom Python env under OSX:
    https://github.com/MacPython/terryfy
    os:
    - linux
    - osx
    

    View Slide

24. Automated testing for
    Windows
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. environment:
    matrix:
    - PYTHON: "C:\\Python27"
    - PYTHON: "C:\\Python34"
    install:
    - "powershell ./install_python_and_pip.ps1”
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - "pip install nose wheel”
    - "python setup.py bdist_wheel"
    - "pip install dist/*"
    build: false
    test_script:
    - "cd C:\\Temp"
    - "nosetests -v mypackage"
    appveyor.yml (without compiled extensions)
    

    View Slide

29. environment:
    global:
    COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd"
    matrix:
    - PYTHON: "C:\\Python27"
    PYTHON_ARCH: "2.7.9"
    PYTHON_ARCH: "32"
    install:
    - "powershell ./install_python_and_pip.ps1"
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - "pip install nose wheel"
    - "%COMPILER_ENV% python setup.py bdist_wheel"
    - "pip install dist/*"
    build: false
    test_script:
    - "cd C:\\Temp"
    - "nosetests -v mypackage"
    appveyor.yml (with compiled extensions)
    

    View Slide

30. Setting your own AppVeyor
    configuration
    • https://packaging.python.org/en/latest/
    appveyor.html
    • https://github.com/ogrisel/python-appveyor-demo
    

    View Slide

31. View Slide

32. Automating releases
    

    View Slide

33. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    

    View Slide

34. View Slide

35. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid
    export WHEELHOUSE_UPLOADER_SECRET=xxx
    python -m wheelhouse_uploader upload container_id
    

    View Slide

36. View Slide

37. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    git checkout 0.16.0
    python setup.py fetch_artifacts
    

    View Slide

38. setup.cfg
    [wheelhouse_uploader]
    artifact_indexes=
    # OSX wheels built by travis (only for specific tags):
    # https://github.com/MacPython/scikit-learn-wheels
    http://wheels.scipy.org/
    # Windows wheels buit by:
    # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/
    http://windows-wheels.scikit-learn.org/
    

    View Slide

39. travis-ci
    OSX
    Rackspace CloudFiles
    wheels container for OS X
    Release Manager’s laptop
    appveyor-ci
    Windows
    Rackspace CloudFiles
    wheels container for
    Windows
    PyPI archive
    for official released
    wheels
    git checkout 0.16.0
    rm -rf dist && python setup.py sdist fetch_artifacts
    twine upload dist/*
    

    View Slide

40. View Slide

41. Windows compilers
    

    View Slide

42. Installing compilers on
    Windows
    1st option: don’t do it, just use AppVeyor
    

    View Slide

43. Installing compilers on
    Windows
    Install MSVC Express 2008 / 2010 + Windows SDK
    • Many non-scriptable manual steps
    • Windows SDK are several GBs in total to
    download
    http://scikit-learn.org/dev/install.html#building-on-
    windows
    

    View Slide

44. View Slide

45. MSVC for Python 2.7
    [+] Small .msi file with a fixed download URL:
    http://download.microsoft.com/download/
    7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/
    VCForPython27.msi
    [+] Support for 32 and 64 bit Python by default
    [-] Only C & C++, no fortran compiler (scipy…)
    [-] No support for Python 3.3 and 3.4 (need MSVC 2010)
    [+] Python 3.5 might be built with MSVC 2015 (forward compat)
    

    View Slide

46. Carl Kleffner’s mingwpy
    • Static MinGW-w64 toolchain for 32 and 64 bit
    • Links against MSVC runtime libs for binary compat with
    official Python interpreters from python.org
    • Provides gcc, g++ and gfortran
    • Still experimental but already used by WinPython, a
    wheel-based binary distribution
    • http://github.com/mingwpy (new, created this morning)

    https://bitbucket.org/carlkl/mingw-w64-for-python (old)
    

    View Slide

47. Dealing with
    external libraries
    

    View Slide

48. External libraries
    • Examples:
    • lxml: libxml2 / libxslt
    • numpy & scipy: ATLAS, OpenBLAS or MKL
    • Under Windows:
    • Python ext: .pyd, Shared lib: .dll
    • Under OSX:
    • Python ext: .so, Shared lib: .dylib
    

    View Slide

49. Possible solutions
    • Assume the library is present on the system:
    • BLAS / LAPACK for numpy & scipy under OSX
    • Use static linking: big .whl files
    • numpy with MKL by Christoph Gohlke
    • lxml under Windows (and OSX)
    • Embed the shared lib in the wheel archive
    

    View Slide

50. Embedding a shared lib
    under OSX
    • delocate by Matthew Brett
    • https://github.com/matthew-brett/delocate
    

    View Slide

51. $ delocate-listdeps scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
    

    View Slide

52. $ delocate-listdeps --all scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    /usr/lib/libSystem.B.dylib
    /usr/lib/libstdc++.6.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
    

    View Slide

53. $ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl
    Copied to package .dylibs directory:
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib
    /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
    

    View Slide

54. Embedding a shared lib
    under Windows
    • Put the .dll files next to the .pyd files under the
    site-packages folder
    • Put the .dll files in the Scripts folder
    • Improve Python to have an official way to
    deploy .dll files shipped with wheels.
    

    View Slide

55. Thank you!
    • https://packaging.python.org
    • http://github.com/ogrisel/wheelhouse-uploader
    • https://speakerdeck.com/ogrisel
    • @ogrisel
    

    View Slide