$30 off During Our Annual Pro Sale. View Details »

Harness the speed of the wheel

Harness the speed of the wheel

Paris.py 5 - October 16th 2014

Xavier Fernandez

October 16, 2014
Tweet

Other Decks in Programming

Transcript

  1. Harness the speed of the wheel
    Xavier Fernandez - @xavierfernandez
    Polyconseil
    October 16, 2014
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 1 / 13

    View Slide

  2. What is Wheel?
    Quick overview
    a package format
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  3. What is Wheel?
    Quick overview
    a package format
    defined in PEP 427 ”The Wheel Binary Package Format 1.0”
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  4. What is Wheel?
    Quick overview
    a package format
    defined in PEP 427 ”The Wheel Binary Package Format 1.0”
    doc available at http://wheel.rtfd.org/
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  5. What is Wheel?
    Quick overview
    a package format
    defined in PEP 427 ”The Wheel Binary Package Format 1.0”
    doc available at http://wheel.rtfd.org/
    ZIP-format archive (with a structure close to PEP376)
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  6. What is Wheel?
    Quick overview
    a package format
    defined in PEP 427 ”The Wheel Binary Package Format 1.0”
    doc available at http://wheel.rtfd.org/
    ZIP-format archive (with a structure close to PEP376)
    versioned
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  7. What is Wheel?
    Quick overview
    a package format
    defined in PEP 427 ”The Wheel Binary Package Format 1.0”
    doc available at http://wheel.rtfd.org/
    ZIP-format archive (with a structure close to PEP376)
    versioned
    with a specially formatted file name and the .whl extension
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13

    View Slide

  8. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Examples
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  9. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Used by the installer to select the ’best’ file
    Examples
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  10. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Used by the installer to select the ’best’ file
    Examples
    Django-1.7-py2.py3-none-any.whl
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  11. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Used by the installer to select the ’best’ file
    Examples
    Django-1.7-py2.py3-none-any.whl
    aioamqp-0.1.1-py33.py34-none-any.whl
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  12. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Used by the installer to select the ’best’ file
    Examples
    Django-1.7-py2.py3-none-any.whl
    aioamqp-0.1.1-py33.py34-none-any.whl
    lxml-3.4.0-cp26-none-win32.whl
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  13. What is Wheel?
    Formatted name
    {distribution}-{version}(-{build tag})?-{python tag}-{abi
    tag}-{platform tag}.whl
    Used by the installer to select the ’best’ file
    Examples
    Django-1.7-py2.py3-none-any.whl
    aioamqp-0.1.1-py33.py34-none-any.whl
    lxml-3.4.0-cp26-none-win32.whl
    Pillow-2.6.1-cp34-cp34m-
    macosx 10 6 intel.macosx 10 9 intel.macosx 10 9 x86 64.whl
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13

    View Slide

  14. Fast
    Installation
    no setup.py execution
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  15. Fast
    Installation
    no setup.py execution
    basically just an unzip
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  16. Fast
    Installation
    no setup.py execution
    basically just an unzip
    (... with minimum magic like for instance shebang rewrite for scripts)
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  17. Fast
    Installation
    no setup.py execution
    basically just an unzip
    (... with minimum magic like for instance shebang rewrite for scripts)
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  18. Fast
    Installation
    no setup.py execution
    basically just an unzip
    (... with minimum magic like for instance shebang rewrite for scripts)
    Experimental protocol
    pip install –no-index –upgrade –force-reinstall
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  19. Fast
    Installation
    no setup.py execution
    basically just an unzip
    (... with minimum magic like for instance shebang rewrite for scripts)
    Experimental protocol
    pip install –no-index –upgrade –force-reinstall
    with –find-links pointing to either wheels or tar.gz
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  20. Fast
    Installation
    no setup.py execution
    basically just an unzip
    (... with minimum magic like for instance shebang rewrite for scripts)
    Experimental protocol
    pip install –no-index –upgrade –force-reinstall
    with –find-links pointing to either wheels or tar.gz
    repeated between 4 and 10 times
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13

    View Slide

  21. Benchmark
    (re)Installation times of pure python modules
    Module .tar.gz .whl Ratio
    pep8-1.5.0 0.65 sec 0.27 sec x 2.4
    flake8-2.1.0 1.84 sec 0.37 sec x 5.0
    mccabe-0.2.1 0.61 sec 0.25 sec x 2.4
    pyflakes-0.8 0.67 sec 0.29 sec x 2.3
    Django-1.6.4 10.6 sec 6.76 sec x 1.6
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 5 / 13

    View Slide

  22. Benchmark
    (re)Installation times of pure python modules
    Module .tar.gz .whl Ratio
    pep8-1.5.0 0.65 sec 0.27 sec x 2.4
    flake8-2.1.0 1.84 sec 0.37 sec x 5.0
    mccabe-0.2.1 0.61 sec 0.25 sec x 2.4
    pyflakes-0.8 0.67 sec 0.29 sec x 2.3
    Django-1.6.4 10.6 sec 6.76 sec x 1.6
    (re)Installation times of python modules needing some compilation
    Module .tar.gz .whl Ratio
    lxml-3.3.5 60 sec 0.42 sec x 143
    numpy-1.8.1 208 sec 2.0 sec x 104
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 5 / 13

    View Slide

  23. So how do we use it?
    First
    Distribute your own pure python libraries in this format, it’s easy:
    pip install wheel
    setup.py based on setuptools:
    python setup.py bdist wheel
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13

    View Slide

  24. So how do we use it?
    First
    Distribute your own pure python libraries in this format, it’s easy:
    pip install wheel
    setup.py based on setuptools:
    python setup.py bdist wheel
    setup.py based on distutils alone:
    python -c "import setuptools; file = ’setup.py’;
    setup code = open( file ).read().replace(’\r\n’, ’\n’);
    exec(compile(setup code, file , ’exec’))" bdist wheel
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13

    View Slide

  25. So how do we use it?
    First
    Distribute your own pure python libraries in this format, it’s easy:
    pip install wheel
    setup.py based on setuptools:
    python setup.py bdist wheel
    setup.py based on distutils alone:
    python -c "import setuptools; file = ’setup.py’;
    setup code = open( file ).read().replace(’\r\n’, ’\n’);
    exec(compile(setup code, file , ’exec’))" bdist wheel
    (so switch to setuptools)
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13

    View Slide

  26. So how do we use it?
    Specify the version
    python setup.py bdist wheel --python-tag=py2
    python setup.py bdist wheel --python-tag=py33.py34
    python setup.py bdist wheel --universal
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13

    View Slide

  27. So how do we use it?
    Specify the version
    python setup.py bdist wheel --python-tag=py2
    python setup.py bdist wheel --python-tag=py33.py34
    python setup.py bdist wheel --universal
    setup.cfg
    [bdist wheel]
    python-tag=py2
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13

    View Slide

  28. So how do we use it?
    Specify the version
    python setup.py bdist wheel --python-tag=py2
    python setup.py bdist wheel --python-tag=py33.py34
    python setup.py bdist wheel --universal
    setup.cfg
    [bdist wheel]
    python-tag=py2
    setup.cfg
    [bdist wheel]
    universal=1
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13

    View Slide

  29. So how do we use it?
    Specify the version
    python setup.py bdist wheel --python-tag=py2
    python setup.py bdist wheel --python-tag=py33.py34
    python setup.py bdist wheel --universal
    setup.cfg
    [bdist wheel]
    python-tag=py2
    setup.cfg
    [bdist wheel]
    universal=1
    Upload to PyPI
    python setup.py bdist wheel upload
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13

    View Slide

  30. So how do we use it?
    Second
    Precompile all your modules and cache them in a wheelhouse
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 8 / 13

    View Slide

  31. So how do we use it?
    Second
    Precompile all your modules and cache them in a wheelhouse
    Disclaimer
    This solution is based on pip and virtualenv
    Pitfall
    You need a fairly recent of pip/setuptools (requires setuptools ≥ 0.8.0).
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 8 / 13

    View Slide

  32. So how do we use it?
    Create your wheel
    In your construction venv:
    pip install wheel
    pip wheel my module --wheel-dir=my wheelhouse path
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 9 / 13

    View Slide

  33. So how do we use it?
    Create your wheel
    In your construction venv:
    pip install wheel
    pip wheel my module --wheel-dir=my wheelhouse path
    Install your wheel
    In the destination venv:
    pip install my module --find-links=file://wheelhouse path
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 9 / 13

    View Slide

  34. Show me the conf!
    pip.conf
    find-links = file://my wheelhouse path
    wheel-dir = my wheelhouse path
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 10 / 13

    View Slide

  35. Show me the conf!
    pip.conf
    find-links = file://my wheelhouse path
    wheel-dir = my wheelhouse path
    Also define a download cache to prevent useless downloads
    [global]
    find-links = /home/xfernandez/.pip/wheelhouse/
    wheel-dir = /home/xfernandez/.pip/wheelhouse
    download cache = /.pip/cache
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 10 / 13

    View Slide

  36. So how do we use it?
    Create your wheel
    In your construction venv:
    pip install wheel
    pip wheel my module
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 11 / 13

    View Slide

  37. So how do we use it?
    Create your wheel
    In your construction venv:
    pip install wheel
    pip wheel my module
    Install your wheel
    In the destination venv:
    pip install my module
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 11 / 13

    View Slide

  38. The future
    PIP
    Add an option in pip to create a wheel auto-magically at the first
    installation of a module, in order to speed up the next ones
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 12 / 13

    View Slide

  39. The future
    PIP
    Add an option in pip to create a wheel auto-magically at the first
    installation of a module, in order to speed up the next ones
    See also
    Check out http://pip2014.com/ to get a script (untested) that’s supposed
    to configure all this (and apparently some more)
    Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 12 / 13

    View Slide

  40. Thanks!

    View Slide