Harness the speed of the wheel

Paris.py 5 - October 16th 2014

Transcript

1. Harness the speed of the wheel Xavier Fernandez - @xavierfernandez Polyconseil October 16, 2014 Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 1 / 13
2. What is Wheel? Quick overview a package format Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
3. What is Wheel? Quick overview a package format defined in PEP 427 ”The Wheel Binary Package Format 1.0” Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
4. What is Wheel? Quick overview a package format defined in PEP 427 ”The Wheel Binary Package Format 1.0” doc available at http://wheel.rtfd.org/ Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
5. What is Wheel? Quick overview a package format defined in PEP 427 ”The Wheel Binary Package Format 1.0” doc available at http://wheel.rtfd.org/ ZIP-format archive (with a structure close to PEP376) Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
6. What is Wheel? Quick overview a package format defined in PEP 427 ”The Wheel Binary Package Format 1.0” doc available at http://wheel.rtfd.org/ ZIP-format archive (with a structure close to PEP376) versioned Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
7. What is Wheel? Quick overview a package format defined in PEP 427 ”The Wheel Binary Package Format 1.0” doc available at http://wheel.rtfd.org/ ZIP-format archive (with a structure close to PEP376) versioned with a specially formatted file name and the .whl extension Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 2 / 13
8. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Examples Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
9. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Used by the installer to select the ’best’ file Examples Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
10. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Used by the installer to select the ’best’ file Examples Django-1.7-py2.py3-none-any.whl Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
11. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Used by the installer to select the ’best’ file Examples Django-1.7-py2.py3-none-any.whl aioamqp-0.1.1-py33.py34-none-any.whl Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
12. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Used by the installer to select the ’best’ file Examples Django-1.7-py2.py3-none-any.whl aioamqp-0.1.1-py33.py34-none-any.whl lxml-3.4.0-cp26-none-win32.whl Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
13. What is Wheel? Formatted name {distribution}-{version}(-{build tag})?-{python tag}-{abi tag}-{platform tag}.whl Used by the installer to select the ’best’ file Examples Django-1.7-py2.py3-none-any.whl aioamqp-0.1.1-py33.py34-none-any.whl lxml-3.4.0-cp26-none-win32.whl Pillow-2.6.1-cp34-cp34m- macosx 10 6 intel.macosx 10 9 intel.macosx 10 9 x86 64.whl Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 3 / 13
14. Fast Installation no setup.py execution Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
15. Fast Installation no setup.py execution basically just an unzip Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
16. Fast Installation no setup.py execution basically just an unzip (... with minimum magic like for instance shebang rewrite for scripts) Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
17. Fast Installation no setup.py execution basically just an unzip (... with minimum magic like for instance shebang rewrite for scripts) Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
18. Fast Installation no setup.py execution basically just an unzip (... with minimum magic like for instance shebang rewrite for scripts) Experimental protocol pip install –no-index –upgrade –force-reinstall Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
19. Fast Installation no setup.py execution basically just an unzip (... with minimum magic like for instance shebang rewrite for scripts) Experimental protocol pip install –no-index –upgrade –force-reinstall with –find-links pointing to either wheels or tar.gz Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
20. Fast Installation no setup.py execution basically just an unzip (... with minimum magic like for instance shebang rewrite for scripts) Experimental protocol pip install –no-index –upgrade –force-reinstall with –find-links pointing to either wheels or tar.gz repeated between 4 and 10 times Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 4 / 13
21. Benchmark (re)Installation times of pure python modules Module .tar.gz .whl Ratio pep8-1.5.0 0.65 sec 0.27 sec x 2.4 flake8-2.1.0 1.84 sec 0.37 sec x 5.0 mccabe-0.2.1 0.61 sec 0.25 sec x 2.4 pyflakes-0.8 0.67 sec 0.29 sec x 2.3 Django-1.6.4 10.6 sec 6.76 sec x 1.6 Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 5 / 13
22. Benchmark (re)Installation times of pure python modules Module .tar.gz .whl Ratio pep8-1.5.0 0.65 sec 0.27 sec x 2.4 flake8-2.1.0 1.84 sec 0.37 sec x 5.0 mccabe-0.2.1 0.61 sec 0.25 sec x 2.4 pyflakes-0.8 0.67 sec 0.29 sec x 2.3 Django-1.6.4 10.6 sec 6.76 sec x 1.6 (re)Installation times of python modules needing some compilation Module .tar.gz .whl Ratio lxml-3.3.5 60 sec 0.42 sec x 143 numpy-1.8.1 208 sec 2.0 sec x 104 Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 5 / 13
23. So how do we use it? First Distribute your own pure python libraries in this format, it’s easy: pip install wheel setup.py based on setuptools: python setup.py bdist wheel Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13
24. So how do we use it? First Distribute your own pure python libraries in this format, it’s easy: pip install wheel setup.py based on setuptools: python setup.py bdist wheel setup.py based on distutils alone: python -c "import setuptools; file = ’setup.py’; setup code = open( file ).read().replace(’\r\n’, ’\n’); exec(compile(setup code, file , ’exec’))" bdist wheel Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13
25. So how do we use it? First Distribute your own pure python libraries in this format, it’s easy: pip install wheel setup.py based on setuptools: python setup.py bdist wheel setup.py based on distutils alone: python -c "import setuptools; file = ’setup.py’; setup code = open( file ).read().replace(’\r\n’, ’\n’); exec(compile(setup code, file , ’exec’))" bdist wheel (so switch to setuptools) Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 6 / 13
26. So how do we use it? Specify the version python setup.py bdist wheel --python-tag=py2 python setup.py bdist wheel --python-tag=py33.py34 python setup.py bdist wheel --universal Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13
27. So how do we use it? Specify the version python setup.py bdist wheel --python-tag=py2 python setup.py bdist wheel --python-tag=py33.py34 python setup.py bdist wheel --universal setup.cfg [bdist wheel] python-tag=py2 Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13
28. So how do we use it? Specify the version python setup.py bdist wheel --python-tag=py2 python setup.py bdist wheel --python-tag=py33.py34 python setup.py bdist wheel --universal setup.cfg [bdist wheel] python-tag=py2 setup.cfg [bdist wheel] universal=1 Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13
29. So how do we use it? Specify the version python setup.py bdist wheel --python-tag=py2 python setup.py bdist wheel --python-tag=py33.py34 python setup.py bdist wheel --universal setup.cfg [bdist wheel] python-tag=py2 setup.cfg [bdist wheel] universal=1 Upload to PyPI python setup.py bdist wheel upload Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 7 / 13
30. So how do we use it? Second Precompile all your modules and cache them in a wheelhouse Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 8 / 13
31. So how do we use it? Second Precompile all your modules and cache them in a wheelhouse Disclaimer This solution is based on pip and virtualenv Pitfall You need a fairly recent of pip/setuptools (requires setuptools ≥ 0.8.0). Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 8 / 13
32. So how do we use it? Create your wheel In your construction venv: pip install wheel pip wheel my module --wheel-dir=my wheelhouse path Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 9 / 13
33. So how do we use it? Create your wheel In your construction venv: pip install wheel pip wheel my module --wheel-dir=my wheelhouse path Install your wheel In the destination venv: pip install my module --find-links=file://wheelhouse path Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 9 / 13
34. Show me the conf! pip.conf find-links = file://my wheelhouse path wheel-dir = my wheelhouse path Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 10 / 13
35. Show me the conf! pip.conf find-links = file://my wheelhouse path wheel-dir = my wheelhouse path Also define a download cache to prevent useless downloads [global] find-links = /home/xfernandez/.pip/wheelhouse/ wheel-dir = /home/xfernandez/.pip/wheelhouse download cache = /.pip/cache Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 10 / 13
36. So how do we use it? Create your wheel In your construction venv: pip install wheel pip wheel my module Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 11 / 13
37. So how do we use it? Create your wheel In your construction venv: pip install wheel pip wheel my module Install your wheel In the destination venv: pip install my module Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 11 / 13
38. The future PIP Add an option in pip to create a wheel auto-magically at the first installation of a module, in order to speed up the next ones Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 12 / 13
39. The future PIP Add an option in pip to create a wheel auto-magically at the first installation of a module, in order to speed up the next ones See also Check out http://pip2014.com/ to get a script (untested) that’s supposed to configure all this (and apparently some more) Xavier Fernandez (Polyconseil) Harness the speed of the wheel October 16, 2014 12 / 13
40. Thanks!