Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

Python Packaging

Python Packaging

Summarized talk about history of Python packaging ecosystem and the future of it.

Hong Minhee (洪 民憙)

March 17, 2013
Tweet

More Decks by Hong Minhee (洪 民憙)

Other Decks in Programming

Transcript

  1. Speaker: Hong Minhee • Wand, sphinxcontrib-httpdomain, libsass, Sider, Okydoky •

    Web developer • Python programmer since 2005 • CDNetworks, StyleShare, Crosspop • http://dahlia.kr/ • irc://irc.ozinger.org/hongminhee
  2. sys.path $ echo 'bar = "ok"' > /tmp/foo.py $ python

    Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path.append('/tmp') >>> import foo >>> foo.bar 'ok'
  3. site-packages (dist-packages in Debian/Ubuntu) Python 2.7.2+ (default, Jul 20 2012...)

    [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path ['', '/usr/lib/python2.7', ..., '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL'...]
  4. site-packages (dist-packages in Debian/Ubuntu) $ ls -p /usr/local/lib/python2.7/dist-packages/ easy-install.pth Sphinx-1.1.3-py2.7.egg

    rsa/ rsa-3.0.1.egg-info tox-1.4.2-py2.7.egg chardet/ virtualenv-1.7.2-py2.7.egg chardet-1.0.1.egg-info docutils-0.9.1-py2.7.egg
  5. *.pth $ cat /.../site-packages/easy-install.pth import sys; sys.__plen = len(sys.path) ./Sphinx-1.1.3-py2.7.egg

    ./docutils-0.9.1-py2.7.egg ./tox-1.4.2-py2.7.egg ./virtualenv-1.7.2-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys. path[sys.__plen:]; p=getattr(sys,'__egginsert', 0); sys.path[p:p]=new; sys.__egginsert = p+len (new)
  6. The way the most of us have packaged $ cat

    requirements.txt Flask==0.9 SQLAlchemy==0.8.0 Wand==0.2.3 $ pip install -r requirements.txt
  7. It is not packaging • No versioning • You can’t

    upload this on PyPI (Others cannot install it using pip or easy_install) • Has no package name • Doesn’t work at all if it contains any C extensions ◦ How is it built? ◦ No binary distributions • No metadata (Who made it? What does it for me?) • requirements.txt is not packaging but using packages
  8. The distutils–packaging dynasty • distutils: zip, tgz, tar.bz2 ({s,b}dist) •

    Platform specific: rpm, deb, msi • setuptools: egg (bdist) • pip: pybundle (sdist), wheel (bdist)
  9. In the beginning, there was distutils $ tree . ├──

    foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 2 directories, 6 files
  10. In the beginning, there was distutils from distutils.core import setup

    setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=['foo', 'foo.bar'])
  11. In the beginning, there was distutils $ python setup.py sdist

    running sdist running check writing manifest file 'MANIFEST' creating Foo-1.0 ... making hard links in Foo-1.0... hard linking setup.py -> Foo-1.0 hard linking foo/__init__.py -> Foo-1.0/foo ... creating dist Creating tar archive removing 'Foo-1.0' (and everything under it)
  12. In the beginning, there was distutils $ tree . ├──

    MANIFEST ├── dist │ └── Foo-1.0.tar.gz ├── foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 3 directories, 8 files
  13. In the beginning, there was distutils $ tar xvfz Foo-1.0.tar.gz

    ... $ cd Foo-1.0 $ python setup.py install running install running build ... running install_lib creating /.../python2.7/site-packages/foo copying build/lib/foo/__init__.py -> /.../python2.7/site- packages/foo creating /.../python2.7/site-packages/foo/bar ... copying build/lib/foo/exc.py -> /.../python2.7/site-packages/foo copying build/lib/foo/util.py -> /.../python2.7/site-packages/foo
  14. In the beginning, there was distutils $ tree lib/python2.7/site-packages/ lib/python2.7/site-packages/

    └── foo ├── __init__.py ├── bar │ ├── __init__.py │ └── mod.py ├── exc.py └── util.py 2 directories, 5 files
  15. In the beginning, there was distutils Python 2.7.2+ (default, Jul

    20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import foo.bar.mod >>> foo.bar.mod.__doc__ 'Useful modules.'
  16. setuptools: De facto standard • Built on top of distutils

    • easy_install • Dependency resolution • .egg binary distribution • Resource management • Dynamic discovery of plugins
  17. setuptools: Built on top of distutils from distutils.core import setup

    setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=['foo', 'foo.bar'])
  18. setuptools: Built on top of distutils from setuptools import setup,

    find_packages setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=find_packages(), install_requires=['Flask', 'Wand>=0. 2.1'])
  19. setuptools: Dependency resolution $ python setup.py install running install running

    bdist_egg running egg_info creating Foo.egg-info writing requirements to Foo.egg-info/requires.txt writing Foo.egg-info/PKG-INFO writing top-level names to Foo.egg-info/top_level.txt writing dependency_links to Foo.egg-info/dependency_links. txt writing manifest file 'Foo.egg-info/SOURCES.txt' ...
  20. setuptools: Dependency resolution Installed /.../python2.7/site-packages/Foo-1.0-py2.7.egg Processing dependencies for Foo==1.0 Searching

    for Wand>=0.2.1 Reading https://pypi.crate.io/simple/Wand/ Best match: Wand 0.2.3 Downloading ... Processing Wand-0.2.3.tar.gz ... Adding Wand 0.2.3 to easy-install.pth file Installed /.../python2.7/site-packages/Wand-0.2.3-py2.7. egg
  21. setuptools: Dependency resolution $ ls -p lib/python2.7/site-packages/ Flask-0.9-py2.7.egg/ easy-install.pth Foo-1.0-py2.7.egg

    pip-1.2.1-py2.7.egg/ Jinja2-2.6-py2.7.egg/ setuptools-0.6c11-py2.7.egg Wand-0.2.3-py2.7.egg setuptools.pth Werkzeug-0.8.3-py2.7.egg/
  22. setuptools: Dependency resolution $ cat lib/python2.7/site-packages/easy-install.pth import sys; sys.__plen =

    len(sys.path) ./setuptools-0.6c11-py2.7.egg ./pip-1.2.1-py2.7.egg ./Foo-1.0-py2.7.egg ./Wand-0.2.3-py2.7.egg ./Flask-0.9-py2.7.egg ./Jinja2-2.6-py2.7.egg ./Werkzeug-0.8.3-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys. __plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p] =new; sys.__egginsert = p+len(new)
  23. $ easy_install Foo ... $ easy_install "Foo >= 1.0" ...

    $ easy_install "Foo == 1.0" ... easy_install does install
  24. C:\> easy_install libsass Searching for libsass Reading http://pypi.python.org/simple/libsass/ Best match:

    libsass 0.2.4 Downloading http://.../libsass-0.2.4-py2.7-win-amd64.egg Processing libsass-0.2.4-py2.7-win-amd64.egg Moving libsass-0.2.4-py2.7-win-amd64.egg to C:\...\site-packages Adding libsass 0.2.4 to easy-install.pth file Installing sassc.py script to C:\...\Scripts Deleting C:\...\Scripts\sassc.py Installing sassc-script.py script to C:\...\Scripts Installing sassc.exe script to C:\...\Scripts Installed C:\...\site-packages\libsass-0.2.4-py2.7-win-amd64.egg Processing dependencies for libsass Finished processing dependencies for libsass egg: Binary distribution
  25. pip: Successor of easy_install • Replaces easy_install (not setuptools) •

    Can uninstall packages • Install packages w/ atomic transaction ◦ Install everything ◦ or nothing • Can freeze current site-packages ◦ Widely used pattern to generate requirements. txt • Very useful install -r option • No easy-install.pth or such
  26. $ easy_install Flask Searching for Flask Reading https://pypi.crate.io/simple/Flask/ Best match:

    Flask 0.9 Downloading ... Processing Flask-0.9.tar.gz ... Adding Flask 0.9 to easy-install.pth file Installed /.../lib/python2.7/site-packages/Flask-0.9-py2.7.egg Processing dependencies for Flask Searching for Jinja2>=2.4 Reading https://pypi.crate.io/simple/Jinja2/ Best match: Jinja2 2.6 Downloading ... ^Cinterrupted pip: Atomic installation
  27. pip: Atomic installation $ pip install Flask Downloading/unpacking Flask Downloading

    Flask-0.9.tar.gz (481kB): 481kB downloaded Running setup.py egg_info for package Flask Downloading/unpacking Werkzeug>=0.7 (from Flask) Downloading Werkzeug-0.8.3.tar.gz (1.1MB): 1.1MB downloaded Running setup.py egg_info for package Werkzeug Downloading/unpacking Jinja2>=2.4 (from Flask) Downloading Jinja2-2.6.tar.gz (389kB): 208kB downloaded Operation cancelled by user Storing complete log in /Users/dahlia/.pip/pip.log
  28. pip: Downsides? • Can’t install egg packages • Hard to

    use pip on Windows • Or you need entire C build toolchain to install ordinary Python packages at least ◦ Visual Studio ◦ cygwin? mingw? • Discourage to declare package metadata using setup.py
  29. Distribute: Maintained fork of setuptools • setuptools hasn’t been maintained

    since 2006 • setuptools doesn’t provide recent Python versions • Distribute is a fork of setuptools • Hence it provides the same namespace to setuptools >>> import setuptools >>> setuptools <module 'setuptools' from /.../site-packages/distribute- 0.6.30-py2.7.egg/setuptools/__init__.pyc'>
  30. packaging • Also known as Distutils2 • To be merged

    to Python standard library • Declarative packaging via setup.cfg • Eliminate imperative packaging (setup.py) • You can’t use it for your package right now • Though it would be possible soon :-)
  31. setup.py can do everything! • Ordinary Python script • That

    means it’s Turing-complete • It gives powerful flexibility ◦ Sync long_description field with README file ◦ Conditional dependencies using sys.platform and if keyword ◦ Platform-aware building process by extending distutils
  32. setup.py can do everything! import sys from setuptools import setup,

    find_packages install_requires = ['Flask', 'Wand>=0.2.1'] if sys.platform == 'win32': install_requires.append('pywin32') setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=find_packages(), install_requires=install_requires)
  33. setup.py knows nothing… • You can even implement a PHP-to-Python

    compiler in the script and generate Python files just-in-time • Nothing can be assumed with setup.py • Cannot be indexed • You need to execute the script to get metadata • Exactly, one of possible metadata variants • The root cause of why installing Python packages is so slow
  34. setup.py knows nothing… from random import randrange from distutils.core import

    setup random_value = str(randrange(1000)) module_name = 'random_module_name_' + random_value with open(module_name + '.py', 'w') as pyfile: print >> pyfile, 'value = ' + random_value setup(name='Crazy', description='You cannot index this', py_modules=[module_name])
  35. setup.py knows nothing… $ python setup.py sdist running sdist running

    check writing manifest file 'MANIFEST' creating Crazy-0.0.0 making hard links in Crazy-0.0.0... hard linking random_module_name_832.py -> Crazy-0.0.0 hard linking setup.py -> Crazy-0.0.0 creating dist Creating tar archive removing 'Crazy-0.0.0' (and everything under it)
  36. Environment markers (PEP 345) Requires-Dist: pywin32 (>1.0); sys.platform == 'win32'

    Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform
  37. TL;DR: Package management • Install distribute (instead of setuptools) first

    • easy_install pip virtualenv • Use pip to install and uninstall • Use easy_install to install C extensions • Isolate site-packages per project using virtualenv
  38. TL;DR: Distributing packages • Write setup.py script • Support both

    distribute and setuptools (for Python 2 packages) • Support distribute (for Python 3 packages) • Submit to PyPI using python setup.py register && python setup.py sdist upload command • Provide binary egg packages for C extensions
  39. TL;DR: The future of Python ecosystem • Ready for packaging

    a.k.a. Distutils2 • Test your package on various Python VMs/versions (Python 2 and 3, CPython, PyPy…) • tox will help • Prefer pip over easy_install • Don’t make your setup.py dynamic • It will be your future headache