Python Packaging Hong Minhee 

Speaker: Hong Minhee ● Wand, sphinxcontrib-httpdomain, libsass, Sider, Okydoky ● Web developer ● Python programmer since 2005 ● CDNetworks, StyleShare, Crosspop ● http://dahlia.kr/ ● irc://irc.ozinger.org/hongminhee 

No content

Basic concepts 

sys.path $ echo 'bar = "ok"' > /tmp/foo.py $ python Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path.append('/tmp') >>> import foo >>> foo.bar 'ok' 

site-packages (dist-packages in Debian/Ubuntu) Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path ['', '/usr/lib/python2.7', ..., '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL'...] 

site-packages (dist-packages in Debian/Ubuntu) $ ls -p /usr/local/lib/python2.7/dist-packages/ easy-install.pth Sphinx-1.1.3-py2.7.egg rsa/ rsa-3.0.1.egg-info tox-1.4.2-py2.7.egg chardet/ virtualenv-1.7.2-py2.7.egg chardet-1.0.1.egg-info docutils-0.9.1-py2.7.egg 

*.pth $ cat /.../site-packages/easy-install.pth import sys; sys.__plen = len(sys.path) ./Sphinx-1.1.3-py2.7.egg ./docutils-0.9.1-py2.7.egg ./tox-1.4.2-py2.7.egg ./virtualenv-1.7.2-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys. path[sys.__plen:]; p=getattr(sys,'__egginsert', 0); sys.path[p:p]=new; sys.__egginsert = p+len (new) 

History of Python packaging 

The way the most of us have packaged $ cat requirements.txt Flask==0.9 SQLAlchemy==0.8.0 Wand==0.2.3 $ pip install -r requirements.txt 

It is not packaging ● No versioning ● You can’t upload this on PyPI (Others cannot install it using pip or easy_install) ● Has no package name ● Doesn’t work at all if it contains any C extensions ○ How is it built? ○ No binary distributions ● No metadata (Who made it? What does it for me?) ● requirements.txt is not packaging but using packages 

The distutils–packaging dynasty ● distutils ● setuptools (easy_install) ● Distribute ● pip ● Distutils2 → packaging 

The distutils–packaging dynasty ● distutils: zip, tgz, tar.bz2 ({s,b}dist) ● Platform specific: rpm, deb, msi ● setuptools: egg (bdist) ● pip: pybundle (sdist), wheel (bdist) 

In the beginning, there was distutils $ tree . ├── foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 2 directories, 6 files 

In the beginning, there was distutils from distutils.core import setup setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=['foo', 'foo.bar']) 

In the beginning, there was distutils $ python setup.py sdist running sdist running check writing manifest file 'MANIFEST' creating Foo-1.0 ... making hard links in Foo-1.0... hard linking setup.py -> Foo-1.0 hard linking foo/__init__.py -> Foo-1.0/foo ... creating dist Creating tar archive removing 'Foo-1.0' (and everything under it) 

In the beginning, there was distutils $ tree . ├── MANIFEST ├── dist │ └── Foo-1.0.tar.gz ├── foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 3 directories, 8 files 

In the beginning, there was distutils $ tar xvfz Foo-1.0.tar.gz ... $ cd Foo-1.0 $ python setup.py install running install running build ... running install_lib creating /.../python2.7/site-packages/foo copying build/lib/foo/__init__.py -> /.../python2.7/site- packages/foo creating /.../python2.7/site-packages/foo/bar ... copying build/lib/foo/exc.py -> /.../python2.7/site-packages/foo copying build/lib/foo/util.py -> /.../python2.7/site-packages/foo 

In the beginning, there was distutils $ tree lib/python2.7/site-packages/ lib/python2.7/site-packages/ └── foo ├── __init__.py ├── bar │ ├── __init__.py │ └── mod.py ├── exc.py └── util.py 2 directories, 5 files 

In the beginning, there was distutils Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import foo.bar.mod >>> foo.bar.mod.__doc__ 'Useful modules.' 

setuptools: De facto standard ● Built on top of distutils ● easy_install ● Dependency resolution ● .egg binary distribution ● Resource management ● Dynamic discovery of plugins 

setuptools: Built on top of distutils from distutils.core import setup setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=['foo', 'foo.bar']) 

setuptools: Built on top of distutils from setuptools import setup, find_packages setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=find_packages(), install_requires=['Flask', 'Wand>=0. 2.1']) 

setuptools: Dependency resolution $ python setup.py install running install running bdist_egg running egg_info creating Foo.egg-info writing requirements to Foo.egg-info/requires.txt writing Foo.egg-info/PKG-INFO writing top-level names to Foo.egg-info/top_level.txt writing dependency_links to Foo.egg-info/dependency_links. txt writing manifest file 'Foo.egg-info/SOURCES.txt' ... 

setuptools: Dependency resolution Installed /.../python2.7/site-packages/Foo-1.0-py2.7.egg Processing dependencies for Foo==1.0 Searching for Wand>=0.2.1 Reading https://pypi.crate.io/simple/Wand/ Best match: Wand 0.2.3 Downloading ... Processing Wand-0.2.3.tar.gz ... Adding Wand 0.2.3 to easy-install.pth file Installed /.../python2.7/site-packages/Wand-0.2.3-py2.7. egg 

setuptools: Dependency resolution $ ls -p lib/python2.7/site-packages/ Flask-0.9-py2.7.egg/ easy-install.pth Foo-1.0-py2.7.egg pip-1.2.1-py2.7.egg/ Jinja2-2.6-py2.7.egg/ setuptools-0.6c11-py2.7.egg Wand-0.2.3-py2.7.egg setuptools.pth Werkzeug-0.8.3-py2.7.egg/ 

setuptools: Dependency resolution $ cat lib/python2.7/site-packages/easy-install.pth import sys; sys.__plen = len(sys.path) ./setuptools-0.6c11-py2.7.egg ./pip-1.2.1-py2.7.egg ./Foo-1.0-py2.7.egg ./Wand-0.2.3-py2.7.egg ./Flask-0.9-py2.7.egg ./Jinja2-2.6-py2.7.egg ./Werkzeug-0.8.3-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys. __plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p] =new; sys.__egginsert = p+len(new) 

setuptools: Submit to PyPI $ python setup.py register ... $ python setup.py sdist upload ... 

$ easy_install Foo ... $ easy_install "Foo >= 1.0" ... $ easy_install "Foo == 1.0" ... easy_install does install 

C:\> easy_install libsass Searching for libsass Reading http://pypi.python.org/simple/libsass/ Best match: libsass 0.2.4 Downloading http://.../libsass-0.2.4-py2.7-win-amd64.egg Processing libsass-0.2.4-py2.7-win-amd64.egg Moving libsass-0.2.4-py2.7-win-amd64.egg to C:\...\site-packages Adding libsass 0.2.4 to easy-install.pth file Installing sassc.py script to C:\...\Scripts Deleting C:\...\Scripts\sassc.py Installing sassc-script.py script to C:\...\Scripts Installing sassc.exe script to C:\...\Scripts Installed C:\...\site-packages\libsass-0.2.4-py2.7-win-amd64.egg Processing dependencies for libsass Finished processing dependencies for libsass egg: Binary distribution 

pip: Successor of easy_install ● Replaces easy_install (not setuptools) ● Can uninstall packages ● Install packages w/ atomic transaction ○ Install everything ○ or nothing ● Can freeze current site-packages ○ Widely used pattern to generate requirements. txt ● Very useful install -r option ● No easy-install.pth or such 

$ easy_install Flask Searching for Flask Reading https://pypi.crate.io/simple/Flask/ Best match: Flask 0.9 Downloading ... Processing Flask-0.9.tar.gz ... Adding Flask 0.9 to easy-install.pth file Installed /.../lib/python2.7/site-packages/Flask-0.9-py2.7.egg Processing dependencies for Flask Searching for Jinja2>=2.4 Reading https://pypi.crate.io/simple/Jinja2/ Best match: Jinja2 2.6 Downloading ... ^Cinterrupted pip: Atomic installation 

$ ls lib/python2.7/site-packages/ Flask-0.9-py2.7.egg pip- 1.2.1-py2.7.egg distribute-0.6.28-py2.7.egg setuptools.pth easy-install.pth pip: Atomic installation 

pip: Atomic installation $ pip install Flask Downloading/unpacking Flask Downloading Flask-0.9.tar.gz (481kB): 481kB downloaded Running setup.py egg_info for package Flask Downloading/unpacking Werkzeug>=0.7 (from Flask) Downloading Werkzeug-0.8.3.tar.gz (1.1MB): 1.1MB downloaded Running setup.py egg_info for package Werkzeug Downloading/unpacking Jinja2>=2.4 (from Flask) Downloading Jinja2-2.6.tar.gz (389kB): 208kB downloaded Operation cancelled by user Storing complete log in /Users/dahlia/.pip/pip.log 

pip: Atomic installation $ ls lib/python2.7/site-packages/ distribute-0.6.28-py2.7.egg pip- 1.2.1-py2.7.egg easy-install.pth setuptools.pth 

pip: Downsides? ● Can’t install egg packages ● Hard to use pip on Windows ● Or you need entire C build toolchain to install ordinary Python packages at least ○ Visual Studio ○ cygwin? mingw? ● Discourage to declare package metadata using setup.py 

Distribute: Maintained fork of setuptools ● setuptools hasn’t been maintained since 2006 ● setuptools doesn’t provide recent Python versions ● Distribute is a fork of setuptools ● Hence it provides the same namespace to setuptools >>> import setuptools >>> setuptools 

Future of Python packaging 

packaging ● Also known as Distutils2 ● To be merged to Python standard library ● Declarative packaging via setup.cfg ● Eliminate imperative packaging (setup.py) ● You can’t use it for your package right now ● Though it would be possible soon :-) 

setup.py can do everything! ● Ordinary Python script ● That means it’s Turing-complete ● It gives powerful flexibility ○ Sync long_description field with README file ○ Conditional dependencies using sys.platform and if keyword ○ Platform-aware building process by extending distutils 

setup.py can do everything! import sys from setuptools import setup, find_packages install_requires = ['Flask', 'Wand>=0.2.1'] if sys.platform == 'win32': install_requires.append('pywin32') setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='[email protected]', url='http://example.com/', packages=find_packages(), install_requires=install_requires) 

setup.py knows nothing… ● You can even implement a PHP-to-Python compiler in the script and generate Python files just-in-time ● Nothing can be assumed with setup.py ● Cannot be indexed ● You need to execute the script to get metadata ● Exactly, one of possible metadata variants ● The root cause of why installing Python packages is so slow 

setup.py knows nothing… from random import randrange from distutils.core import setup random_value = str(randrange(1000)) module_name = 'random_module_name_' + random_value with open(module_name + '.py', 'w') as pyfile: print >> pyfile, 'value = ' + random_value setup(name='Crazy', description='You cannot index this', py_modules=[module_name]) 

setup.py knows nothing… $ python setup.py sdist running sdist running check writing manifest file 'MANIFEST' creating Crazy-0.0.0 making hard links in Crazy-0.0.0... hard linking random_module_name_832.py -> Crazy-0.0.0 hard linking setup.py -> Crazy-0.0.0 creating dist Creating tar archive removing 'Crazy-0.0.0' (and everything under it) 

Environment markers (PEP 345) Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform 

TL;DR: Package management ● Install distribute (instead of setuptools) first ● easy_install pip virtualenv ● Use pip to install and uninstall ● Use easy_install to install C extensions ● Isolate site-packages per project using virtualenv 

TL;DR: Distributing packages ● Write setup.py script ● Support both distribute and setuptools (for Python 2 packages) ● Support distribute (for Python 3 packages) ● Submit to PyPI using python setup.py register && python setup.py sdist upload command ● Provide binary egg packages for C extensions 

TL;DR: The future of Python ecosystem ● Ready for packaging a.k.a. Distutils2 ● Test your package on various Python VMs/versions (Python 2 and 3, CPython, PyPy…) ● tox will help ● Prefer pip over easy_install ● Don’t make your setup.py dynamic ● It will be your future headache 

j.mp/pykr2013-packaging 

Any questions?