Bento presentation for Pycon APAC 2012

Slides I presented in June 2012 at Singapore for Pycon APAC for bento, a pythonic packaging software.

Present the main features of bento

Transcript

1. ห౰(BENTO) A Python solution for packaging Saturday, 9 June 12
2. WHO AM I ? • Started to be involved with numpy/scipy around 2006/2007 • Particularly involved with build/installer issues • Why do I care about packaging ? Saturday, 9 June 12
3. WHAT IS PACKAGING ? • Going from sources to the user • Different users require different workflows: not just tarballs ! • allow for creating binary installers • integrating with native systems (.deb, .rpm, etc...) Saturday, 9 June 12
4. DISTUTILS • De-facto solution since ±2000 • Works well for simple packages • Setuptools/distribute: extensions on top of distutils • main feature: package dependency (pip, easy_install) Saturday, 9 June 12
5. DISTUTILS ISSUES • Non-trivial customisation is hard: • new compiler support • creating new commands (test, build doc, etc...) • Anything related to compilation is hard and fragile • Extensions based on monkey- patching Saturday, 9 June 12
6. DISTUTILS ISSUES (2) • Numpy/Scipy distutils extensions ± 10 KLOC (fortran, etc...) • Numerous issues when setuptools appeared (monkey patching overrides, etc...) • Poor code quality, e.g.: • attributes added at runtime in some conditions • lack of common interfaces between compiler classes • Numscons: tentative to integrate scons and distutils Saturday, 9 June 12
7. BENTO • Learning from the numscons experience • Written as a library from the ground up: • low coupling of commands • no global state • Scale down and up: simpler for simple packages, more flexible for bigger ones Saturday, 9 June 12
8. BASICS Saturday, 9 June 12
9. A SIMPLE BENTO EXAMPLE Name: foo Version: 0.1 Description: Package description. Library: Packages: foo, foo.core Saturday, 9 June 12
10. BENTOMAKER $ bentomaker configure $ bentomaker build $ bentomaker install # Or more simply (automatically # runs configure and build) $ bentomaker install Saturday, 9 June 12
11. ADDING DATA-FILES Datafiles: pdf_documentation TargetDir: $pdfdir Files: main.pdf $ bentomaker configure --pdfdir=/usr/pdfdoc # ant-like glob format supported Datafiles: pdf_documentation TargetDir: $pdfdoc Files: doc/**/*.pdf Saturday, 9 June 12
12. CUSTOM INSTALL PATHS Path: foodir Default: $datadir/foo Description: foo directory Datafiles: fubar-doc TargetDir: $foodir Files: main.foo $ bentomaker configure --foodir=/usr/foo Saturday, 9 June 12
13. FITTING INTO THE EXISTING PYTHON ECOSYSTEM Saturday, 9 June 12
14. FROM DISTUTILS TO BENTO # convert setup.py to bento.info $ bentomaker convert • Works for packages using distutils, setuptools, etc... • bentomaker may be distributed as a single file to avoid extra dependencies Saturday, 9 June 12
15. PIP SUPPORT Simple setup.py to “look” like a distutils package: # setup.py import setuptools import bento.distutils bento.distutils.monkey_patch() # Grab metadata from bento.info setuptools.setup() Those 4 lines of code allow the package to be pip- installable Saturday, 9 June 12
16. ADVANCED USAGE Saturday, 9 June 12
17. HOOK FILES (1) • Tweak or extend the packaging process with code, e.g. new command $ bentomaker hello hello from bento.commands import hooks @hooks.command def hello(context): print "Hello" • Hooks are pure python without any ‘magic’ mechanism Saturday, 9 June 12
18. HOOKS FILES (2) from bento.commands import hooks @hooks.command def hello(context): print "hello nice world" @hooks.command def goodbye(context): print "goodbye cruel world !" @hooks.startup def startup(context): context.set_before("goodbye", "hello") Dependencies defined outside classes for lower coupling between commands Saturday, 9 June 12
19. HOOK FILES (3) • Bento gives multiple “hooks” (pre/post/etc...) to customise things: • customise compilation flags, conditionally do things, etc... • bento written to be highly customised (integration with 3rd party tools to build C extensions, etc...) Saturday, 9 June 12
20. PACKAGE METADATA (1) # bento.info content Version: 0.2 Author: John Doe MetaTemplateFile: foo/__info.py.in # foo/__info.py.in template content version = $VERSION author = $AUTHOR Allow for a package to know its own metadata: Saturday, 9 June 12
21. PACKAGE METADATA (2) New metadata can be defined in the hook file: # foo/__info.py.in template content version = $VERSION git_revision = $GIT_REVISION author = $AUTHOR @hooks.pre_build def pre_build(context): context.register_metadata("git_revision", compute_git_revision()) Saturday, 9 June 12
22. BUILD BACKENDS • Bento has the notion of “backends” to heavily customise some aspects of the build process • currently 3 pluggable backends (default, waf and ‘distutils’ mode) • code for a backend is relatively small (few 100 LOC) • Example Saturday, 9 June 12
23. OUT-OF-TREE BUILDS Out-of-tree builds is building from a directory outside the source tree source_tree$ ls bento.info foo/__init__.py ... build_tree$ bentomaker --bento-info=.../bento.info Example Saturday, 9 June 12
24. MORE • Ability to add options to commands • additional configure fine tuning (e.g. 3rd library location) • all options available from any command • Simple mechanism to retrieve resources (no more pkg_resources non-sense) • More to come Saturday, 9 June 12
25. ADVANCED EXAMPLES: NUMPY • Numpy: initial rationale for bento • build scripts LOC reduced by 50 % • parallel and reliable partial build support • bento’s LOC ± numpy.distutils extensions ! Saturday, 9 June 12
26. INTERNALS Saturday, 9 June 12
27. DECOUPLED ARCHITECTURE • bento written as a library from the ground up • no global variable or implicitly passed state • commands are not coupled with each other: • dependencies declared outside the commands • install and binary installers driven by a build manifest (json file) • options can be added outside commands Saturday, 9 June 12
28. IDEMPOTENCY • idempotency: running the same command twice should give the same result • explicit list of installed files (no stalled files) • support for out-of-tree builds Saturday, 9 June 12
29. WHAT’S NEXT • A few missing features to replace distutils (msi support, etc...) • Improve parser (faster and more reliable) • bentoya (aka bentoshop): pypi-like service + local db of installed packages: • uninstall support • apt-get-like features Saturday, 9 June 12
30. WHERE TO GET IT • On github: http://github.com/cournape/Bento.git • Documentation: http://cournape.github.com/Bento • Mailing List: bento@librelist.com Saturday, 9 June 12