Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bento presentation for Pycon APAC 2012

Bento presentation for Pycon APAC 2012

Slides I presented in June 2012 at Singapore for Pycon APAC for bento, a pythonic packaging software.

Present the main features of bento


June 09, 2012

More Decks by cournape

Other Decks in Programming


  1. WHO AM I ? • Started to be involved with

    numpy/scipy around 2006/2007 • Particularly involved with build/installer issues • Why do I care about packaging ? Saturday, 9 June 12
  2. WHAT IS PACKAGING ? • Going from sources to the

    user • Different users require different workflows: not just tarballs ! • allow for creating binary installers • integrating with native systems (.deb, .rpm, etc...) Saturday, 9 June 12
  3. DISTUTILS • De-facto solution since ±2000 • Works well for

    simple packages • Setuptools/distribute: extensions on top of distutils • main feature: package dependency (pip, easy_install) Saturday, 9 June 12
  4. DISTUTILS ISSUES • Non-trivial customisation is hard: • new compiler

    support • creating new commands (test, build doc, etc...) • Anything related to compilation is hard and fragile • Extensions based on monkey- patching Saturday, 9 June 12
  5. DISTUTILS ISSUES (2) • Numpy/Scipy distutils extensions ± 10 KLOC

    (fortran, etc...) • Numerous issues when setuptools appeared (monkey patching overrides, etc...) • Poor code quality, e.g.: • attributes added at runtime in some conditions • lack of common interfaces between compiler classes • Numscons: tentative to integrate scons and distutils Saturday, 9 June 12
  6. BENTO • Learning from the numscons experience • Written as

    a library from the ground up: • low coupling of commands • no global state • Scale down and up: simpler for simple packages, more flexible for bigger ones Saturday, 9 June 12
  7. A SIMPLE BENTO EXAMPLE Name: foo Version: 0.1 Description: Package

    description. Library: Packages: foo, foo.core Saturday, 9 June 12
  8. BENTOMAKER $ bentomaker configure $ bentomaker build $ bentomaker install

    # Or more simply (automatically # runs configure and build) $ bentomaker install Saturday, 9 June 12
  9. ADDING DATA-FILES Datafiles: pdf_documentation TargetDir: $pdfdir Files: main.pdf $ bentomaker

    configure --pdfdir=/usr/pdfdoc # ant-like glob format supported Datafiles: pdf_documentation TargetDir: $pdfdoc Files: doc/**/*.pdf Saturday, 9 June 12
  10. CUSTOM INSTALL PATHS Path: foodir Default: $datadir/foo Description: foo directory

    Datafiles: fubar-doc TargetDir: $foodir Files: main.foo $ bentomaker configure --foodir=/usr/foo Saturday, 9 June 12
  11. FROM DISTUTILS TO BENTO # convert setup.py to bento.info $

    bentomaker convert • Works for packages using distutils, setuptools, etc... • bentomaker may be distributed as a single file to avoid extra dependencies Saturday, 9 June 12
  12. PIP SUPPORT Simple setup.py to “look” like a distutils package:

    # setup.py import setuptools import bento.distutils bento.distutils.monkey_patch() # Grab metadata from bento.info setuptools.setup() Those 4 lines of code allow the package to be pip- installable Saturday, 9 June 12
  13. HOOK FILES (1) • Tweak or extend the packaging process

    with code, e.g. new command $ bentomaker hello hello from bento.commands import hooks @hooks.command def hello(context): print "Hello" • Hooks are pure python without any ‘magic’ mechanism Saturday, 9 June 12
  14. HOOKS FILES (2) from bento.commands import hooks @hooks.command def hello(context):

    print "hello nice world" @hooks.command def goodbye(context): print "goodbye cruel world !" @hooks.startup def startup(context): context.set_before("goodbye", "hello") Dependencies defined outside classes for lower coupling between commands Saturday, 9 June 12
  15. HOOK FILES (3) • Bento gives multiple “hooks” (pre/post/etc...) to

    customise things: • customise compilation flags, conditionally do things, etc... • bento written to be highly customised (integration with 3rd party tools to build C extensions, etc...) Saturday, 9 June 12
  16. PACKAGE METADATA (1) # bento.info content Version: 0.2 Author: John

    Doe MetaTemplateFile: foo/__info.py.in # foo/__info.py.in template content version = $VERSION author = $AUTHOR Allow for a package to know its own metadata: Saturday, 9 June 12
  17. PACKAGE METADATA (2) New metadata can be defined in the

    hook file: # foo/__info.py.in template content version = $VERSION git_revision = $GIT_REVISION author = $AUTHOR @hooks.pre_build def pre_build(context): context.register_metadata("git_revision", compute_git_revision()) Saturday, 9 June 12
  18. BUILD BACKENDS • Bento has the notion of “backends” to

    heavily customise some aspects of the build process • currently 3 pluggable backends (default, waf and ‘distutils’ mode) • code for a backend is relatively small (few 100 LOC) • Example Saturday, 9 June 12
  19. OUT-OF-TREE BUILDS Out-of-tree builds is building from a directory outside

    the source tree source_tree$ ls bento.info foo/__init__.py ... build_tree$ bentomaker --bento-info=.../bento.info Example Saturday, 9 June 12
  20. MORE • Ability to add options to commands • additional

    configure fine tuning (e.g. 3rd library location) • all options available from any command • Simple mechanism to retrieve resources (no more pkg_resources non-sense) • More to come Saturday, 9 June 12
  21. ADVANCED EXAMPLES: NUMPY • Numpy: initial rationale for bento •

    build scripts LOC reduced by 50 % • parallel and reliable partial build support • bento’s LOC ± numpy.distutils extensions ! Saturday, 9 June 12
  22. DECOUPLED ARCHITECTURE • bento written as a library from the

    ground up • no global variable or implicitly passed state • commands are not coupled with each other: • dependencies declared outside the commands • install and binary installers driven by a build manifest (json file) • options can be added outside commands Saturday, 9 June 12
  23. IDEMPOTENCY • idempotency: running the same command twice should give

    the same result • explicit list of installed files (no stalled files) • support for out-of-tree builds Saturday, 9 June 12
  24. WHAT’S NEXT • A few missing features to replace distutils

    (msi support, etc...) • Improve parser (faster and more reliable) • bentoya (aka bentoshop): pypi-like service + local db of installed packages: • uninstall support • apt-get-like features Saturday, 9 June 12
  25. WHERE TO GET IT • On github: http://github.com/cournape/Bento.git • Documentation:

    http://cournape.github.com/Bento • Mailing List: [email protected] Saturday, 9 June 12