Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bento presentation for Pycon APAC 2012

Bento presentation for Pycon APAC 2012

Slides I presented in June 2012 at Singapore for Pycon APAC for bento, a pythonic packaging software.

Present the main features of bento

cournape

June 09, 2012
Tweet

More Decks by cournape

Other Decks in Programming

Transcript

  1. ห౰(BENTO)
    A Python solution for packaging
    Saturday, 9 June 12

    View Slide

  2. WHO AM I ?
    • Started to be involved with numpy/scipy around 2006/2007
    • Particularly involved with build/installer issues
    • Why do I care about packaging ?
    Saturday, 9 June 12

    View Slide

  3. WHAT IS PACKAGING ?
    • Going from sources to the user
    • Different users require different workflows: not just tarballs !
    • allow for creating binary installers
    • integrating with native systems (.deb, .rpm, etc...)
    Saturday, 9 June 12

    View Slide

  4. DISTUTILS
    • De-facto solution since ±2000
    • Works well for simple packages
    • Setuptools/distribute: extensions on top of distutils
    • main feature: package dependency (pip, easy_install)
    Saturday, 9 June 12

    View Slide

  5. DISTUTILS ISSUES
    • Non-trivial customisation is hard:
    • new compiler support
    • creating new commands (test, build
    doc, etc...)
    • Anything related to compilation is
    hard and fragile
    • Extensions based on monkey-
    patching
    Saturday, 9 June 12

    View Slide

  6. DISTUTILS ISSUES (2)
    • Numpy/Scipy distutils extensions ± 10 KLOC (fortran, etc...)
    • Numerous issues when setuptools appeared (monkey
    patching overrides, etc...)
    • Poor code quality, e.g.:
    • attributes added at runtime in some conditions
    • lack of common interfaces between compiler classes
    • Numscons: tentative to integrate scons and distutils
    Saturday, 9 June 12

    View Slide

  7. BENTO
    • Learning from the numscons experience
    • Written as a library from the ground up:
    • low coupling of commands
    • no global state
    • Scale down and up: simpler for simple packages, more flexible
    for bigger ones
    Saturday, 9 June 12

    View Slide

  8. BASICS
    Saturday, 9 June 12

    View Slide

  9. A SIMPLE BENTO EXAMPLE
    Name: foo
    Version: 0.1
    Description:
    Package description.
    Library:
    Packages: foo, foo.core
    Saturday, 9 June 12

    View Slide

  10. BENTOMAKER
    $ bentomaker configure
    $ bentomaker build
    $ bentomaker install
    # Or more simply (automatically
    # runs configure and build)
    $ bentomaker install
    Saturday, 9 June 12

    View Slide

  11. ADDING DATA-FILES
    Datafiles: pdf_documentation
    TargetDir: $pdfdir
    Files: main.pdf
    $ bentomaker configure --pdfdir=/usr/pdfdoc
    # ant-like glob format supported
    Datafiles: pdf_documentation
    TargetDir: $pdfdoc
    Files: doc/**/*.pdf
    Saturday, 9 June 12

    View Slide

  12. CUSTOM INSTALL PATHS
    Path: foodir
    Default: $datadir/foo
    Description: foo directory
    Datafiles: fubar-doc
    TargetDir: $foodir
    Files: main.foo
    $ bentomaker configure --foodir=/usr/foo
    Saturday, 9 June 12

    View Slide

  13. FITTING INTO THE EXISTING
    PYTHON ECOSYSTEM
    Saturday, 9 June 12

    View Slide

  14. FROM DISTUTILS TO BENTO
    # convert setup.py to bento.info
    $ bentomaker convert
    • Works for packages using distutils, setuptools, etc...
    • bentomaker may be distributed as a single file to avoid extra
    dependencies
    Saturday, 9 June 12

    View Slide

  15. PIP SUPPORT
    Simple setup.py to “look” like a distutils package:
    # setup.py
    import setuptools
    import bento.distutils
    bento.distutils.monkey_patch()
    # Grab metadata from bento.info
    setuptools.setup()
    Those 4 lines of code allow the package to be pip-
    installable
    Saturday, 9 June 12

    View Slide

  16. ADVANCED USAGE
    Saturday, 9 June 12

    View Slide

  17. HOOK FILES (1)
    • Tweak or extend the packaging process with code, e.g. new
    command
    $ bentomaker hello
    hello
    from bento.commands import hooks
    @hooks.command
    def hello(context):
    print "Hello"
    • Hooks are pure python without any ‘magic’ mechanism
    Saturday, 9 June 12

    View Slide

  18. HOOKS FILES (2)
    from bento.commands import hooks
    @hooks.command
    def hello(context):
    print "hello nice world"
    @hooks.command
    def goodbye(context):
    print "goodbye cruel world !"
    @hooks.startup
    def startup(context):
    context.set_before("goodbye", "hello")
    Dependencies defined outside classes for lower
    coupling between commands
    Saturday, 9 June 12

    View Slide

  19. HOOK FILES (3)
    • Bento gives multiple “hooks” (pre/post/etc...) to customise
    things:
    • customise compilation flags, conditionally do things, etc...
    • bento written to be highly customised (integration with 3rd
    party tools to build C extensions, etc...)
    Saturday, 9 June 12

    View Slide

  20. PACKAGE METADATA (1)
    # bento.info content
    Version: 0.2
    Author: John Doe
    MetaTemplateFile: foo/__info.py.in
    # foo/__info.py.in template content
    version = $VERSION
    author = $AUTHOR
    Allow for a package to know its own metadata:
    Saturday, 9 June 12

    View Slide

  21. PACKAGE METADATA (2)
    New metadata can be defined in the hook file:
    # foo/__info.py.in template content
    version = $VERSION
    git_revision = $GIT_REVISION
    author = $AUTHOR
    @hooks.pre_build
    def pre_build(context):
    context.register_metadata("git_revision",
    compute_git_revision())
    Saturday, 9 June 12

    View Slide

  22. BUILD BACKENDS
    • Bento has the notion of “backends” to heavily customise some
    aspects of the build process
    • currently 3 pluggable backends (default, waf and ‘distutils’
    mode)
    • code for a backend is relatively small (few 100 LOC)
    • Example
    Saturday, 9 June 12

    View Slide

  23. OUT-OF-TREE BUILDS
    Out-of-tree builds is building from a directory outside the
    source tree
    source_tree$ ls
    bento.info
    foo/__init__.py
    ...
    build_tree$ bentomaker --bento-info=.../bento.info
    Example
    Saturday, 9 June 12

    View Slide

  24. MORE
    • Ability to add options to commands
    • additional configure fine tuning (e.g. 3rd library location)
    • all options available from any command
    • Simple mechanism to retrieve resources (no more
    pkg_resources non-sense)
    • More to come
    Saturday, 9 June 12

    View Slide

  25. ADVANCED EXAMPLES:
    NUMPY
    • Numpy: initial rationale for bento
    • build scripts LOC reduced by 50 %
    • parallel and reliable partial build support
    • bento’s LOC ± numpy.distutils extensions !
    Saturday, 9 June 12

    View Slide

  26. INTERNALS
    Saturday, 9 June 12

    View Slide

  27. DECOUPLED ARCHITECTURE
    • bento written as a library from the ground up
    • no global variable or implicitly passed state
    • commands are not coupled with each other:
    • dependencies declared outside the commands
    • install and binary installers driven by a build manifest (json
    file)
    • options can be added outside commands
    Saturday, 9 June 12

    View Slide

  28. IDEMPOTENCY
    • idempotency: running the same command twice should give
    the same result
    • explicit list of installed files (no stalled files)
    • support for out-of-tree builds
    Saturday, 9 June 12

    View Slide

  29. WHAT’S NEXT
    • A few missing features to replace distutils (msi support, etc...)
    • Improve parser (faster and more reliable)
    • bentoya (aka bentoshop): pypi-like service + local db of
    installed packages:
    • uninstall support
    • apt-get-like features
    Saturday, 9 June 12

    View Slide

  30. WHERE TO GET IT
    • On github: http://github.com/cournape/Bento.git
    • Documentation: http://cournape.github.com/Bento
    • Mailing List: [email protected]
    Saturday, 9 June 12

    View Slide