WHO AM I ? • Started to be involved with numpy/scipy around 2006/2007 • Particularly involved with build/installer issues • Why do I care about packaging ? Saturday, 9 June 12
WHAT IS PACKAGING ? • Going from sources to the user • Different users require different workflows: not just tarballs ! • allow for creating binary installers • integrating with native systems (.deb, .rpm, etc...) Saturday, 9 June 12
DISTUTILS • De-facto solution since ±2000 • Works well for simple packages • Setuptools/distribute: extensions on top of distutils • main feature: package dependency (pip, easy_install) Saturday, 9 June 12
DISTUTILS ISSUES • Non-trivial customisation is hard: • new compiler support • creating new commands (test, build doc, etc...) • Anything related to compilation is hard and fragile • Extensions based on monkey- patching Saturday, 9 June 12
DISTUTILS ISSUES (2) • Numpy/Scipy distutils extensions ± 10 KLOC (fortran, etc...) • Numerous issues when setuptools appeared (monkey patching overrides, etc...) • Poor code quality, e.g.: • attributes added at runtime in some conditions • lack of common interfaces between compiler classes • Numscons: tentative to integrate scons and distutils Saturday, 9 June 12
BENTO • Learning from the numscons experience • Written as a library from the ground up: • low coupling of commands • no global state • Scale down and up: simpler for simple packages, more flexible for bigger ones Saturday, 9 June 12
FROM DISTUTILS TO BENTO # convert setup.py to bento.info $ bentomaker convert • Works for packages using distutils, setuptools, etc... • bentomaker may be distributed as a single file to avoid extra dependencies Saturday, 9 June 12
PIP SUPPORT Simple setup.py to “look” like a distutils package: # setup.py import setuptools import bento.distutils bento.distutils.monkey_patch() # Grab metadata from bento.info setuptools.setup() Those 4 lines of code allow the package to be pip- installable Saturday, 9 June 12
HOOK FILES (1) • Tweak or extend the packaging process with code, e.g. new command $ bentomaker hello hello from bento.commands import hooks @hooks.command def hello(context): print "Hello" • Hooks are pure python without any ‘magic’ mechanism Saturday, 9 June 12
HOOK FILES (3) • Bento gives multiple “hooks” (pre/post/etc...) to customise things: • customise compilation flags, conditionally do things, etc... • bento written to be highly customised (integration with 3rd party tools to build C extensions, etc...) Saturday, 9 June 12
PACKAGE METADATA (1) # bento.info content Version: 0.2 Author: John Doe MetaTemplateFile: foo/__info.py.in # foo/__info.py.in template content version = $VERSION author = $AUTHOR Allow for a package to know its own metadata: Saturday, 9 June 12
PACKAGE METADATA (2) New metadata can be defined in the hook file: # foo/__info.py.in template content version = $VERSION git_revision = $GIT_REVISION author = $AUTHOR @hooks.pre_build def pre_build(context): context.register_metadata("git_revision", compute_git_revision()) Saturday, 9 June 12
BUILD BACKENDS • Bento has the notion of “backends” to heavily customise some aspects of the build process • currently 3 pluggable backends (default, waf and ‘distutils’ mode) • code for a backend is relatively small (few 100 LOC) • Example Saturday, 9 June 12
OUT-OF-TREE BUILDS Out-of-tree builds is building from a directory outside the source tree source_tree$ ls bento.info foo/__init__.py ... build_tree$ bentomaker --bento-info=.../bento.info Example Saturday, 9 June 12
MORE • Ability to add options to commands • additional configure fine tuning (e.g. 3rd library location) • all options available from any command • Simple mechanism to retrieve resources (no more pkg_resources non-sense) • More to come Saturday, 9 June 12
ADVANCED EXAMPLES: NUMPY • Numpy: initial rationale for bento • build scripts LOC reduced by 50 % • parallel and reliable partial build support • bento’s LOC ± numpy.distutils extensions ! Saturday, 9 June 12
DECOUPLED ARCHITECTURE • bento written as a library from the ground up • no global variable or implicitly passed state • commands are not coupled with each other: • dependencies declared outside the commands • install and binary installers driven by a build manifest (json file) • options can be added outside commands Saturday, 9 June 12
IDEMPOTENCY • idempotency: running the same command twice should give the same result • explicit list of installed files (no stalled files) • support for out-of-tree builds Saturday, 9 June 12
WHAT’S NEXT • A few missing features to replace distutils (msi support, etc...) • Improve parser (faster and more reliable) • bentoya (aka bentoshop): pypi-like service + local db of installed packages: • uninstall support • apt-get-like features Saturday, 9 June 12
WHERE TO GET IT • On github: http://github.com/cournape/Bento.git • Documentation: http://cournape.github.com/Bento • Mailing List: [email protected] Saturday, 9 June 12
cournape
psj59129
youtalk
philipschwarz
jpt_corp_official
efbmzwyk999
uhooi
kojipole
minma_curama
ninoid
hiranabe
yuhta28
seike460
matthewcrist
jonyablonski
wjessup
yeseniaperezcruz
lara
rocio
lemiorhan
chrislema
aki_iinuma
smashingmag
lauravandoore
zakiwarfel