Upgrade to Pro — share decks privately, control downloads, hide ads and more …

re-inventing packaging and testing with Python

re-inventing packaging and testing with Python

Keynote talk at Pycon Russia 25th Feburary 2013

holger krekel

February 25, 2013
Tweet

More Decks by holger krekel

Other Decks in Programming

Transcript

  1. Re-inventing Python Packaging &
    Testing (0.9)
    Holger Krekel, http://twitter.com/hpk42 (http://twitter.com/hpk42)
    Pycon Russia, Ekaterinburg, 25th Feb 2013
    (XKCD of course)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    1 of 41 03/01/2013 12:04 PM

    View full-size slide

  2. Very happy to be here!
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    2 of 41 03/01/2013 12:04 PM

    View full-size slide

  3. My background
    started games programming 1986
    studied computer science in the 1990ties
    lots of Assembler, C, C++, distributed programming
    came to Python around 2001
    co-founded PyPy project in 2003, core dev for many years
    author of pytest, tox, execnet libraries
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    3 of 41 03/01/2013 12:04 PM

    View full-size slide

  4. And now for something completely
    different
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    4 of 41 03/01/2013 12:04 PM

    View full-size slide

  5. Remember
    Perl:
    There is more than one way to do it.
    Python:
    There should be one -- and preferably only one -- obvious way to do it.
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    5 of 41 03/01/2013 12:04 PM

    View full-size slide

  6. Second version syndrome
    Python3 may have beaten Perl6
    (http://www.theregister.co.uk/2008/12
    /04/python_3point0_release/)
    (from theregister.co.uk)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    6 of 41 03/01/2013 12:04 PM

    View full-size slide

  7. but Perl is coming back to haunt us ...
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    7 of 41 03/01/2013 12:04 PM

    View full-size slide

  8. Comprehensive Perl Archive network
    (CPAN)
    cpan.pm comes with base Perl installs:
    downloads/builds/installs and tests packages and deps
    cpan index has hundreds of public mirrors
    uses declarative metadata since 2003
    can verify cryptographic signing of packages
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    8 of 41 03/01/2013 12:04 PM

    View full-size slide

  9. CPAN server / uploading
    PAUSE is the upload server for perl packages:
    supports/can enforce versioning semantics
    development releases are mirrored but not indexed
    policy for taking over maintenance of abandoned packages
    provides a mirrored un-indexed "home" area for authors/uploaders for patches, little
    docs
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    9 of 41 03/01/2013 12:04 PM

    View full-size slide

  10. Python Installer situation Feb 2013
    no download+install tool coming with core Python
    two ways to download/install packages (pip, easy_install):
    pip installs only source distributions
    easy_install also binary distributions
    slow: dependency discovery requires client-side crawling for packages in all
    download and homepage sites
    no reliable way to test before install
    no cryptographic signature verification
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    10 of 41 03/01/2013 12:04 PM

    View full-size slide

  11. upload/PYPI server interaction today
    pypi.python.org server code not easily deployable on your laptop
    no enforced versioning semantics
    no policy for taking over maintenance of abandoned packages
    brittle protocol: network and server failures of any download/homepage server
    lead to aborted installs or installation of wrong versions
    metadata through code (setup.py) has tons of problems, has been hard so far
    to move away from.
    and as to mirrors ...
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    11 of 41 03/01/2013 12:04 PM

    View full-size slide

  12. CPAN Mirror status
    (http://mirrors.cpan.org/)
    (http://mirrors.cpan.org/ (http://mirrors.cpan.org/) )
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    12 of 41 03/01/2013 12:04 PM

    View full-size slide

  13. PyPI Mirror status
    (http://www.pypi-
    mirrors.org)
    (http://www.pypi-mirrors.org (http://www.pypi-mirrors.org) )
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    13 of 41 03/01/2013 12:04 PM

    View full-size slide

  14. We can stop the comparison already ...
    (from www.cavstheblog.com)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    14 of 41 03/01/2013 12:04 PM

    View full-size slide

  15. Three Lessons learned
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    15 of 41 03/01/2013 12:04 PM

    View full-size slide

  16. Comparing with others can quickly
    make you unhappy
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    16 of 41 03/01/2013 12:04 PM

    View full-size slide

  17. Perl and Python both not living up to
    their mantras
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    17 of 41 03/01/2013 12:04 PM

    View full-size slide

  18. There is a lot to improve on
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    18 of 41 03/01/2013 12:04 PM

    View full-size slide

  19. Python Tools and PEPs to improve
    setuptools/distribute projects
    pip/distlib/pkglib/... projects
    PEP386 new version comparison for distutils
    PEP345 Packaging Metadata 1.2
    PEP376 infrastructure for managing distributions locally
    PEP405 virtual environments
    PEP426 Packaging Metadata Version 1.3 (finishing)
    PEP427 Wheel binary package format (APPROVED)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    19 of 41 03/01/2013 12:04 PM

    View full-size slide

  20. beware: the standardization trap
    (one of the many great XKCD comics)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    20 of 41 03/01/2013 12:04 PM

    View full-size slide

  21. To avoid the standardization trap:
    Don't demand that the world changes first
    before your tool/idea can be used
    (was also my motto when starting PyPy)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    21 of 41 03/01/2013 12:04 PM

    View full-size slide

  22. Rather: write meta tools
    configure and invoke existing tools
    make it work for most use cases
    enable new facilities/standards from there
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    22 of 41 03/01/2013 12:04 PM

    View full-size slide

  23. example from the testing arena
    (you do know the goat is Python's testing mascott, right?)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    23 of 41 03/01/2013 12:04 PM

    View full-size slide

  24. Python Testing tools
    nose: popular unittest-runner
    py.test: popular generic test runner
    unittest/2: standard library TestCases
    zope.testing: layered-setup unittest-runner
    twisted.trial: running twisted/async test cases
    many other methods, including "make test"
    setuptools has "setup.py test"
    Would inventing a new test runner help to standardize testing?
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    24 of 41 03/01/2013 12:04 PM

    View full-size slide

  25. tox: a "meta" test running tool
    mission: standardize testing in Python
    configure multiple dependency configs
    install packages in virtualenv environments
    run tests with your tools of choice
    act as a frontend to CI servers
    See http://tox.testrun.org (http://tox.testrun.org) for details.
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    25 of 41 03/01/2013 12:04 PM

    View full-size slide

  26. But that's about testing ...
    what about packaging/pypi?
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    26 of 41 03/01/2013 12:04 PM

    View full-size slide

  27. I did a little prototyping ...
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    27 of 41 03/01/2013 12:04 PM

    View full-size slide

  28. shown first here at Pycon Russia :)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    28 of 41 03/01/2013 12:04 PM

    View full-size slide

  29. devpi: a meta tool for packaging
    activities
    devpi serves two purposes:
    it's a new compatible index and upload server
    subcommands for managing release and QA workflows
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    29 of 41 03/01/2013 12:04 PM

    View full-size slide

  30. why a new approach to index serving?
    there is no good existing pypi server to build these features:
    multiple indexes and staging (dev->QA->PROD)
    merging of upstream (pypi.python.org) indexes
    server-side crawling of download/homepages
    recording of automated test results with each package
    seemlessly allow offline usage (e. g. on a laptop)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    30 of 41 03/01/2013 12:04 PM

    View full-size slide

  31. Open Source principle:
    talk is cheap, show working code!
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    31 of 41 03/01/2013 12:04 PM

    View full-size slide

  32. $ devpi server
    multiple indexes, organized as ~USERNAME/INDEXNAME
    each index has an upstream index whose packages are seemlessly merged
    each index is "crawl-clean" for clients
    uploading a package to an index will not push it to upstream
    compatible for pip, easy_install and setup.py upload
    enough theory - on to a a little demo :)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    32 of 41 03/01/2013 12:04 PM

    View full-size slide

  33. summary of devpi server demo insights
    upstream merging makes a private overlay index simple
    crawling at server-side makes it easy and fast for installers
    devpi maintains a nginx-servable static view
    but:
    typing in index server urls, maintaining .pypirc config is burdensome with
    multiple indexes ... therefore ...
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    33 of 41 03/01/2013 12:04 PM

    View full-size slide

  34. devpi also provides "workflow"
    subcommands
    use set current pypi index
    install download and install one or more packages
    upload build and upload packages from a checkout
    test download and test a package
    push push a package to another index
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    34 of 41 03/01/2013 12:04 PM

    View full-size slide

  35. $ devpi test
    example:
    $ devpi test -e py25,py33 mypkg
    download and unpack a package from current index
    call tox for provisioning and performing tests
    stream test results from a test run (currently pytest only, nose/unittest to follow)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    35 of 41 03/01/2013 12:04 PM

    View full-size slide

  36. $ devpi push
    example:
    $ devpi push mypkg https://pypi.python.org
    pushes uploaded packages to another index
    currently specified by "PKGNAME-VER"
    in the future this could be done to constraints like "all tests on windows need to
    pass"
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    36 of 41 03/01/2013 12:04 PM

    View full-size slide

  37. devpi: typical Open Source work flow
    start a localhost index server:
    $ devpi server
    1.
    use a user-specific index:
    $ devpi use http://localhost:3141/~hpk42/dev/ (http://localhost:3141/~hpk42
    /dev/)
    2.
    upload a package from a checkout:
    $ devpi upload
    3.
    perform tests on multiple machines from multiple users:
    $ devpi test PKGNAME
    4.
    test results at http://localhost:3141/~hpk42/dev/tests (http://localhost:3141/~hpk42
    /dev/tests)
    5.
    when release is ready:
    $ devpi push PKG https://pypi.python.org
    6.
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    37 of 41 03/01/2013 12:04 PM

    View full-size slide

  38. devpi implementation plan
    to be MIT licensed
    test-driven development
    get early adopters and streamline processes according to real-life usage
    don't aim for standard lib or PEPs (for now)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    38 of 41 03/01/2013 12:04 PM

    View full-size slide

  39. my priorities
    use cases from contracting companies
    use cases from my and others open source work
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    39 of 41 03/01/2013 12:04 PM

    View full-size slide

  40. Open areas / future work
    use VCS for index persistence?
    plugin system for upload/serve/test/install
    implement mirroring of index as static files
    support declarative metadata / Wheel format /...
    support server-side PGP signing and verification
    a public, redundant,global web service
    CI: per-commit automatic integration, testing and packaging
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    40 of 41 03/01/2013 12:04 PM

    View full-size slide

  41. devpi going public mid 2013 ...
    aiming for a <1.0 public release, to involve early adopters and contributors for finalizing
    processes and good defaults.
    Approach me here at Pycon Russia 2013
    and thanks for listening!
    No public channels yet, but am going to announce them via:
    http://twitter.com/hpk42 (http://twitter.com/hpk42)
    http://holgerkrekel.net (http://holgerkrekel.net)
    file:///home/hpk/p/pycon-russia-keynote/html/index.html#...
    41 of 41 03/01/2013 12:04 PM

    View full-size slide