re-inventing packaging and testing with Python

Keynote talk at Pycon Russia 25th Feburary 2013

Transcript

1. Re-inventing Python Packaging & Testing (0.9) Holger Krekel, http://twitter.com/hpk42 (http://twitter.com/hpk42) Pycon Russia, Ekaterinburg, 25th Feb 2013 (XKCD of course) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 1 of 41 03/01/2013 12:04 PM
2. Very happy to be here! file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 2 of 41 03/01/2013 12:04 PM
3. My background started games programming 1986 studied computer science in the 1990ties lots of Assembler, C, C++, distributed programming came to Python around 2001 co-founded PyPy project in 2003, core dev for many years author of pytest, tox, execnet libraries file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 3 of 41 03/01/2013 12:04 PM
4. And now for something completely different file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 4 of 41 03/01/2013 12:04 PM
5. Remember Perl: There is more than one way to do it. Python: There should be one -- and preferably only one -- obvious way to do it. file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 5 of 41 03/01/2013 12:04 PM
6. Second version syndrome Python3 may have beaten Perl6 (http://www.theregister.co.uk/2008/12 /04/python_3point0_release/) (from theregister.co.uk) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 6 of 41 03/01/2013 12:04 PM
7. but Perl is coming back to haunt us ... file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 7 of 41 03/01/2013 12:04 PM
8. Comprehensive Perl Archive network (CPAN) cpan.pm comes with base Perl installs: downloads/builds/installs and tests packages and deps cpan index has hundreds of public mirrors uses declarative metadata since 2003 can verify cryptographic signing of packages file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 8 of 41 03/01/2013 12:04 PM
9. CPAN server / uploading PAUSE is the upload server for perl packages: supports/can enforce versioning semantics development releases are mirrored but not indexed policy for taking over maintenance of abandoned packages provides a mirrored un-indexed "home" area for authors/uploaders for patches, little docs file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 9 of 41 03/01/2013 12:04 PM
10. Python Installer situation Feb 2013 no download+install tool coming with core Python two ways to download/install packages (pip, easy_install): pip installs only source distributions easy_install also binary distributions slow: dependency discovery requires client-side crawling for packages in all download and homepage sites no reliable way to test before install no cryptographic signature verification file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 10 of 41 03/01/2013 12:04 PM
11. upload/PYPI server interaction today pypi.python.org server code not easily deployable on your laptop no enforced versioning semantics no policy for taking over maintenance of abandoned packages brittle protocol: network and server failures of any download/homepage server lead to aborted installs or installation of wrong versions metadata through code (setup.py) has tons of problems, has been hard so far to move away from. and as to mirrors ... file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 11 of 41 03/01/2013 12:04 PM
12. CPAN Mirror status (http://mirrors.cpan.org/) (http://mirrors.cpan.org/ (http://mirrors.cpan.org/) ) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 12 of 41 03/01/2013 12:04 PM
13. PyPI Mirror status (http://www.pypi- mirrors.org) (http://www.pypi-mirrors.org (http://www.pypi-mirrors.org) ) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 13 of 41 03/01/2013 12:04 PM
14. We can stop the comparison already ... (from www.cavstheblog.com) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 14 of 41 03/01/2013 12:04 PM
15. Three Lessons learned file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 15 of 41 03/01/2013 12:04 PM
16. Comparing with others can quickly make you unhappy file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 16 of 41 03/01/2013 12:04 PM
17. Perl and Python both not living up to their mantras file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 17 of 41 03/01/2013 12:04 PM
18. There is a lot to improve on file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 18 of 41 03/01/2013 12:04 PM
19. Python Tools and PEPs to improve setuptools/distribute projects pip/distlib/pkglib/... projects PEP386 new version comparison for distutils PEP345 Packaging Metadata 1.2 PEP376 infrastructure for managing distributions locally PEP405 virtual environments PEP426 Packaging Metadata Version 1.3 (finishing) PEP427 Wheel binary package format (APPROVED) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 19 of 41 03/01/2013 12:04 PM
20. beware: the standardization trap (one of the many great XKCD comics) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 20 of 41 03/01/2013 12:04 PM
21. To avoid the standardization trap: Don't demand that the world changes first before your tool/idea can be used (was also my motto when starting PyPy) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 21 of 41 03/01/2013 12:04 PM
22. Rather: write meta tools configure and invoke existing tools make it work for most use cases enable new facilities/standards from there file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 22 of 41 03/01/2013 12:04 PM
23. example from the testing arena (you do know the goat is Python's testing mascott, right?) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 23 of 41 03/01/2013 12:04 PM
24. Python Testing tools nose: popular unittest-runner py.test: popular generic test runner unittest/2: standard library TestCases zope.testing: layered-setup unittest-runner twisted.trial: running twisted/async test cases many other methods, including "make test" setuptools has "setup.py test" Would inventing a new test runner help to standardize testing? file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 24 of 41 03/01/2013 12:04 PM
25. tox: a "meta" test running tool mission: standardize testing in Python configure multiple dependency configs install packages in virtualenv environments run tests with your tools of choice act as a frontend to CI servers See http://tox.testrun.org (http://tox.testrun.org) for details. file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 25 of 41 03/01/2013 12:04 PM
26. But that's about testing ... what about packaging/pypi? file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 26 of 41 03/01/2013 12:04 PM
27. I did a little prototyping ... file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 27 of 41 03/01/2013 12:04 PM
28. shown first here at Pycon Russia :) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 28 of 41 03/01/2013 12:04 PM
29. devpi: a meta tool for packaging activities devpi serves two purposes: it's a new compatible index and upload server subcommands for managing release and QA workflows file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 29 of 41 03/01/2013 12:04 PM
30. why a new approach to index serving? there is no good existing pypi server to build these features: multiple indexes and staging (dev->QA->PROD) merging of upstream (pypi.python.org) indexes server-side crawling of download/homepages recording of automated test results with each package seemlessly allow offline usage (e. g. on a laptop) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 30 of 41 03/01/2013 12:04 PM
31. Open Source principle: talk is cheap, show working code! file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 31 of 41 03/01/2013 12:04 PM
32. $ devpi server multiple indexes, organized as ~USERNAME/INDEXNAME each index has an upstream index whose packages are seemlessly merged each index is "crawl-clean" for clients uploading a package to an index will not push it to upstream compatible for pip, easy_install and setup.py upload enough theory - on to a a little demo :) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 32 of 41 03/01/2013 12:04 PM
33. summary of devpi server demo insights upstream merging makes a private overlay index simple crawling at server-side makes it easy and fast for installers devpi maintains a nginx-servable static view but: typing in index server urls, maintaining .pypirc config is burdensome with multiple indexes ... therefore ... file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 33 of 41 03/01/2013 12:04 PM
34. devpi also provides "workflow" subcommands use set current pypi index install download and install one or more packages upload build and upload packages from a checkout test download and test a package push push a package to another index file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 34 of 41 03/01/2013 12:04 PM
35. $ devpi test example: $ devpi test -e py25,py33 mypkg download and unpack a package from current index call tox for provisioning and performing tests stream test results from a test run (currently pytest only, nose/unittest to follow) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 35 of 41 03/01/2013 12:04 PM
36. $ devpi push example: $ devpi push mypkg https://pypi.python.org pushes uploaded packages to another index currently specified by "PKGNAME-VER" in the future this could be done to constraints like "all tests on windows need to pass" file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 36 of 41 03/01/2013 12:04 PM
37. devpi: typical Open Source work flow start a localhost index server: $ devpi server 1. use a user-specific index: $ devpi use http://localhost:3141/~hpk42/dev/ (http://localhost:3141/~hpk42 /dev/) 2. upload a package from a checkout: $ devpi upload 3. perform tests on multiple machines from multiple users: $ devpi test PKGNAME 4. test results at http://localhost:3141/~hpk42/dev/tests (http://localhost:3141/~hpk42 /dev/tests) 5. when release is ready: $ devpi push PKG https://pypi.python.org 6. file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 37 of 41 03/01/2013 12:04 PM
38. devpi implementation plan to be MIT licensed test-driven development get early adopters and streamline processes according to real-life usage don't aim for standard lib or PEPs (for now) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 38 of 41 03/01/2013 12:04 PM
39. my priorities use cases from contracting companies use cases from my and others open source work file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 39 of 41 03/01/2013 12:04 PM
40. Open areas / future work use VCS for index persistence? plugin system for upload/serve/test/install implement mirroring of index as static files support declarative metadata / Wheel format /... support server-side PGP signing and verification a public, redundant,global web service CI: per-commit automatic integration, testing and packaging file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 40 of 41 03/01/2013 12:04 PM
41. devpi going public mid 2013 ... aiming for a <1.0 public release, to involve early adopters and contributors for finalizing processes and good defaults. Approach me here at Pycon Russia 2013 and thanks for listening! No public channels yet, but am going to announce them via: http://twitter.com/hpk42 (http://twitter.com/hpk42) http://holgerkrekel.net (http://holgerkrekel.net) file:///home/hpk/p/pycon-russia-keynote/html/index.html#... 41 of 41 03/01/2013 12:04 PM