PyCon 2013 Tutorial: Script to PyPI to GitHub & More

How do you start a new project? How do you deliver a script to co-workers? How do you develop it with best practices? How do you use virtualenv and pip? How do you package it? How do you automate testing, building, uploading to PyPI?

1. Script to PyPi to Github Script to PyPi to Github @__mharrison__ @__mharrison__ http://hairysun.com http://hairysun.com
2. About Me About Me • 12 years Python 12 years Python • Worked in HA, Search, Open Source, BI Worked in HA, Search, Open Source, BI and Storage and Storage • Author of multiple Python Books Author of multiple Python Books
3. Continuums Continuums More like perl-TMTOWTDI More like perl-TMTOWTDI
4. Agenda Agenda • Project Development Project Development – Versioning Versioning – Configuration Configuration – Logging Logging – File input File input – Shell invocation Shell invocation • Environment Layout * virtualenv * pip Environment Layout * virtualenv * pip • Project layout Project layout
5. Agenda (2) Agenda (2) • Documentation Documentation • Automation/Makefile Automation/Makefile • Packaging Packaging – setup.py setup.py – PyPi PyPi • Testing Testing • Github Github • Travis CI Travis CI • Poachplate Poachplate
6. Begin Begin
7. Warning Warning • Starting from basic Python knowledge Starting from basic Python knowledge • Hands on Hands on – (short) lecture (short) lecture – (short) code (short) code – repeat until time is gone repeat until time is gone
8. Project Project Create pycat. Pythonic implementation of Create pycat. Pythonic implementation of cat cat
9. Scripting Scripting
10. hello world hello world cat.py cat.py import import sys sys for for line line in in sys sys. .stdin: stdin: print print line, line,
11. hello world - Python 3 hello world - Python 3 import import sys sys for for line line in in sys sys. .stdin: stdin: print print(line, end (line, end= ='' '') )
12. 2 or 3? 2 or 3?
13. 2 or 3? 2 or 3? • Better legacy/library support for 2 Better legacy/library support for 2 • Possible to support both Possible to support both
14. hello world hello world import import sys sys for for line line in in sys sys. .stdin: stdin: sys sys. .stdout stdout. .write(line) write(line)
15. Assignment Assignment create create cat.py cat.py
16. Single File or Project? Single File or Project?
17. Layout Layout Can depend on distribution mechanism: Can depend on distribution mechanism: • Single file Single file • Zip file Zip file – Python entry point Python entry point – Splat/run Splat/run • System package System package • PyPi/Distribute/pip package PyPi/Distribute/pip package
18. Single File Single File • chmod chmod and place in and place in $PATH $PATH • add add #!/usr/bin/env python #!/usr/bin/env python
19. Single File (2) Single File (2) • No reuse No reuse
20. Zip File Zip File • PYTHONPATH=cat.zip python -m PYTHONPATH=cat.zip python -m __main__ __main__ • tar -zxvf cat.zip; cd cat; tar -zxvf cat.zip; cd cat; python cat.py python cat.py
21. Zip File (2) Zip File (2) • No reuse No reuse • No stacktrace No stacktrace
22. System Package System Package • emerge -av qtile emerge -av qtile (rpm|apt|brew) (rpm|apt|brew)
23. System Package (2) System Package (2) • Requires root Requires root • At mercy of packager (maybe worse than At mercy of packager (maybe worse than PyPi) PyPi) • Reuse Reuse • Limited to single version Limited to single version • python -m modulename python -m modulename
24. Pip Package Pip Package “ “Best practice” is combo of: Best practice” is combo of: • Distribute Distribute • Virtualenv Virtualenv • Pip Pip
25. Pip Package (2) Pip Package (2) $ virtualenv catenv $ virtualenv catenv $ source catenv/bin/activate $ source catenv/bin/activate $ pip install pycat $ pip install pycat
26. Pip Package (3) Pip Package (3) • Multiple versions Multiple versions • Reuse Reuse • Effort to create Effort to create setup.py setup.py
27. Minimal Example Layout Minimal Example Layout Project/ Project/ README.txt README.txt project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
28. Better Example Layout Better Example Layout Project/ Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
29. Assignment Assignment create layout for create layout for cat.py cat.py
30. Semantic Versioning Semantic Versioning
31. Versioning Versioning http://semver.org http://semver.org Formal spec for versioning projects Formal spec for versioning projects
32. Python Versioning Python Versioning PEP 386 PEP 386 N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN] N.N[.n]+[{a|b|c}N[.N]+][.postN][.devN] 1.0a1 < 1.0a2 < 1.0b2.post345 1.0a1 < 1.0a2 < 1.0b2.post345
33. setup.py setup.py from from distutils.core distutils.core import import setup setup setup(name setup(name= ='PyCat' 'PyCat', ,.... .... version version= ='1.0' '1.0') )
34. Where to store version? Where to store version? In In module.__version__ module.__version__ (might cause (might cause importing issues) importing issues)
35. Version Version If using Sphinx for docs, be sure to update: If using Sphinx for docs, be sure to update: docs/ docs/ conf.py conf.py
36. argparse argparse ap ap = = argparse argparse. .ArgumentParser(version ArgumentParser(version= ='1.0' '1.0') ) ... ... ap ap. .print_version() print_version()
37. Assignment Assignment Add version to project Add version to project
38. Configuration Configuration
39. Configuration Configuration • Python Python – Django ( Django (settings.py settings.py) ) – Python ( Python (site.py site.py, , setup.py setup.py) ) • JSON/YAML JSON/YAML – Google App Engine Google App Engine • Environment Variables Environment Variables – Python ( Python (PYTHONPATH PYTHONPATH) ) • .ini ( .ini (ConfigParser ConfigParser, , ConfigObj ConfigObj) ) – matplotlib matplotlib – ipython ipython • Command line options ( Command line options (argparse argparse) ) • Sqlite blob Sqlite blob • Shelve/pickle blob Shelve/pickle blob
40. Configuration (2) Configuration (2) Are not mutually exclusive Are not mutually exclusive
41. Configuration (3) Configuration (3) (Unix) Hierarchy: (Unix) Hierarchy: • System rc (run control) ( System rc (run control) (/etc/conf.d /etc/conf.d) ) • User rc ( User rc (~/.config/app/... ~/.config/app/...) ) • Environment variables Environment variables • Command line options Command line options http://www.faqs.org/docs/artu/ch10s02.html http://www.faqs.org/docs/artu/ch10s02.html Filesystem Hierarchy Standard: Filesystem Hierarchy Standard: http://www.pathname.com/fhs/ http://www.pathname.com/fhs/
42. Configuration (4) Configuration (4) • Plain text config is easily approachable Plain text config is easily approachable • Careful with Python config on process Careful with Python config on process run by root run by root
43. Assignment Assignment Add configuration Add configuration -n -n to to show line numbers show line numbers
44. Logging Logging
45. Logging Logging logging logging module provides feature-rich module provides feature-rich logging logging
46. Logging (2) Logging (2) import import logging logging logging logging. .basicConfig(level basicConfig(level= =logging logging. .ERROR, ERROR, filename filename= ='.log' '.log') ) ... ... logging logging. .error( error('Error encountered in...' 'Error encountered in...') )
47. Assignment Assignment Add configuration Add configuration --verbose --verbose to log file to log file being “catted” being “catted”
48. Dealing with File Input Dealing with File Input
49. “ “Files” Files” Dealing with? Dealing with? • Filename Filename • file object file object • string data string data
50. Filename Filename Somewhat analogous to an Somewhat analogous to an Iterable Iterable. . • Can open/iterate many times Can open/iterate many times • Implementation depends on file Implementation depends on file • Need to manage closing file Need to manage closing file
51. File Object File Object Somewhat analogous to an Somewhat analogous to an Iterator Iterator. . • Can iterate once (unless Can iterate once (unless seek seeked) ed) • Can accept Can accept file file, , StringIO StringIO, , socket socket, , generator, etc generator, etc • Memory friendly - scalable Memory friendly - scalable
52. String Data String Data No iterator analogy. No iterator analogy. • Memory hog - less scalable Memory hog - less scalable
53. Stdlib Examples Stdlib Examples Module String Data File Filename json loads load pickle loads load xml.etree.Element Tree fromstring parse parse xml.dom.minidom parseString parse parse ConfigParser cp.readfp cp.read(filenames ) csv reader DictReader pyyaml 3rd party load, safe_load load, safe_load
54. Stdlib Take-aways Stdlib Take-aways • mostly functions mostly functions • file interface is required, others optional file interface is required, others optional • parse parse or or load load
55. Example Example >>> >>> import import sys sys >>> >>> def def parse parse(fin): (fin): ... ... for for line line in in upper(fin): upper(fin): ... ... sys sys. .stdout stdout. .write(line) write(line) >>> >>> def def upper upper(iterable): (iterable): ... ... for for item item in in iterable: iterable: ... ... yield yield str str(item) (item). .upper() upper()
56. Create file to parse Create file to parse >>> >>> with with open open( ('/tmp/data' '/tmp/data', , 'w' 'w') ) as as fout: fout: ... ... fout fout. .write( write('line1 'line1\n \n' ') ) ... ... fout fout. .write( write('line2 'line2\n \n' ') )
57. Filename to Filename to file file >>> >>> filename filename = = '/tmp/data' '/tmp/data' >>> >>> with with open open(filename) (filename) as as fin: fin: ... ... parse(fin) parse(fin) LINE1 LINE1 LINE2 LINE2
58. String data to String data to file file >>> >>> data data = = "string "string\n \ndata data\n \n" " >>> >>> import import StringIO StringIO >>> >>> parse(StringIO parse(StringIO. .StringIO(data)) StringIO(data)) STRING STRING DATA DATA
59. Parse Iterable Parse Iterable >>> >>> data data = = [ ['foo 'foo\n \n' ', , 'bar 'bar\n \n' '] ] >>> >>> parse(data) parse(data) FOO FOO BAR BAR
60. More More file file benefits benefits • Combine with generators to filter, tweak Combine with generators to filter, tweak • Easier to test Easier to test
61. Assignment Assignment Add a Add a parse parse function function
62. Invoking Shell Invoking Shell Commands Commands
63. Reading output Reading output >>> >>> import import subprocess subprocess >>> >>> p p = = subprocess subprocess. .Popen( Popen('id -u' 'id -u', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stderr PIPE, stderr= =subprocess subprocess. .PIPE) PIPE) >>> >>> p p. .stdout stdout. .read() read() '1000\n' '1000\n' >>> >>> p p. .returncode returncode # None means not done # None means not done >>> >>> print print p p. .wait() wait() 0 0
64. Feeding Feeding stdin stdin Can use Can use communicate communicate or or p2.stdin.write p2.stdin.write w/ w/ flush/close flush/close. . >>> >>> p2 p2 = = subprocess subprocess. .Popen( Popen('wc -l' 'wc -l', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, PIPE, stderr stderr= =subprocess subprocess. .PIPE) PIPE) >>> >>> out, err out, err = = p2 p2. .communicate( communicate('foo 'foo\n \nbar bar\n \n' ') ) #p.stdin.flush() #p.stdin.flush() >>> >>> out out '2\n' '2\n' >>> >>> p2 p2. .returncode returncode 0 0
65. Chaining scripts Chaining scripts Chaining is pretty straightforward make sure to Chaining is pretty straightforward make sure to close close stdin stdin. . http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subpr http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subpr ocess-pipeline ocess-pipeline >>> >>> p3 p3 = = subprocess subprocess. .Popen( Popen('sort' 'sort', shell , shell= =True True, , ... ... stdout stdout= =subprocess subprocess. .PIPE, PIPE, ... ... stdin stdin= =subprocess subprocess. .PIPE) PIPE) >>> >>> p4 p4 = = subprocess subprocess. .Popen( Popen('uniq' 'uniq', shell , shell= =True True, , ... ... stdout stdout= =subprocess subprocess. .PIPE, PIPE, ... ... stdin stdin= =p3 p3. .stdout, stdout, ... ... close_fds close_fds= =True True) ) # hangs w/o close_fds # hangs w/o close_fds >>> >>> p3 p3. .stdin stdin. .write( write('1 '1\n \n2 2\n \n1 1\n \n' ') ) >>> >>> p3 p3. .stdin stdin. .flush(); p3 flush(); p3. .stdin stdin. .close(); close(); >>> >>> p4 p4. .stdout stdout. .read() read() '1\n2\n' '1\n2\n'
66. Chaining scripts and python Chaining scripts and python cat cat 0-2, add 10 to them (in python) and 0-2, add 10 to them (in python) and wc wc -l -l results. results. >>> >>> import import os os >>> >>> p5 p5 = = subprocess subprocess. .Popen( Popen('cat' 'cat', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, close_fds PIPE, close_fds= =True True) ) >>> >>> def def p6 p6( (input input): ): ... ... ''' add 10 to line in input ''' ''' add 10 to line in input ''' ... ... for for line line in in input input: : ... ... yield yield ' '%d%s %d%s' ' % %( (int int(line (line. .strip()) strip())+10 +10, os , os. .linesep) linesep)
67. Chaining scripts and python Chaining scripts and python (2) (2) >>> >>> p7 p7 = = subprocess subprocess. .Popen( Popen('wc -l' 'wc -l', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, close_fds PIPE, close_fds= =True True) ) >>> >>> [p5 [p5. .stdin stdin. .write( write(str str(x) (x)+ +os os. .linesep) linesep) for for x x in in xrange xrange( (3 3)] )] >>> >>> p5 p5. .stdin stdin. .close() close() >>> >>> [p7 [p7. .stdin stdin. .write(x) write(x) for for x x in in p6(p5 p6(p5. .stdout stdout. .xreadlines())] xreadlines())] >>> >>> p7 p7. .stdin stdin. .close() close() >>> >>> p7 p7. .stdout stdout. .read() read() '3\n' '3\n'
68. Environment Environment
69. Python Environment Python Environment • System Python or “Virtual” Python System Python or “Virtual” Python • Installation method Installation method
70. Which Python Which Python System Python System Python • Requires root Requires root • Only one version of library Only one version of library • System packages may be out of date System packages may be out of date
71. Which Python (2) Which Python (2) “ “Virtual” Python Virtual” Python • Runs as user Runs as user • Specify version Specify version • Sandboxed from system Sandboxed from system • Create multiple sandboxes Create multiple sandboxes
72. Which Python (3) Which Python (3) Install Install virtualenv virtualenv
73. virtualenv virtualenv Installation: Installation: • System package System package $ sudo apt-get install $ sudo apt-get install python-virtualenv python-virtualenv • With pip With pip $ pip install virtualenv $ pip install virtualenv • $ easy_install virtualenv $ easy_install virtualenv • $ wget $ wget https://raw.github.com/pypa/virtualenv/master/ https://raw.github.com/pypa/virtualenv/master/ virtualenv.py; python virtualenv.py virtualenv.py; python virtualenv.py
74. virtualenv virtualenv (2) (2) Create virtual environment: Create virtual environment: $ virtualenv env_name $ virtualenv env_name
75. Directory Structure Directory Structure env_name/ env_name/ bin/ bin/ activate activate python python pip pip lib/ lib/ python2.7/ python2.7/ site-packages/ site-packages/
76. virtualenv virtualenv (3) (3) Activate virtual environment: Activate virtual environment: $ source env_name/bin/activate $ source env_name/bin/activate
77. virtualenv virtualenv (4) (4) Windows activate virtual environment: Windows activate virtual environment: > \path\to\env\Scripts\activate > \path\to\env\Scripts\activate May require May require PS C:\> PS C:\> Set-ExecutionPolicy AllSigned Set-ExecutionPolicy AllSigned
78. virtualenv virtualenv (5) (5) Comes with Comes with pip pip to install packages: to install packages: $ pip install sqlalchemy $ pip install sqlalchemy
79. virtualenv virtualenv (6) (6) $ which python # not /usr/bin/python $ which python # not /usr/bin/python /home/matt/work/courses/script-pypi-github/env/b /home/matt/work/courses/script-pypi-github/env/b in/python in/python
80. virtualenv virtualenv (7) (7) >>> sys.path >>> sys.path ['', ..., ['', ..., '/home/matt/work/courses/script-py '/home/matt/work/courses/script-py pi-github/env/lib/python2.7/site-p pi-github/env/lib/python2.7/site-p ackages'] ackages']
81. virtualenv virtualenv (8) (8) Use Use deactivate deactivate shell function to reset shell function to reset PATH PATH: : $ deactivate $ deactivate
82. Assignment Assignment create virtualenv create virtualenv env env
83. pip pip
84. pip pip Recursive Acronym - Recursive Acronym - P Pip ip I Installs nstalls P Packages ackages • install install • upgrade upgrade • uninstall uninstall • “ “pin” versions pin” versions • requirements.txt requirements.txt
85. pip pip (2) (2) Install: Install: $ pip install sqlalchemy $ pip install sqlalchemy
86. pip pip (3) (3) Upgrade: Upgrade: $ pip install --upgrade sqlalchemy $ pip install --upgrade sqlalchemy
87. pip pip (4) (4) Uninstall: Uninstall: $ pip uninstall sqlalchemy $ pip uninstall sqlalchemy
88. pip pip (5) (5) “ “Pin” version: Pin” version: $ pip install sqlalchemy==0.7 $ pip install sqlalchemy==0.7
89. pip pip (6) (6) Requirements file: Requirements file: $ pip install -r requirements.txt $ pip install -r requirements.txt
90. pip pip (7) (7) Requirements file Requirements file $ cat requirements.txt $ cat requirements.txt: : sqlalchemy==0.7 sqlalchemy==0.7 foobar foobar bazlib>=1.0 bazlib>=1.0
91. pip pip (8) (8) Create requirement file from env: Create requirement file from env: $ pip freeze > req.txt $ pip freeze > req.txt
92. pip pip (9) (9) • pip docs say to install from virtualenv pip docs say to install from virtualenv • virtualenv docs say to install from pip virtualenv docs say to install from pip
93. pip pip (10) (10) Install from directory: Install from directory: pip install --download packages -r pip install --download packages -r requirements.txt requirements.txt pip install --no-index pip install --no-index --find-links=file://full/path/to/p --find-links=file://full/path/to/p ackages -r requirements.txt ackages -r requirements.txt
94. distribute distribute, , setuptools setuptools, , distutils distutils • pip pip wraps wraps distribute distribute adds uninstall, adds uninstall, req.txt req.txt • distribute distribute fork of fork of setuptools setuptools • setuptools setuptools unmaintained, adds eggs, unmaintained, adds eggs, dependencies, dependencies, easy_install easy_install • distutils distutils stdlib packaging library stdlib packaging library
95. Assignment Assignment use pip to install use pip to install package from internet package from internet
96. More Layout More Layout
97. Better Example Layout Better Example Layout Project/ Project/ .gitignore .gitignore doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
98. .gitignore .gitignore *.py[cod] *.py[cod] # C extensions # C extensions *.so *.so # Packages # Packages *.egg *.egg *.egg-info *.egg-info dist dist build build eggs eggs parts parts bin bin var var sdist sdist
99. .gitignore .gitignore develop-eggs develop-eggs .installed.cfg .installed.cfg lib lib lib64 lib64 __pycache__ __pycache__ # Installer logs # Installer logs pip-log.txt pip-log.txt # Unit test / coverage reports # Unit test / coverage reports .coverage .coverage .tox .tox nosetests.xml nosetests.xml # Translations # Translations *.mo *.mo https://github.com/github/gitignore https://github.com/github/gitignore
100. .gitignore .gitignore (2) (2) See See blaze-core blaze-core for example of C and for example of C and Docs. Docs. https://github.com/ContinuumIO/blaze-cor https://github.com/ContinuumIO/blaze-cor e e
101. .gitignore .gitignore (3) (3) From From requests requests: : .coverage .coverage MANIFEST MANIFEST coverage.xml coverage.xml nosetests.xml nosetests.xml junit-report.xml junit-report.xml pylint.txt pylint.txt toy.py toy.py violations.pyflakes.txt violations.pyflakes.txt cover/ cover/ docs/_build docs/_build requests.egg-info/ requests.egg-info/ *.pyc *.pyc *.swp *.swp env/ env/ .workon .workon t.py t.py t2.py t2.py https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests
102. Other tips Other tips • localsettings.py localsettings.py (for django) (for django) • *~ *~ (emacs) (emacs) • Run Run git status git status and add outliers to and add outliers to .gitignore .gitignore • Make settings global: Make settings global: git config --global core.excludesfile git config --global core.excludesfile Python.gitignore Python.gitignore git config --global core.excludesfile git config --global core.excludesfile Python.gitignore Python.gitignore
103. Assignment Assignment add (premptive) add (premptive) .gitignore .gitignore
104. Documentation Documentation
105. Documentation Documentation Two types: Two types: • Developer docs (README, INSTALL, Developer docs (README, INSTALL, HACKING, etc) HACKING, etc) • End user End user
106. Developer Developer README README - main entry point for project - main entry point for project • Brief intro Brief intro • Links/Contact Links/Contact • License License
107. README README For github integration name it For github integration name it README.rst README.rst or or README.md README.md
108. LICENSE LICENSE Include text of license. Templates at Include text of license. Templates at http://opensource.org/licenses/index.html http://opensource.org/licenses/index.html
109. Licensing Licensing Some include dunder meta in project docstring ( Some include dunder meta in project docstring (requests requests __init__.py __init__.py): ): :copyright: (c) 2013 by Kenneth Reitz. :copyright: (c) 2013 by Kenneth Reitz. :license: Apache 2.0, see LICENSE for more :license: Apache 2.0, see LICENSE for more details. details. (note IANAL) (note IANAL)
110. Licensing (2) Licensing (2) Some include dunder meta in project ( Some include dunder meta in project (requests requests __init__.py __init__.py): ): __title__ = 'requests' __title__ = 'requests' __version__ = '1.1.0' __version__ = '1.1.0' __build__ = 0x010100 __build__ = 0x010100 __author__ = 'Kenneth Reitz' __author__ = 'Kenneth Reitz' __license__ = 'Apache 2.0' __license__ = 'Apache 2.0' __copyright__ = 'Copyright 2013 Kenneth Reitz' __copyright__ = 'Copyright 2013 Kenneth Reitz' (note IANAL) (note IANAL)
111. Other files Other files • AUTHORS AUTHORS • HISTORY/CHANGELOG HISTORY/CHANGELOG • TODO TODO
112. Assignment Assignment create simple create simple README README
113. End User Docs End User Docs Sphinx is a tool that makes it easy to create Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, intelligent and beautiful documentation, written by Georg Brandl and licensed written by Georg Brandl and licensed under the BSD license. under the BSD license. http://sphinx-doc.org http://sphinx-doc.org
114. Suggested setup Suggested setup Project/ Project/ doc/ doc/ # sphinx stuff # sphinx stuff Makefile Makefile
115. Sphinx in 4 Lines Sphinx in 4 Lines $ cd docs $ cd docs $ sphinx-quickstart $ sphinx-quickstart $ $EDITOR index.rst $ $EDITOR index.rst $ make html $ make html
116. Assignment Assignment write docs at a later write docs at a later time :) time :)
117. Makefile Makefile
118. Motivation Motivation Running commands often: Running commands often: • nosetests nosetests (plus options) (plus options) • create sdist create sdist • upload to PyPi upload to PyPi • create virtualenv create virtualenv • install dependencies install dependencies • cleanup cruft cleanup cruft • create TAGS create TAGS • profile profile • sdist sdist • PyPi - register and upload PyPi - register and upload • creating pngs from svgs creating pngs from svgs • docs docs • Python 3 testing Python 3 testing • etc... etc...
119. Makefile Makefile • Knows about executing (build) Knows about executing (build) commands commands • Knows about dependencies Knows about dependencies
120. Example Example To test: To test: • Make virtualenv Make virtualenv • install code dependencies install code dependencies • install nose (+coverage) install nose (+coverage) • run tests run tests
121. Clean checkout Clean checkout $ make test $ make test vs vs $ virtualenv env $ virtualenv env $ env/bin/activate $ env/bin/activate $ pip install -r deps.txt $ pip install -r deps.txt $ pip install nose coverage.py $ pip install nose coverage.py $ nosestests $ nosestests
122. Enough Enough make make knowledge to be knowledge to be dangerous dangerous
123. Makefile Makefile (1) (1) Syntax of Syntax of Makefile Makefile: : file: dependentfile file: dependentfile <TAB>Command1 <TAB>Command1 ... ... <TAB>CommandN <TAB>CommandN
124. Makefile Makefile (2) (2) Running (runs Running (runs Makefile Makefile by default): by default): $ make file $ make file # will build dependentfile if # will build dependentfile if necessary necessary # then build file # then build file
125. Makefile Makefile (3) (3) Example: Example: foo: foo.c foo.h foo: foo.c foo.h <TAB>cc -c foo.c <TAB>cc -c foo.c <TAB>cc -o foo foo.o <TAB>cc -o foo foo.o
126. Makefile Makefile (4) (4) Running (echoes commands by default Running (echoes commands by default -s -s for silent): for silent): $ make $ make cc -c foo.c cc -c foo.c cc -o foo foo.o cc -o foo foo.o
127. Makefile Makefile (5) (5) Subsequent runs do nothing: Subsequent runs do nothing: $ make $ make make: `foo' is up to date. make: `foo' is up to date.
128. Makefile Makefile (6) (6) Add a Add a clean clean command: command: .PHONY: clean .PHONY: clean clean: clean: <TAB>rm foo.o foo <TAB>rm foo.o foo
129. Makefile Makefile (7) (7) Since Since clean clean isn't a file, need to use isn't a file, need to use .PHONY .PHONY to indicate that to to indicate that to make make. (If you had a file . (If you had a file named named clean clean it wouldn't try to build it). it wouldn't try to build it).
130. Makefile Makefile (8) (8) (Simply Expanded) Variables (expanded when set): (Simply Expanded) Variables (expanded when set): BIN := env/bin BIN := env/bin PY := $(BIN)/python PY := $(BIN)/python NOSE := $(BIN)/nosetests NOSE := $(BIN)/nosetests .PHONY: build .PHONY: build build: env build: env <TAB>$(PY) setup.py sdist <TAB>$(PY) setup.py sdist
131. Makefile Makefile (9) (9) (Recursively Expanded) Variables (expanded when used): (Recursively Expanded) Variables (expanded when used): FILE = foo FILE = foo DATA = $(FILE) DATA = $(FILE) # If DATA expanded would be foo # If DATA expanded would be foo FILE = bar FILE = bar # If DATA expanded would be bar # If DATA expanded would be bar
132. Makefile Makefile (10) (10) Shell functions: Shell functions: .PHONY: pwd .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc
133. Makefile Makefile (11) (11) Invoking: Invoking: $ make pwd $ make pwd pushd /etc pushd /etc make: pushd: Command not found make: pushd: Command not found make: *** [pwd] Error 127 make: *** [pwd] Error 127 ( (pushd pushd is a bash function) is a bash function)
134. Makefile Makefile (12) (12) Shell functions: Shell functions: SHELL := /bin/bash SHELL := /bin/bash .PHONY: pwd .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc
135. Makefile Makefile (13) (13) Multiple commands: Multiple commands: SHELL := /bin/bash SHELL := /bin/bash .PHONY: pwd .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc <TAB>pwd <TAB>pwd <TAB>popd <TAB>popd
136. Makefile Makefile (14) (14) Multiple commands: Multiple commands: $ make pwd $ make pwd pushd /etc pushd /etc /etc /tmp/foo /etc /tmp/foo pwd pwd /tmp/foo /tmp/foo popd popd /bin/bash: line 0: popd: directory stack empty /bin/bash: line 0: popd: directory stack empty
137. Makefile Makefile (15) (15) Each tab indented command runs in its Each tab indented command runs in its own process. Use own process. Use ; ; and put in one line or and put in one line or use use \ \ for line continuation for line continuation
138. Makefile Makefile (16) (16) Multiple commands (use line continuation Multiple commands (use line continuation \ \): ): SHELL := /bin/bash SHELL := /bin/bash .PHONY: pwd2 .PHONY: pwd2 pwd2: pwd2: <TAB>pushd /etc; \ <TAB>pushd /etc; \ <TAB>pwd; \ <TAB>pwd; \ <TAB>popd <TAB>popd
139. Makefile Makefile (17) (17) Shell variables: Shell variables: .PHONY: path .PHONY: path path: path: <TAB>echo $PATH <TAB>echo $PATH
140. Makefile Makefile (18) (18) Make thinks they are make variables: Make thinks they are make variables: $ make path $ make path echo ATH echo ATH ATH ATH
141. Makefile Makefile (19) (19) $ $ needs to be escaped with needs to be escaped with $ $: : .PHONY: path .PHONY: path path: path: <TAB>echo $$PATH <TAB>echo $$PATH
142. Makefile Makefile (18) (18) Now it works: Now it works: $ make path $ make path echo $PATH echo $PATH /tmp/maketest /tmp/maketest
143. Makefiles for Python Makefiles for Python Projects Projects
144. Inspired by Rick Harding's Inspired by Rick Harding's Talk Talk http://pyvideo.org/video/1354/starting-you http://pyvideo.org/video/1354/starting-you r-project-right-setup-and-automation r-project-right-setup-and-automation https://github.com/mitechie/pyohio_2012 https://github.com/mitechie/pyohio_2012
145. Makefile Makefile for Python for Python Make virtualenv: Make virtualenv: env: env: <TAB>virtualenv env <TAB>virtualenv env
146. Makefile Makefile for Python (2) for Python (2) Make dependencies: Make dependencies: .PHONY: deps .PHONY: deps deps: env deps: env <TAB>$(PIP) install -r <TAB>$(PIP) install -r requirements.txt requirements.txt
147. Makefile Makefile for Python (3) for Python (3) Testing with Testing with nose nose: : .PHONY: test .PHONY: test test: nose deps test: nose deps <TAB>$(NOSE) <TAB>$(NOSE) # nose depends on the nosetests binary # nose depends on the nosetests binary nose: $(NOSE) nose: $(NOSE) $(NOSE): env $(NOSE): env <TAB>$(PIP) install nose <TAB>$(PIP) install nose
148. Contrary Opinions Contrary Opinions “ “Dynamic languages don't need anything Dynamic languages don't need anything like like make make, unless they have some , unless they have some compile-time interface dependencies compile-time interface dependencies between modules” between modules” http://stackoverflow.com/questions/758093 http://stackoverflow.com/questions/758093 9/why-are-there-no-makefiles-for-automati 9/why-are-there-no-makefiles-for-automati on-in-python-projects on-in-python-projects
149. Other options Other options • paver paver • fabric fabric • buildout buildout
150. Packaging Packaging
151. setup.py setup.py overloaded overloaded • create sdist (source distribution) create sdist (source distribution) • upload to pypi upload to pypi • install package install package
152. setup.py setup.py wart wart require require keyword of keyword of distutils distutils doesn't doesn't download reqs only download reqs only documents documents them. Use them. Use requirements.txt requirements.txt in combo with in combo with pip pip. .
153. setup.py setup.py example example From From requests requests: : setup( setup( name='requests', name='requests', version=requests.__version__, version=requests.__version__, description='Python HTTP for Humans.', description='Python HTTP for Humans.', long_description=open('README.rst').read() + long_description=open('README.rst').read() + '\n\n' + '\n\n' + open('HISTORY.rst').read(), open('HISTORY.rst').read(), author='Kenneth Reitz', author='Kenneth Reitz', author_email='me@kennethreitz.com', author_email='me@kennethreitz.com', url='http://python-requests.org', url='http://python-requests.org',
154. setup.py setup.py example (2) example (2) packages=packages, packages=packages, package_data={'': ['LICENSE', 'NOTICE'], package_data={'': ['LICENSE', 'NOTICE'], 'requests': ['*.pem']}, 'requests': ['*.pem']}, package_dir={'requests': 'requests'}, package_dir={'requests': 'requests'}, include_package_data=True, include_package_data=True, install_requires=requires, install_requires=requires, license=open('LICENSE').read(), license=open('LICENSE').read(), zip_safe=False, zip_safe=False,
155. setup.py setup.py example (3) example (3) classifiers=( classifiers=( 'Development Status :: 5 - Production/Stable', 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Intended Audience :: Developers', 'Natural Language :: English', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3', # 'Programming Language :: Python :: 3.0', # 'Programming Language :: Python :: 3.0', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.3', ), ), ) )
156. setup.py setup.py modules modules If project consists of a few modules this If project consists of a few modules this may be easiest may be easiest
157. setup.py setup.py packages packages Need to explicitly list Need to explicitly list all all packages, not just packages, not just root root
158. setup.py setup.py scripts scripts Add executable Python files here Add executable Python files here
159. setup.py setup.py non-Python files non-Python files • add files to add files to MANIFEST.in MANIFEST.in (include in (include in package) package) • add files to add files to package_data package_data in in setup.py setup.py (include in install) Not recursive (include in install) Not recursive
160. MANIFEST.in MANIFEST.in language language • include|exclude pat1 pat2 ... include|exclude pat1 pat2 ... • recursive-(include|exclude) dir pat1 pat2 ... recursive-(include|exclude) dir pat1 pat2 ... • global-(include|exclude) dir pat1 pat2 ... global-(include|exclude) dir pat1 pat2 ... • prune dir prune dir • graft dir graft dir http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-c http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-c md md
161. setup.py setup.py classifiers classifiers Almost 600 different classifiers. Almost 600 different classifiers. Not used by pip to enforce versions. For UI Not used by pip to enforce versions. For UI only only
162. Create sdist Create sdist $ python setup.py sdist $ python setup.py sdist
163. PyPi PyPi Validate Validate setup.py setup.py: : $ python setup.py check $ python setup.py check
164. PyPi Register PyPi Register Click on “Register” on right hand box Click on “Register” on right hand box https://pypi.python.org/pypi?%3Aaction=re https://pypi.python.org/pypi?%3Aaction=re gister_form gister_form
165. PyPi Upload PyPi Upload $ python setup.py sdist register $ python setup.py sdist register upload upload
166. PyPi Upload (2) PyPi Upload (2) $ python setup.py sdist register upload $ python setup.py sdist register upload ... ... Creating tar archive Creating tar archive removing 'rst2odp-0.2.4' (and everything under it) removing 'rst2odp-0.2.4' (and everything under it) running register running register running check running check We need to know who you are, so please choose either: We need to know who you are, so please choose either: 1. use your existing login, 1. use your existing login, 2. register as a new user, 2. register as a new user, 3. have the server generate a new password for you (and email it to you), 3. have the server generate a new password for you (and email it to you), or or 4. quit 4. quit Your selection [default 1]: Your selection [default 1]: 1 1
167. PyPi Upload (3) PyPi Upload (3) Username: mharrison Username: mharrison Password: Password: Registering rst2odp to http://pypi.python.org/pypi Registering rst2odp to http://pypi.python.org/pypi Server response (200): OK Server response (200): OK I can store your PyPI login so future submissions will be faster. I can store your PyPI login so future submissions will be faster. (the login will be stored in /home/matt/.pypirc) (the login will be stored in /home/matt/.pypirc) Save your login (y/N)?y Save your login (y/N)?y running upload running upload Submitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypi Submitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypi Server response (200): OK Server response (200): OK
168. PyPi Note PyPi Note Though PyPi packages are signed there is Though PyPi packages are signed there is no verification step during package no verification step during package installation installation
169. Non PyPi URL Non PyPi URL $ pip install --no-index -f $ pip install --no-index -f http://dist.plone.org/thirdparty/ http://dist.plone.org/thirdparty/ -U PIL -U PIL
170. Personal PyPi Personal PyPi https://github.com/benliles/djangopypi https://github.com/benliles/djangopypi
171. PyPi PyPi Makefile Makefile integration: integration: # --------- PyPi ---------- # --------- PyPi ---------- .PHONY: build .PHONY: build build: env build: env <TAB>$(PY) setup.py sdist <TAB>$(PY) setup.py sdist .PHONY: upload .PHONY: upload upload: env upload: env <TAB>$(PY) setup.py sdist register upload <TAB>$(PY) setup.py sdist register upload
172. Testing Testing
173. Testing Testing Add tests to your project Add tests to your project
174. Testing (2) Testing (2) • use use doctest doctest or or unittest unittest
175. Testing (3) Testing (3) Use Use nose nose to run: to run: $ env/bin/nosetests $ env/bin/nosetests .. .. ------------- ------------- Ran 2 tests in 0.007s Ran 2 tests in 0.007s OK OK
176. Testing (4) Testing (4) Makefile Makefile integration: integration: NOSE := env/bin/nosetests NOSE := env/bin/nosetests # --------- Testing ---------- # --------- Testing ---------- .PHONY: test .PHONY: test test: nose deps test: nose deps <TAB>$(NOSE) <TAB>$(NOSE) # nose depends on the nosetests binary # nose depends on the nosetests binary nose: $(NOSE) nose: $(NOSE) $(NOSE): env $(NOSE): env <TAB>$(PIP) install nose <TAB>$(PIP) install nose
177. Github Github
178. Github Github Just a satisfied user Just a satisfied user
179. Why Github? Why Github? Not bazaar nor mercurial Not bazaar nor mercurial
180. Why Github? (2) Why Github? (2) Code is a first class object Code is a first class object
181. Github Github Don't check in keys/passwords! Don't check in keys/passwords! http://ejohn.org/blog/keeping-passwords-i http://ejohn.org/blog/keeping-passwords-i n-source-control/#postcomment n-source-control/#postcomment
182. A Branching Strategy A Branching Strategy http://github.com/nvie/gitflow http://github.com/nvie/gitflow
183. Travis CI Travis CI
184. Travis CI Travis CI CI (continuous integration) for Github CI (continuous integration) for Github
185. Travis CI (2) Travis CI (2) Illustrates power of (web) hooks Illustrates power of (web) hooks
186. 5 Steps 5 Steps • Sign-in with github Sign-in with github • Sync repos on Profile page Sync repos on Profile page • Enable repo Enable repo • Create a Create a .travis.yml .travis.yml file on github file on github • Push a commit on github Push a commit on github
187. travis.yml travis.yml language: python language: python python: python: - "3.3" - "3.3" - "2.7" - "2.7" - "2.6" - "2.6" - "pypy" - "pypy" # command to run tests # command to run tests script: make test script: make test
188. More Info More Info http://about.travis-ci.org/ http://about.travis-ci.org/
189. Poachplate Poachplate
190. poachplate poachplate Example of most of what we have talked Example of most of what we have talked about today about today https://github.com/mattharrison/poachplat https://github.com/mattharrison/poachplat e e
191. That's all That's all Questions? Tweet or email me Questions? Tweet or email me matthewharrison@gmail.com matthewharrison@gmail.com @__mharrison__ @__mharrison__ http://hairysun.com http://hairysun.com