Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PyCon 2013 Tutorial: Script to PyPI to GitHub & More

PyCon 2013 Tutorial: Script to PyPI to GitHub & More

How do you start a new project? How do you deliver a script to co-workers? How do you develop it with best practices? How do you use virtualenv and pip? How do you package it? How do you automate testing, building, uploading to PyPI?

matt harrison

March 14, 2013
Tweet

Other Decks in Technology

Transcript

  1. Script to PyPi to Github Script to PyPi to Github

    @__mharrison__ @__mharrison__ http://hairysun.com http://hairysun.com
  2. About Me About Me • 12 years Python 12 years

    Python • Worked in HA, Search, Open Source, BI Worked in HA, Search, Open Source, BI and Storage and Storage • Author of multiple Python Books Author of multiple Python Books
  3. Agenda Agenda • Project Development Project Development – Versioning Versioning

    – Configuration Configuration – Logging Logging – File input File input – Shell invocation Shell invocation • Environment Layout * virtualenv * pip Environment Layout * virtualenv * pip • Project layout Project layout
  4. Agenda (2) Agenda (2) • Documentation Documentation • Automation/Makefile Automation/Makefile

    • Packaging Packaging – setup.py setup.py – PyPi PyPi • Testing Testing • Github Github • Travis CI Travis CI • Poachplate Poachplate
  5. Warning Warning • Starting from basic Python knowledge Starting from

    basic Python knowledge • Hands on Hands on – (short) lecture (short) lecture – (short) code (short) code – repeat until time is gone repeat until time is gone
  6. hello world hello world cat.py cat.py import import sys sys

    for for line line in in sys sys. .stdin: stdin: print print line, line,
  7. hello world - Python 3 hello world - Python 3

    import import sys sys for for line line in in sys sys. .stdin: stdin: print print(line, end (line, end= ='' '') )
  8. 2 or 3? 2 or 3? • Better legacy/library support

    for 2 Better legacy/library support for 2 • Possible to support both Possible to support both
  9. hello world hello world import import sys sys for for

    line line in in sys sys. .stdin: stdin: sys sys. .stdout stdout. .write(line) write(line)
  10. Layout Layout Can depend on distribution mechanism: Can depend on

    distribution mechanism: • Single file Single file • Zip file Zip file – Python entry point Python entry point – Splat/run Splat/run • System package System package • PyPi/Distribute/pip package PyPi/Distribute/pip package
  11. Single File Single File • chmod chmod and place in

    and place in $PATH $PATH • add add #!/usr/bin/env python #!/usr/bin/env python
  12. Zip File Zip File • PYTHONPATH=cat.zip python -m PYTHONPATH=cat.zip python

    -m __main__ __main__ • tar -zxvf cat.zip; cd cat; tar -zxvf cat.zip; cd cat; python cat.py python cat.py
  13. Zip File (2) Zip File (2) • No reuse No

    reuse • No stacktrace No stacktrace
  14. System Package (2) System Package (2) • Requires root Requires

    root • At mercy of packager (maybe worse than At mercy of packager (maybe worse than PyPi) PyPi) • Reuse Reuse • Limited to single version Limited to single version • python -m modulename python -m modulename
  15. Pip Package Pip Package “ “Best practice” is combo of:

    Best practice” is combo of: • Distribute Distribute • Virtualenv Virtualenv • Pip Pip
  16. Pip Package (2) Pip Package (2) $ virtualenv catenv $

    virtualenv catenv $ source catenv/bin/activate $ source catenv/bin/activate $ pip install pycat $ pip install pycat
  17. Pip Package (3) Pip Package (3) • Multiple versions Multiple

    versions • Reuse Reuse • Effort to create Effort to create setup.py setup.py
  18. Minimal Example Layout Minimal Example Layout Project/ Project/ README.txt README.txt

    project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
  19. Better Example Layout Better Example Layout Project/ Project/ .gitignore .gitignore

    doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
  20. setup.py setup.py from from distutils.core distutils.core import import setup setup

    setup(name setup(name= ='PyCat' 'PyCat', ,.... .... version version= ='1.0' '1.0') )
  21. Where to store version? Where to store version? In In

    module.__version__ module.__version__ (might cause (might cause importing issues) importing issues)
  22. Version Version If using Sphinx for docs, be sure to

    update: If using Sphinx for docs, be sure to update: docs/ docs/ conf.py conf.py
  23. argparse argparse ap ap = = argparse argparse. .ArgumentParser(version ArgumentParser(version=

    ='1.0' '1.0') ) ... ... ap ap. .print_version() print_version()
  24. Configuration Configuration • Python Python – Django ( Django (settings.py

    settings.py) ) – Python ( Python (site.py site.py, , setup.py setup.py) ) • JSON/YAML JSON/YAML – Google App Engine Google App Engine • Environment Variables Environment Variables – Python ( Python (PYTHONPATH PYTHONPATH) ) • .ini ( .ini (ConfigParser ConfigParser, , ConfigObj ConfigObj) ) – matplotlib matplotlib – ipython ipython • Command line options ( Command line options (argparse argparse) ) • Sqlite blob Sqlite blob • Shelve/pickle blob Shelve/pickle blob
  25. Configuration (3) Configuration (3) (Unix) Hierarchy: (Unix) Hierarchy: • System

    rc (run control) ( System rc (run control) (/etc/conf.d /etc/conf.d) ) • User rc ( User rc (~/.config/app/... ~/.config/app/...) ) • Environment variables Environment variables • Command line options Command line options http://www.faqs.org/docs/artu/ch10s02.html http://www.faqs.org/docs/artu/ch10s02.html Filesystem Hierarchy Standard: Filesystem Hierarchy Standard: http://www.pathname.com/fhs/ http://www.pathname.com/fhs/
  26. Configuration (4) Configuration (4) • Plain text config is easily

    approachable Plain text config is easily approachable • Careful with Python config on process Careful with Python config on process run by root run by root
  27. Logging (2) Logging (2) import import logging logging logging logging.

    .basicConfig(level basicConfig(level= =logging logging. .ERROR, ERROR, filename filename= ='.log' '.log') ) ... ... logging logging. .error( error('Error encountered in...' 'Error encountered in...') )
  28. Assignment Assignment Add configuration Add configuration --verbose --verbose to log

    file to log file being “catted” being “catted”
  29. “ “Files” Files” Dealing with? Dealing with? • Filename Filename

    • file object file object • string data string data
  30. Filename Filename Somewhat analogous to an Somewhat analogous to an

    Iterable Iterable. . • Can open/iterate many times Can open/iterate many times • Implementation depends on file Implementation depends on file • Need to manage closing file Need to manage closing file
  31. File Object File Object Somewhat analogous to an Somewhat analogous

    to an Iterator Iterator. . • Can iterate once (unless Can iterate once (unless seek seeked) ed) • Can accept Can accept file file, , StringIO StringIO, , socket socket, , generator, etc generator, etc • Memory friendly - scalable Memory friendly - scalable
  32. String Data String Data No iterator analogy. No iterator analogy.

    • Memory hog - less scalable Memory hog - less scalable
  33. Stdlib Examples Stdlib Examples Module String Data File Filename json

    loads load pickle loads load xml.etree.Element Tree fromstring parse parse xml.dom.minidom parseString parse parse ConfigParser cp.readfp cp.read(filenames ) csv reader DictReader pyyaml 3rd party load, safe_load load, safe_load
  34. Stdlib Take-aways Stdlib Take-aways • mostly functions mostly functions •

    file interface is required, others optional file interface is required, others optional • parse parse or or load load
  35. Example Example >>> >>> import import sys sys >>> >>>

    def def parse parse(fin): (fin): ... ... for for line line in in upper(fin): upper(fin): ... ... sys sys. .stdout stdout. .write(line) write(line) >>> >>> def def upper upper(iterable): (iterable): ... ... for for item item in in iterable: iterable: ... ... yield yield str str(item) (item). .upper() upper()
  36. Create file to parse Create file to parse >>> >>>

    with with open open( ('/tmp/data' '/tmp/data', , 'w' 'w') ) as as fout: fout: ... ... fout fout. .write( write('line1 'line1\n \n' ') ) ... ... fout fout. .write( write('line2 'line2\n \n' ') )
  37. Filename to Filename to file file >>> >>> filename filename

    = = '/tmp/data' '/tmp/data' >>> >>> with with open open(filename) (filename) as as fin: fin: ... ... parse(fin) parse(fin) LINE1 LINE1 LINE2 LINE2
  38. String data to String data to file file >>> >>>

    data data = = "string "string\n \ndata data\n \n" " >>> >>> import import StringIO StringIO >>> >>> parse(StringIO parse(StringIO. .StringIO(data)) StringIO(data)) STRING STRING DATA DATA
  39. Parse Iterable Parse Iterable >>> >>> data data = =

    [ ['foo 'foo\n \n' ', , 'bar 'bar\n \n' '] ] >>> >>> parse(data) parse(data) FOO FOO BAR BAR
  40. More More file file benefits benefits • Combine with generators

    to filter, tweak Combine with generators to filter, tweak • Easier to test Easier to test
  41. Reading output Reading output >>> >>> import import subprocess subprocess

    >>> >>> p p = = subprocess subprocess. .Popen( Popen('id -u' 'id -u', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stderr PIPE, stderr= =subprocess subprocess. .PIPE) PIPE) >>> >>> p p. .stdout stdout. .read() read() '1000\n' '1000\n' >>> >>> p p. .returncode returncode # None means not done # None means not done >>> >>> print print p p. .wait() wait() 0 0
  42. Feeding Feeding stdin stdin Can use Can use communicate communicate

    or or p2.stdin.write p2.stdin.write w/ w/ flush/close flush/close. . >>> >>> p2 p2 = = subprocess subprocess. .Popen( Popen('wc -l' 'wc -l', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, PIPE, stderr stderr= =subprocess subprocess. .PIPE) PIPE) >>> >>> out, err out, err = = p2 p2. .communicate( communicate('foo 'foo\n \nbar bar\n \n' ') ) #p.stdin.flush() #p.stdin.flush() >>> >>> out out '2\n' '2\n' >>> >>> p2 p2. .returncode returncode 0 0
  43. Chaining scripts Chaining scripts Chaining is pretty straightforward make sure

    to Chaining is pretty straightforward make sure to close close stdin stdin. . http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subpr http://stackoverflow.com/questions/1595492/blocks-send-input-to-python-subpr ocess-pipeline ocess-pipeline >>> >>> p3 p3 = = subprocess subprocess. .Popen( Popen('sort' 'sort', shell , shell= =True True, , ... ... stdout stdout= =subprocess subprocess. .PIPE, PIPE, ... ... stdin stdin= =subprocess subprocess. .PIPE) PIPE) >>> >>> p4 p4 = = subprocess subprocess. .Popen( Popen('uniq' 'uniq', shell , shell= =True True, , ... ... stdout stdout= =subprocess subprocess. .PIPE, PIPE, ... ... stdin stdin= =p3 p3. .stdout, stdout, ... ... close_fds close_fds= =True True) ) # hangs w/o close_fds # hangs w/o close_fds >>> >>> p3 p3. .stdin stdin. .write( write('1 '1\n \n2 2\n \n1 1\n \n' ') ) >>> >>> p3 p3. .stdin stdin. .flush(); p3 flush(); p3. .stdin stdin. .close(); close(); >>> >>> p4 p4. .stdout stdout. .read() read() '1\n2\n' '1\n2\n'
  44. Chaining scripts and python Chaining scripts and python cat cat

    0-2, add 10 to them (in python) and 0-2, add 10 to them (in python) and wc wc -l -l results. results. >>> >>> import import os os >>> >>> p5 p5 = = subprocess subprocess. .Popen( Popen('cat' 'cat', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, close_fds PIPE, close_fds= =True True) ) >>> >>> def def p6 p6( (input input): ): ... ... ''' add 10 to line in input ''' ''' add 10 to line in input ''' ... ... for for line line in in input input: : ... ... yield yield ' '%d%s %d%s' ' % %( (int int(line (line. .strip()) strip())+10 +10, os , os. .linesep) linesep)
  45. Chaining scripts and python Chaining scripts and python (2) (2)

    >>> >>> p7 p7 = = subprocess subprocess. .Popen( Popen('wc -l' 'wc -l', shell , shell= =True True, , stdout stdout= =subprocess subprocess. .PIPE, stdin PIPE, stdin= =subprocess subprocess. .PIPE, close_fds PIPE, close_fds= =True True) ) >>> >>> [p5 [p5. .stdin stdin. .write( write(str str(x) (x)+ +os os. .linesep) linesep) for for x x in in xrange xrange( (3 3)] )] >>> >>> p5 p5. .stdin stdin. .close() close() >>> >>> [p7 [p7. .stdin stdin. .write(x) write(x) for for x x in in p6(p5 p6(p5. .stdout stdout. .xreadlines())] xreadlines())] >>> >>> p7 p7. .stdin stdin. .close() close() >>> >>> p7 p7. .stdout stdout. .read() read() '3\n' '3\n'
  46. Python Environment Python Environment • System Python or “Virtual” Python

    System Python or “Virtual” Python • Installation method Installation method
  47. Which Python Which Python System Python System Python • Requires

    root Requires root • Only one version of library Only one version of library • System packages may be out of date System packages may be out of date
  48. Which Python (2) Which Python (2) “ “Virtual” Python Virtual”

    Python • Runs as user Runs as user • Specify version Specify version • Sandboxed from system Sandboxed from system • Create multiple sandboxes Create multiple sandboxes
  49. virtualenv virtualenv Installation: Installation: • System package System package $

    sudo apt-get install $ sudo apt-get install python-virtualenv python-virtualenv • With pip With pip $ pip install virtualenv $ pip install virtualenv • $ easy_install virtualenv $ easy_install virtualenv • $ wget $ wget https://raw.github.com/pypa/virtualenv/master/ https://raw.github.com/pypa/virtualenv/master/ virtualenv.py; python virtualenv.py virtualenv.py; python virtualenv.py
  50. Directory Structure Directory Structure env_name/ env_name/ bin/ bin/ activate activate

    python python pip pip lib/ lib/ python2.7/ python2.7/ site-packages/ site-packages/
  51. virtualenv virtualenv (3) (3) Activate virtual environment: Activate virtual environment:

    $ source env_name/bin/activate $ source env_name/bin/activate
  52. virtualenv virtualenv (4) (4) Windows activate virtual environment: Windows activate

    virtual environment: > \path\to\env\Scripts\activate > \path\to\env\Scripts\activate May require May require PS C:\> PS C:\> Set-ExecutionPolicy AllSigned Set-ExecutionPolicy AllSigned
  53. virtualenv virtualenv (5) (5) Comes with Comes with pip pip

    to install packages: to install packages: $ pip install sqlalchemy $ pip install sqlalchemy
  54. virtualenv virtualenv (6) (6) $ which python # not /usr/bin/python

    $ which python # not /usr/bin/python /home/matt/work/courses/script-pypi-github/env/b /home/matt/work/courses/script-pypi-github/env/b in/python in/python
  55. virtualenv virtualenv (7) (7) >>> sys.path >>> sys.path ['', ...,

    ['', ..., '/home/matt/work/courses/script-py '/home/matt/work/courses/script-py pi-github/env/lib/python2.7/site-p pi-github/env/lib/python2.7/site-p ackages'] ackages']
  56. virtualenv virtualenv (8) (8) Use Use deactivate deactivate shell function

    to reset shell function to reset PATH PATH: : $ deactivate $ deactivate
  57. pip pip Recursive Acronym - Recursive Acronym - P Pip

    ip I Installs nstalls P Packages ackages • install install • upgrade upgrade • uninstall uninstall • “ “pin” versions pin” versions • requirements.txt requirements.txt
  58. pip pip (3) (3) Upgrade: Upgrade: $ pip install --upgrade

    sqlalchemy $ pip install --upgrade sqlalchemy
  59. pip pip (5) (5) “ “Pin” version: Pin” version: $

    pip install sqlalchemy==0.7 $ pip install sqlalchemy==0.7
  60. pip pip (6) (6) Requirements file: Requirements file: $ pip

    install -r requirements.txt $ pip install -r requirements.txt
  61. pip pip (7) (7) Requirements file Requirements file $ cat

    requirements.txt $ cat requirements.txt: : sqlalchemy==0.7 sqlalchemy==0.7 foobar foobar bazlib>=1.0 bazlib>=1.0
  62. pip pip (8) (8) Create requirement file from env: Create

    requirement file from env: $ pip freeze > req.txt $ pip freeze > req.txt
  63. pip pip (9) (9) • pip docs say to install

    from virtualenv pip docs say to install from virtualenv • virtualenv docs say to install from pip virtualenv docs say to install from pip
  64. pip pip (10) (10) Install from directory: Install from directory:

    pip install --download packages -r pip install --download packages -r requirements.txt requirements.txt pip install --no-index pip install --no-index --find-links=file://full/path/to/p --find-links=file://full/path/to/p ackages -r requirements.txt ackages -r requirements.txt
  65. distribute distribute, , setuptools setuptools, , distutils distutils • pip

    pip wraps wraps distribute distribute adds uninstall, adds uninstall, req.txt req.txt • distribute distribute fork of fork of setuptools setuptools • setuptools setuptools unmaintained, adds eggs, unmaintained, adds eggs, dependencies, dependencies, easy_install easy_install • distutils distutils stdlib packaging library stdlib packaging library
  66. Assignment Assignment use pip to install use pip to install

    package from internet package from internet
  67. Better Example Layout Better Example Layout Project/ Project/ .gitignore .gitignore

    doc/ doc/ Makefile Makefile index.rst index.rst README.txt README.txt Makefile Makefile bin/ bin/ runfoo.py runfoo.py project/ project/ __init__.py __init__.py other.py other.py ... ... setup.py setup.py
  68. .gitignore .gitignore *.py[cod] *.py[cod] # C extensions # C extensions

    *.so *.so # Packages # Packages *.egg *.egg *.egg-info *.egg-info dist dist build build eggs eggs parts parts bin bin var var sdist sdist
  69. .gitignore .gitignore develop-eggs develop-eggs .installed.cfg .installed.cfg lib lib lib64 lib64

    __pycache__ __pycache__ # Installer logs # Installer logs pip-log.txt pip-log.txt # Unit test / coverage reports # Unit test / coverage reports .coverage .coverage .tox .tox nosetests.xml nosetests.xml # Translations # Translations *.mo *.mo https://github.com/github/gitignore https://github.com/github/gitignore
  70. .gitignore .gitignore (2) (2) See See blaze-core blaze-core for example

    of C and for example of C and Docs. Docs. https://github.com/ContinuumIO/blaze-cor https://github.com/ContinuumIO/blaze-cor e e
  71. .gitignore .gitignore (3) (3) From From requests requests: : .coverage

    .coverage MANIFEST MANIFEST coverage.xml coverage.xml nosetests.xml nosetests.xml junit-report.xml junit-report.xml pylint.txt pylint.txt toy.py toy.py violations.pyflakes.txt violations.pyflakes.txt cover/ cover/ docs/_build docs/_build requests.egg-info/ requests.egg-info/ *.pyc *.pyc *.swp *.swp env/ env/ .workon .workon t.py t.py t2.py t2.py https://github.com/kennethreitz/requests https://github.com/kennethreitz/requests
  72. Other tips Other tips • localsettings.py localsettings.py (for django) (for

    django) • *~ *~ (emacs) (emacs) • Run Run git status git status and add outliers to and add outliers to .gitignore .gitignore • Make settings global: Make settings global: git config --global core.excludesfile git config --global core.excludesfile Python.gitignore Python.gitignore git config --global core.excludesfile git config --global core.excludesfile Python.gitignore Python.gitignore
  73. Documentation Documentation Two types: Two types: • Developer docs (README,

    INSTALL, Developer docs (README, INSTALL, HACKING, etc) HACKING, etc) • End user End user
  74. Developer Developer README README - main entry point for project

    - main entry point for project • Brief intro Brief intro • Links/Contact Links/Contact • License License
  75. README README For github integration name it For github integration

    name it README.rst README.rst or or README.md README.md
  76. LICENSE LICENSE Include text of license. Templates at Include text

    of license. Templates at http://opensource.org/licenses/index.html http://opensource.org/licenses/index.html
  77. Licensing Licensing Some include dunder meta in project docstring (

    Some include dunder meta in project docstring (requests requests __init__.py __init__.py): ): :copyright: (c) 2013 by Kenneth Reitz. :copyright: (c) 2013 by Kenneth Reitz. :license: Apache 2.0, see LICENSE for more :license: Apache 2.0, see LICENSE for more details. details. (note IANAL) (note IANAL)
  78. Licensing (2) Licensing (2) Some include dunder meta in project

    ( Some include dunder meta in project (requests requests __init__.py __init__.py): ): __title__ = 'requests' __title__ = 'requests' __version__ = '1.1.0' __version__ = '1.1.0' __build__ = 0x010100 __build__ = 0x010100 __author__ = 'Kenneth Reitz' __author__ = 'Kenneth Reitz' __license__ = 'Apache 2.0' __license__ = 'Apache 2.0' __copyright__ = 'Copyright 2013 Kenneth Reitz' __copyright__ = 'Copyright 2013 Kenneth Reitz' (note IANAL) (note IANAL)
  79. End User Docs End User Docs Sphinx is a tool

    that makes it easy to create Sphinx is a tool that makes it easy to create intelligent and beautiful documentation, intelligent and beautiful documentation, written by Georg Brandl and licensed written by Georg Brandl and licensed under the BSD license. under the BSD license. http://sphinx-doc.org http://sphinx-doc.org
  80. Sphinx in 4 Lines Sphinx in 4 Lines $ cd

    docs $ cd docs $ sphinx-quickstart $ sphinx-quickstart $ $EDITOR index.rst $ $EDITOR index.rst $ make html $ make html
  81. Motivation Motivation Running commands often: Running commands often: • nosetests

    nosetests (plus options) (plus options) • create sdist create sdist • upload to PyPi upload to PyPi • create virtualenv create virtualenv • install dependencies install dependencies • cleanup cruft cleanup cruft • create TAGS create TAGS • profile profile • sdist sdist • PyPi - register and upload PyPi - register and upload • creating pngs from svgs creating pngs from svgs • docs docs • Python 3 testing Python 3 testing • etc... etc...
  82. Makefile Makefile • Knows about executing (build) Knows about executing

    (build) commands commands • Knows about dependencies Knows about dependencies
  83. Example Example To test: To test: • Make virtualenv Make

    virtualenv • install code dependencies install code dependencies • install nose (+coverage) install nose (+coverage) • run tests run tests
  84. Clean checkout Clean checkout $ make test $ make test

    vs vs $ virtualenv env $ virtualenv env $ env/bin/activate $ env/bin/activate $ pip install -r deps.txt $ pip install -r deps.txt $ pip install nose coverage.py $ pip install nose coverage.py $ nosestests $ nosestests
  85. Makefile Makefile (1) (1) Syntax of Syntax of Makefile Makefile:

    : file: dependentfile file: dependentfile <TAB>Command1 <TAB>Command1 ... ... <TAB>CommandN <TAB>CommandN
  86. Makefile Makefile (2) (2) Running (runs Running (runs Makefile Makefile

    by default): by default): $ make file $ make file # will build dependentfile if # will build dependentfile if necessary necessary # then build file # then build file
  87. Makefile Makefile (3) (3) Example: Example: foo: foo.c foo.h foo:

    foo.c foo.h <TAB>cc -c foo.c <TAB>cc -c foo.c <TAB>cc -o foo foo.o <TAB>cc -o foo foo.o
  88. Makefile Makefile (4) (4) Running (echoes commands by default Running

    (echoes commands by default -s -s for silent): for silent): $ make $ make cc -c foo.c cc -c foo.c cc -o foo foo.o cc -o foo foo.o
  89. Makefile Makefile (5) (5) Subsequent runs do nothing: Subsequent runs

    do nothing: $ make $ make make: `foo' is up to date. make: `foo' is up to date.
  90. Makefile Makefile (6) (6) Add a Add a clean clean

    command: command: .PHONY: clean .PHONY: clean clean: clean: <TAB>rm foo.o foo <TAB>rm foo.o foo
  91. Makefile Makefile (7) (7) Since Since clean clean isn't a

    file, need to use isn't a file, need to use .PHONY .PHONY to indicate that to to indicate that to make make. (If you had a file . (If you had a file named named clean clean it wouldn't try to build it). it wouldn't try to build it).
  92. Makefile Makefile (8) (8) (Simply Expanded) Variables (expanded when set):

    (Simply Expanded) Variables (expanded when set): BIN := env/bin BIN := env/bin PY := $(BIN)/python PY := $(BIN)/python NOSE := $(BIN)/nosetests NOSE := $(BIN)/nosetests .PHONY: build .PHONY: build build: env build: env <TAB>$(PY) setup.py sdist <TAB>$(PY) setup.py sdist
  93. Makefile Makefile (9) (9) (Recursively Expanded) Variables (expanded when used):

    (Recursively Expanded) Variables (expanded when used): FILE = foo FILE = foo DATA = $(FILE) DATA = $(FILE) # If DATA expanded would be foo # If DATA expanded would be foo FILE = bar FILE = bar # If DATA expanded would be bar # If DATA expanded would be bar
  94. Makefile Makefile (10) (10) Shell functions: Shell functions: .PHONY: pwd

    .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc
  95. Makefile Makefile (11) (11) Invoking: Invoking: $ make pwd $

    make pwd pushd /etc pushd /etc make: pushd: Command not found make: pushd: Command not found make: *** [pwd] Error 127 make: *** [pwd] Error 127 ( (pushd pushd is a bash function) is a bash function)
  96. Makefile Makefile (12) (12) Shell functions: Shell functions: SHELL :=

    /bin/bash SHELL := /bin/bash .PHONY: pwd .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc
  97. Makefile Makefile (13) (13) Multiple commands: Multiple commands: SHELL :=

    /bin/bash SHELL := /bin/bash .PHONY: pwd .PHONY: pwd pwd: pwd: <TAB>pushd /etc <TAB>pushd /etc <TAB>pwd <TAB>pwd <TAB>popd <TAB>popd
  98. Makefile Makefile (14) (14) Multiple commands: Multiple commands: $ make

    pwd $ make pwd pushd /etc pushd /etc /etc /tmp/foo /etc /tmp/foo pwd pwd /tmp/foo /tmp/foo popd popd /bin/bash: line 0: popd: directory stack empty /bin/bash: line 0: popd: directory stack empty
  99. Makefile Makefile (15) (15) Each tab indented command runs in

    its Each tab indented command runs in its own process. Use own process. Use ; ; and put in one line or and put in one line or use use \ \ for line continuation for line continuation
  100. Makefile Makefile (16) (16) Multiple commands (use line continuation Multiple

    commands (use line continuation \ \): ): SHELL := /bin/bash SHELL := /bin/bash .PHONY: pwd2 .PHONY: pwd2 pwd2: pwd2: <TAB>pushd /etc; \ <TAB>pushd /etc; \ <TAB>pwd; \ <TAB>pwd; \ <TAB>popd <TAB>popd
  101. Makefile Makefile (17) (17) Shell variables: Shell variables: .PHONY: path

    .PHONY: path path: path: <TAB>echo $PATH <TAB>echo $PATH
  102. Makefile Makefile (18) (18) Make thinks they are make variables:

    Make thinks they are make variables: $ make path $ make path echo ATH echo ATH ATH ATH
  103. Makefile Makefile (19) (19) $ $ needs to be escaped

    with needs to be escaped with $ $: : .PHONY: path .PHONY: path path: path: <TAB>echo $$PATH <TAB>echo $$PATH
  104. Makefile Makefile (18) (18) Now it works: Now it works:

    $ make path $ make path echo $PATH echo $PATH /tmp/maketest /tmp/maketest
  105. Inspired by Rick Harding's Inspired by Rick Harding's Talk Talk

    http://pyvideo.org/video/1354/starting-you http://pyvideo.org/video/1354/starting-you r-project-right-setup-and-automation r-project-right-setup-and-automation https://github.com/mitechie/pyohio_2012 https://github.com/mitechie/pyohio_2012
  106. Makefile Makefile for Python for Python Make virtualenv: Make virtualenv:

    env: env: <TAB>virtualenv env <TAB>virtualenv env
  107. Makefile Makefile for Python (2) for Python (2) Make dependencies:

    Make dependencies: .PHONY: deps .PHONY: deps deps: env deps: env <TAB>$(PIP) install -r <TAB>$(PIP) install -r requirements.txt requirements.txt
  108. Makefile Makefile for Python (3) for Python (3) Testing with

    Testing with nose nose: : .PHONY: test .PHONY: test test: nose deps test: nose deps <TAB>$(NOSE) <TAB>$(NOSE) # nose depends on the nosetests binary # nose depends on the nosetests binary nose: $(NOSE) nose: $(NOSE) $(NOSE): env $(NOSE): env <TAB>$(PIP) install nose <TAB>$(PIP) install nose
  109. Contrary Opinions Contrary Opinions “ “Dynamic languages don't need anything

    Dynamic languages don't need anything like like make make, unless they have some , unless they have some compile-time interface dependencies compile-time interface dependencies between modules” between modules” http://stackoverflow.com/questions/758093 http://stackoverflow.com/questions/758093 9/why-are-there-no-makefiles-for-automati 9/why-are-there-no-makefiles-for-automati on-in-python-projects on-in-python-projects
  110. setup.py setup.py overloaded overloaded • create sdist (source distribution) create

    sdist (source distribution) • upload to pypi upload to pypi • install package install package
  111. setup.py setup.py wart wart require require keyword of keyword of

    distutils distutils doesn't doesn't download reqs only download reqs only documents documents them. Use them. Use requirements.txt requirements.txt in combo with in combo with pip pip. .
  112. setup.py setup.py example example From From requests requests: : setup(

    setup( name='requests', name='requests', version=requests.__version__, version=requests.__version__, description='Python HTTP for Humans.', description='Python HTTP for Humans.', long_description=open('README.rst').read() + long_description=open('README.rst').read() + '\n\n' + '\n\n' + open('HISTORY.rst').read(), open('HISTORY.rst').read(), author='Kenneth Reitz', author='Kenneth Reitz', author_email='[email protected]', author_email='[email protected]', url='http://python-requests.org', url='http://python-requests.org',
  113. setup.py setup.py example (2) example (2) packages=packages, packages=packages, package_data={'': ['LICENSE',

    'NOTICE'], package_data={'': ['LICENSE', 'NOTICE'], 'requests': ['*.pem']}, 'requests': ['*.pem']}, package_dir={'requests': 'requests'}, package_dir={'requests': 'requests'}, include_package_data=True, include_package_data=True, install_requires=requires, install_requires=requires, license=open('LICENSE').read(), license=open('LICENSE').read(), zip_safe=False, zip_safe=False,
  114. setup.py setup.py example (3) example (3) classifiers=( classifiers=( 'Development Status

    :: 5 - Production/Stable', 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Intended Audience :: Developers', 'Natural Language :: English', 'Natural Language :: English', 'License :: OSI Approved :: Apache Software License', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python', 'Programming Language :: Python', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.6', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3', # 'Programming Language :: Python :: 3.0', # 'Programming Language :: Python :: 3.0', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.1', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.2', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.3', ), ), ) )
  115. setup.py setup.py modules modules If project consists of a few

    modules this If project consists of a few modules this may be easiest may be easiest
  116. setup.py setup.py packages packages Need to explicitly list Need to

    explicitly list all all packages, not just packages, not just root root
  117. setup.py setup.py non-Python files non-Python files • add files to

    add files to MANIFEST.in MANIFEST.in (include in (include in package) package) • add files to add files to package_data package_data in in setup.py setup.py (include in install) Not recursive (include in install) Not recursive
  118. MANIFEST.in MANIFEST.in language language • include|exclude pat1 pat2 ... include|exclude

    pat1 pat2 ... • recursive-(include|exclude) dir pat1 pat2 ... recursive-(include|exclude) dir pat1 pat2 ... • global-(include|exclude) dir pat1 pat2 ... global-(include|exclude) dir pat1 pat2 ... • prune dir prune dir • graft dir graft dir http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-c http://docs.python.org/release/1.6/dist/sdist-cmd.html#sdist-c md md
  119. setup.py setup.py classifiers classifiers Almost 600 different classifiers. Almost 600

    different classifiers. Not used by pip to enforce versions. For UI Not used by pip to enforce versions. For UI only only
  120. PyPi Register PyPi Register Click on “Register” on right hand

    box Click on “Register” on right hand box https://pypi.python.org/pypi?%3Aaction=re https://pypi.python.org/pypi?%3Aaction=re gister_form gister_form
  121. PyPi Upload PyPi Upload $ python setup.py sdist register $

    python setup.py sdist register upload upload
  122. PyPi Upload (2) PyPi Upload (2) $ python setup.py sdist

    register upload $ python setup.py sdist register upload ... ... Creating tar archive Creating tar archive removing 'rst2odp-0.2.4' (and everything under it) removing 'rst2odp-0.2.4' (and everything under it) running register running register running check running check We need to know who you are, so please choose either: We need to know who you are, so please choose either: 1. use your existing login, 1. use your existing login, 2. register as a new user, 2. register as a new user, 3. have the server generate a new password for you (and email it to you), 3. have the server generate a new password for you (and email it to you), or or 4. quit 4. quit Your selection [default 1]: Your selection [default 1]: 1 1
  123. PyPi Upload (3) PyPi Upload (3) Username: mharrison Username: mharrison

    Password: Password: Registering rst2odp to http://pypi.python.org/pypi Registering rst2odp to http://pypi.python.org/pypi Server response (200): OK Server response (200): OK I can store your PyPI login so future submissions will be faster. I can store your PyPI login so future submissions will be faster. (the login will be stored in /home/matt/.pypirc) (the login will be stored in /home/matt/.pypirc) Save your login (y/N)?y Save your login (y/N)?y running upload running upload Submitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypi Submitting dist/rst2odp-0.2.4.tar.gz to http://pypi.python.org/pypi Server response (200): OK Server response (200): OK
  124. PyPi Note PyPi Note Though PyPi packages are signed there

    is Though PyPi packages are signed there is no verification step during package no verification step during package installation installation
  125. Non PyPi URL Non PyPi URL $ pip install --no-index

    -f $ pip install --no-index -f http://dist.plone.org/thirdparty/ http://dist.plone.org/thirdparty/ -U PIL -U PIL
  126. PyPi PyPi Makefile Makefile integration: integration: # --------- PyPi ----------

    # --------- PyPi ---------- .PHONY: build .PHONY: build build: env build: env <TAB>$(PY) setup.py sdist <TAB>$(PY) setup.py sdist .PHONY: upload .PHONY: upload upload: env upload: env <TAB>$(PY) setup.py sdist register upload <TAB>$(PY) setup.py sdist register upload
  127. Testing (3) Testing (3) Use Use nose nose to run:

    to run: $ env/bin/nosetests $ env/bin/nosetests .. .. ------------- ------------- Ran 2 tests in 0.007s Ran 2 tests in 0.007s OK OK
  128. Testing (4) Testing (4) Makefile Makefile integration: integration: NOSE :=

    env/bin/nosetests NOSE := env/bin/nosetests # --------- Testing ---------- # --------- Testing ---------- .PHONY: test .PHONY: test test: nose deps test: nose deps <TAB>$(NOSE) <TAB>$(NOSE) # nose depends on the nosetests binary # nose depends on the nosetests binary nose: $(NOSE) nose: $(NOSE) $(NOSE): env $(NOSE): env <TAB>$(PIP) install nose <TAB>$(PIP) install nose
  129. Why Github? (2) Why Github? (2) Code is a first

    class object Code is a first class object
  130. Github Github Don't check in keys/passwords! Don't check in keys/passwords!

    http://ejohn.org/blog/keeping-passwords-i http://ejohn.org/blog/keeping-passwords-i n-source-control/#postcomment n-source-control/#postcomment
  131. Travis CI (2) Travis CI (2) Illustrates power of (web)

    hooks Illustrates power of (web) hooks
  132. 5 Steps 5 Steps • Sign-in with github Sign-in with

    github • Sync repos on Profile page Sync repos on Profile page • Enable repo Enable repo • Create a Create a .travis.yml .travis.yml file on github file on github • Push a commit on github Push a commit on github
  133. travis.yml travis.yml language: python language: python python: python: - "3.3"

    - "3.3" - "2.7" - "2.7" - "2.6" - "2.6" - "pypy" - "pypy" # command to run tests # command to run tests script: make test script: make test
  134. poachplate poachplate Example of most of what we have talked

    Example of most of what we have talked about today about today https://github.com/mattharrison/poachplat https://github.com/mattharrison/poachplat e e
  135. That's all That's all Questions? Tweet or email me Questions?

    Tweet or email me [email protected] [email protected] @__mharrison__ @__mharrison__ http://hairysun.com http://hairysun.com