$30 off During Our Annual Pro Sale. View Details »

Automated Testing: Philosophy & Tools

Automated Testing: Philosophy & Tools

A rundown of the benefits of automated testing and how to practically do it with tox and travis.

Stephen Burrows

June 25, 2013
Tweet

Other Decks in Programming

Transcript

  1. Automated Testing Philosophy & Tools

  2. I wi" cover • package layout • requirements files •

    tox • travis
  3. I won’t cover • Jenkins • pip and virtualenv •

    deployment • python 3 compatibility
  4. The ideal • Test with every supported python version •

    Test with every database/search backend • Test with every supported dependency version • Test every possible combination
  5. Without automation ~Test with every supported python version ⨉Test with

    every database/search backend ~Test with every supported dependency version ⨉Test every possible combination
  6. With automation ✓Test with every supported python version ~Test with

    every database/search backend ~Test with every supported dependency version ~Test every possible combination
  7. Package layout django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/

    docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
  8. Test project django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/

    docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
  9. • Sets up django environment • Provides canonical test settings

    • Hides requirements files
  10. Requirements files django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/

    docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
  11. • Reproducible test environment(s) • Same today as next week

    as next year • Precise dependencies
  12. ⨉ No django-nose south mock six Pillow Pillow>2.0.0 django<1.5 -e

    git+https://github.com/jbalogh/django-nose.git@ master#egg=django-nose south>0.8 mock six
  13. ✓ Yes Pillow==2.0.0 nose==1.3.0 django==1.4.5 south==0.8.1 mock==1.0.1 -e git+https://github.com/jbalogh/django-nose.git@ 4908503d475144c91f679e945a6e8d8781291f2a#egg=djang

    o-nose six==1.3.0
  14. tox: local testing django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini

    daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
  15. • Isolated virtual environments • Multiple versions of python •

    Local == pre-commit • .../django-daguerre$ tox
  16. tox.ini (basic) [tox] envlist = py26, py27, py33 [testenv] changedir

    = {toxinidir}/test_project deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt commands = {envpython} manage.py test daguerre
  17. tox.ini (matrix) [tox] envlist = py26-django14, py27-django14, py26-django15, py27-django15, [testenv]

    changedir = {toxinidir}/test_project commands = {envpython} manage.py test daguerre # Continued
  18. tox.ini (matrix) [testenv:py26-django14] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py27-django14]

    basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py26-django15] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt [testenv:py27-django15] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt
  19. travis-ci: remote testing django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini

    daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
  20. • Isolated virtual environments • Multiple versions of python •

    Good matrix support • Built-in support for MySQL, PostgreSQL, Elasticsearch, Redis, and more • Remote == post-push • Integration for public Github repos only
  21. .travis.yml (basic) language: python python: - "2.6" - "2.7" -

    "3.3" install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r test_project/requirements-1.5.txt script: - cd test_project - ./manage.py test --verbosity=2 daguerre
  22. tox.ini (basic) [tox] envlist = py26, py27, py33 [testenv] changedir

    = {toxinidir}/test_project deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt commands = {envpython} manage.py test daguerre
  23. .travis.yml (matrix) language: python python: - "2.6" - "2.7" -

    "3.3" env: - REQUIREMENTS=test_project/requirements-1.4.txt - REQUIREMENTS=test_project/requirements-1.5.txt matrix: exclude: - python: "3.3" env: REQUIREMENTS=test_project/requirements-1.4.txt install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r $REQUIREMENTS script: - cd test_project - ./manage.py test --verbosity=2 daguerre
  24. tox.ini (matrix) [tox] envlist = py26-django14, py27-django14, py26-django15, py27-django15, [testenv]

    changedir = {toxinidir}/test_project commands = {envpython} manage.py test daguerre # Continued
  25. tox.ini (matrix) [testenv:py26-django14] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py27-django14]

    basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py26-django15] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt [testenv:py27-django15] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt
  26. .travis.yml (matrix 2) language: python python: - "2.6" - "2.7"

    - "3.3" env: - DB=sqlite3 REQUIREMENTS=test_project/requirements-1.4.txt - DB=mysql REQUIREMENTS=test_project/requirements-1.4.txt - DB=sqlite3 REQUIREMENTS=test_project/requirements-1.5.txt - DB=mysql REQUIREMENTS=test_project/requirements-1.5.txt matrix: exclude: - python: "3.3" env: DB=sqlite3 REQUIREMENTS=test_project/requirements-1.4.txt - python: "3.3" env: DB=mysql REQUIREMENTS=test_project/requirements-1.4.txt - python: "3.3" env: DB=mysql REQUIREMENTS=test_project/requirements-1.5.txt install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r $REQUIREMENTS - if [ $DB == 'mysql' ]; then pip install --use-mirrors --no-deps mysql-python; fi before_script: - if [ $DB == 'mysql' ]; then mysql -e 'CREATE DATABASE daguerre_test;'; fi script: - cd test_project - ./manage.py test --verbosity=2 daguerre
  27. (in settings.py) import os DB = os.environ.get('DB') if DB ==

    'mysql': DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'daguerre_test', 'USER': 'root', 'TEST_CHARSET': 'utf8', 'TEST_COLLATION': 'utf8_general_ci', } } else: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(PROJECT_DIR, 'db.sl3'), } }
  28. Summary (tox) • Multiple versions of python: trivial • Two-dimensional

    matrix: work, but possible • For example, multiple versions of django • Higher-dimensional matrix: not really feasible • For example multiple versions of django and multiple database settings
  29. Summary (travis) • Multiple versions of python: trivial • Two-dimensional

    matrix: simple • For example, multiple versions of django • Higher-dimensional matrix: work, but possible • For example multiple versions of django and multiple database settings
  30. Automated testing?

  31. MAKE IT SO

  32. Thanks! • Stephen Burrows (melinath) • github.com/melinath/django-daguerre • travis-ci.org •

    tox.readthedocs.org