Automated Testing: Philosophy & Tools

A rundown of the benefits of automated testing and how to practically do it with tox and travis.

Transcript

1. Automated Testing Philosophy & Tools
2. I wi" cover • package layout • requirements files • tox • travis
3. I won’t cover • Jenkins • pip and virtualenv • deployment • python 3 compatibility
4. The ideal • Test with every supported python version • Test with every database/search backend • Test with every supported dependency version • Test every possible combination
5. Without automation ~Test with every supported python version ⨉Test with every database/search backend ~Test with every supported dependency version ⨉Test every possible combination
6. With automation ✓Test with every supported python version ~Test with every database/search backend ~Test with every supported dependency version ~Test every possible combination
7. Package layout django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
8. Test project django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
9. • Sets up django environment • Provides canonical test settings • Hides requirements files
10. Requirements files django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
11. • Reproducible test environment(s) • Same today as next week as next year • Precise dependencies
12. ⨉ No django-nose south mock six Pillow Pillow>2.0.0 django<1.5 -e git+https://github.com/jbalogh/django-nose.git@ master#egg=django-nose south>0.8 mock six
13. ✓ Yes Pillow==2.0.0 nose==1.3.0 django==1.4.5 south==0.8.1 mock==1.0.1 -e git+https://github.com/jbalogh/django-nose.git@ 4908503d475144c91f679e945a6e8d8781291f2a#egg=djang o-nose six==1.3.0
14. tox: local testing django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
15. • Isolated virtual environments • Multiple versions of python • Local == pre-commit • .../django-daguerre$ tox
16. tox.ini (basic) [tox] envlist = py26, py27, py33 [testenv] changedir = {toxinidir}/test_project deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt commands = {envpython} manage.py test daguerre
17. tox.ini (matrix) [tox] envlist = py26-django14, py27-django14, py26-django15, py27-django15, [testenv] changedir = {toxinidir}/test_project commands = {envpython} manage.py test daguerre # Continued
18. tox.ini (matrix) [testenv:py26-django14] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py27-django14] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py26-django15] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt [testenv:py27-django15] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt
19. travis-ci: remote testing django-daguerre/ .travis.yml AUTHORS LICENSE MANIFEST.in setup.py tox.ini daguerre/ docs/ test_project/ requirements-1.4.txt requirements-1.5.txt manage.py test_project/
20. • Isolated virtual environments • Multiple versions of python • Good matrix support • Built-in support for MySQL, PostgreSQL, Elasticsearch, Redis, and more • Remote == post-push • Integration for public Github repos only
21. .travis.yml (basic) language: python python: - "2.6" - "2.7" - "3.3" install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r test_project/requirements-1.5.txt script: - cd test_project - ./manage.py test --verbosity=2 daguerre
22. tox.ini (basic) [tox] envlist = py26, py27, py33 [testenv] changedir = {toxinidir}/test_project deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt commands = {envpython} manage.py test daguerre
23. .travis.yml (matrix) language: python python: - "2.6" - "2.7" - "3.3" env: - REQUIREMENTS=test_project/requirements-1.4.txt - REQUIREMENTS=test_project/requirements-1.5.txt matrix: exclude: - python: "3.3" env: REQUIREMENTS=test_project/requirements-1.4.txt install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r $REQUIREMENTS script: - cd test_project - ./manage.py test --verbosity=2 daguerre
24. tox.ini (matrix) [tox] envlist = py26-django14, py27-django14, py26-django15, py27-django15, [testenv] changedir = {toxinidir}/test_project commands = {envpython} manage.py test daguerre # Continued
25. tox.ini (matrix) [testenv:py26-django14] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py27-django14] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.4.txt [testenv:py26-django15] basepython=python2.6 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt [testenv:py27-django15] basepython=python2.7 deps = --no-deps --use-mirrors -r{toxinidir}/test_project/requirements-1.5.txt
26. .travis.yml (matrix 2) language: python python: - "2.6" - "2.7" - "3.3" env: - DB=sqlite3 REQUIREMENTS=test_project/requirements-1.4.txt - DB=mysql REQUIREMENTS=test_project/requirements-1.4.txt - DB=sqlite3 REQUIREMENTS=test_project/requirements-1.5.txt - DB=mysql REQUIREMENTS=test_project/requirements-1.5.txt matrix: exclude: - python: "3.3" env: DB=sqlite3 REQUIREMENTS=test_project/requirements-1.4.txt - python: "3.3" env: DB=mysql REQUIREMENTS=test_project/requirements-1.4.txt - python: "3.3" env: DB=mysql REQUIREMENTS=test_project/requirements-1.5.txt install: - pip install . --no-deps - pip install --use-mirrors --no-deps -r $REQUIREMENTS - if [ $DB == 'mysql' ]; then pip install --use-mirrors --no-deps mysql-python; fi before_script: - if [ $DB == 'mysql' ]; then mysql -e 'CREATE DATABASE daguerre_test;'; fi script: - cd test_project - ./manage.py test --verbosity=2 daguerre
27. (in settings.py) import os DB = os.environ.get('DB') if DB == 'mysql': DATABASES = { 'default': { 'ENGINE': 'django.db.backends.mysql', 'NAME': 'daguerre_test', 'USER': 'root', 'TEST_CHARSET': 'utf8', 'TEST_COLLATION': 'utf8_general_ci', } } else: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.sqlite3', 'NAME': os.path.join(PROJECT_DIR, 'db.sl3'), } }
28. Summary (tox) • Multiple versions of python: trivial • Two-dimensional matrix: work, but possible • For example, multiple versions of django • Higher-dimensional matrix: not really feasible • For example multiple versions of django and multiple database settings
29. Summary (travis) • Multiple versions of python: trivial • Two-dimensional matrix: simple • For example, multiple versions of django • Higher-dimensional matrix: work, but possible • For example multiple versions of django and multiple database settings
30. Automated testing?
31. MAKE IT SO
32. Thanks! • Stephen Burrows (melinath) • github.com/melinath/django-daguerre • travis-ci.org • tox.readthedocs.org