Lessons in Testing - DjangoCon 2012

Lessons in Testing
David Cramer
twitter.com/zeeg
Wednesday, September 5, 12

View Slide

Who am I?

View Slide

I lead Infrastructure at DISQUS

View Slide


View Slide

DISQUS sees a lot of tra c
Google Analytics: Aug 2 2012 - Sep 1 2012

View Slide

Lots of Django
(and things like Flask)

View Slide

Growing Startup
(less than 20 engineers)

View Slide


View Slide

We are also terrible at testing
(but we try not to be)

View Slide

This presentation is about
Lessons and Aspirations

View Slide

Lessons Mostly Learned

View Slide

Lesson #1:
No Want Likes Writing Tests

View Slide

Time Consuming to Write

View Slide

Simple Feature: 10 lines of code
from django.db import transaction
class AutocommitMiddleware(object):
def process_view(self, request, *a, **kw):
if request.method == 'POST':
enter_transaction_management(True)
def process_response(self, request, response):
leave_transaction_management()
return response
def process_exception(self, *a, **kw):
leave_transaction_management()

View Slide

with precision testing
class AutocommitMiddlewareTest:
@mock.patch('enter_transaction_management')
def test_something_precise(self, etm):
request = WsgiHttpRequest()
request.method = 'POST'
middleware = AutocommitMiddleware()
middleware.process_view(request)
etm.assert_called_once_with(True)

View Slide

turns into 36 lines of tests
import mock
from mock_django.http import WsgiHttpRequest
from django.http import HttpResponse
from disqus.middleware.autocommit import AutocommitMiddleware
from disqus.tests import DisqusTest
class AutocommitMiddlewareTest(DisqusTest):
@mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management')
def test_changes_to_transactional_on_post(self, enter_transaction_management):
middleware.process_view(request, lambda: "", [], {})
enter_transaction_management.assert_called_once_with(True, using='default')
@mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management')
def test_leaves_autocommit_on_get(self, enter_transaction_management):
request.method = 'GET'
middleware.process_view(request, lambda: "", [], {})
self.assertFalse(enter_transaction_management.called)
@mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management')
def test_changes_to_autocommit_on_response(self, leave_transaction_management):
response = HttpResponse()
middleware.process_response(request, response)
leave_transaction_management.assert_called_once_with(using='default')
@mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management')
def test_changes_to_autocommit_on_exception(self, leave_transaction_management):
exception = Exception()
middleware.process_exception(request, exception)
leave_transaction_management.assert_called_once_with(using='default')

View Slide

~50% of the time was writing tests
(often more)

View Slide

Legacy (Untested) Code is Expensive

View Slide

Add Tests for New Code

View Slide

Add Tests for Regressions

View Slide

You Pick: Fast or Accurate
(you only get to have one)

View Slide

Spend more time writing tests
or more time running them

View Slide

You Can Buy Faster Hardware

View Slide

def test_something_precise(self, etm):

View Slide

def setUp(self):
assert 'AutocommitMiddleware' in INSTALLED_APPS
def test_something_useful(self):
with self.assertInTransaction():
# use Django’s integrated test client
self.client.post('/')

View Slide

Higher Level Tests are Slower
But More “Correct”

View Slide

Unit vs Integration Tests

View Slide

def test_etm_called(self, etm):
request = WsgiHttpRequest(method='POST')

View Slide

# use Django’s integrated test client

View Slide

Which is better?
(you should have both)

View Slide

Mocking is Fragile

View Slide

# what happens if the path changes?
def test_something_useful(self, etm):

View Slide

# what if we change the arguments passed?

View Slide

Useful for Testing External Services
(Twitter, Facebook, Disqus)

View Slide

Capture Live Data for Mocking

View Slide

class TwitterTest:
@mock.patch('twitter.Twitter.getresponse')
def test_something_useful(self, gr):
gr.return_value = TWITTER_RESPONSE
res = Twitter().foo()
assert res == [1, 2, 3]

View Slide

Assume APIs Don’t Change
(it’s mostly true)

View Slide

Test The Lifecycle of Requests

View Slide

Selenium Kind of Works
(but it’s extremely brittle)

View Slide

Lesson #2:
Don’t Admit Defeat

View Slide

Start with a Goal

View Slide

Goal #1:
Write Testable Code

View Slide

A lot of Code is Hard to Test
(but it doesn’t have to be)

View Slide

def static_media(request, path):
"""
Serve static files below a given point in the directory structure.
"""
from django.utils.http import http_date
from django.views.static import was_modified_since
import mimetypes
import os.path
import posixpath
import stat
import urllib
document_root = os.path.join(settings.STATIC_ROOT)
path = posixpath.normpath(urllib.unquote(path))
path = path.lstrip('/')
newpath = ''
for part in path.split('/'):
if not part:
# Strip empty path components.
continue
drive, part = os.path.splitdrive(part)
head, part = os.path.split(part)
if part in (os.curdir, os.pardir):
# Strip '.' and '..' in path.
continue
newpath = os.path.join(newpath, part).replace('\\', '/')
if newpath and path != newpath:
return HttpResponseRedirect(newpath)
fullpath = os.path.join(document_root, newpath)
if os.path.isdir(fullpath):
raise Http404("Directory indexes are not allowed here.")
if not os.path.exists(fullpath):
raise Http404('"%s" does not exist' % fullpath)
# Respect the If-Modified-Since header.
statobj = os.stat(fullpath)
mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream'
if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'),
statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]):
return HttpResponseNotModified(mimetype=mimetype)
contents = open(fullpath, 'rb').read()
response = HttpResponse(contents, mimetype=mimetype)
response["Last-Modified"] = http_date(statobj[stat.ST_MTIME])
response["Content-Length"] = len(contents)
return response

View Slide

Break Up Your Code

View Slide

def static_media(request, path):
"""
Serve static files below a given point in the directory structure.
"""
from django.utils.http import http_date
from django.views.static import was_modified_since
import mimetypes
import os.path
import posixpath
import stat
import urllib
document_root = os.path.join(settings.STATIC_ROOT)
path = posixpath.normpath(urllib.unquote(path))
path = path.lstrip('/')
newpath = ''
for part in path.split('/'):
if not part:
# Strip empty path components.
continue
drive, part = os.path.splitdrive(part)
head, part = os.path.split(part)
if part in (os.curdir, os.pardir):
# Strip '.' and '..' in path.
continue
newpath = os.path.join(newpath, part).replace('\\', '/')
if newpath and path != newpath:
return HttpResponseRedirect(newpath)
fullpath = os.path.join(document_root, newpath)
if os.path.isdir(fullpath):
raise Http404("Directory indexes are not allowed here.")
if not os.path.exists(fullpath):
raise Http404('"%s" does not exist' % fullpath)
# Respect the If-Modified-Since header.
statobj = os.stat(fullpath)
mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream'
if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'),
statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]):
return HttpResponseNotModified(mimetype=mimetype)
contents = open(fullpath, 'rb').read()
response = HttpResponse(contents, mimetype=mimetype)
response["Last-Modified"] = http_date(statobj[stat.ST_MTIME])
response["Content-Length"] = len(contents)
return response

View Slide

Goal #2:
Start Writing Tests

View Slide

Make Testing Easy

View Slide

class AutocommitMiddlewareTest(DisqusTest):
def test_changes_to_transactional_on_post(self):
# adding a simple assert helper makes this
# test much easier to write
# changing this to an integration test
# doesn’t require an understanding of
# Django middleware internals

View Slide

Create Structure

View Slide

tests
!"" __init__.py
!"" integration
# !"" __init__.py
# !"" disqus
# # !"" __init__.py
# # !"" db
# # # !"" __init__.py
# # # !"" annotations
# # # # !"" __init__.py
# # # # !"" models.py
# # # # %"" tests.py
# # # !"" cachedbymodel
# # # # !"" __init__.py
# # # # !"" models.py
# # # # %"" tests.py
# # # !"" disqusmanager
# # # # !"" __init__.py
# # # # !"" models.py
# # # # %"" tests.py
# # # !"" iterablequeryset
# # # # !"" __init__.py
# # # # %"" tests.py

View Slide

Document Best Practices

View Slide


View Slide

Prevent Mistakes

View Slide

NOSE_PLUGINS = [
'disqus.tests.plugins.redis.RedisPlugin',
'disqus.tests.plugins.sockets.ErroringSocketWhitelistPlugin',
'disqus.tests.plugins.impermium.MockImpermiumPlugin',
'disqus.tests.plugins.mail.FixMailHostnamePlugin',
'disqus.tests.plugins.compat.UnitTestPlugin',
'disqus.tests.plugins.tags.TagSelector',
'disqus.tests.plugins.timing.TimingPlugin',
'disqus.tests.plugins.themes.DefaultThemePlugin',
'disqus.tests.plugins.apps.AddInstalledApps',
'disqus.tests.plugins.selenium.ScreenshotFailurePlugin',
'disqus.tests.plugins.tasks.SeppukuTasksPlugin',
'disqus.tests.plugins.django.DjangoSetUpPlugin',
]

View Slide

Goal #3:
Continuously Run Tests
(via automation)

View Slide


View Slide

Drive It Into Your Culture

View Slide

Development Workﬂow
Review
Integration
Deploy
Failed Build
Reporting
Rollback
Commit

View Slide

Use Code Review

View Slide


View Slide

Test Throughout The Process

View Slide

Test Lifecycle
Pre-Review Tests
CI
Smoke Tests
Failed Build
Commit

View Slide

Tools of the Trade

View Slide

Use the Right Tools
(if they don’t exist, build them)

View Slide

Nose
A Better Test Runner
nose.readthedocs.org

View Slide

Plugins for (Almost) Everything

View Slide

# dump XML results
nosetests --with-xunit
# drop into a PDB on errors and failures
nosetests --pdb --pdb-failure

View Slide

Coverage.py
Find Untested Code
nedbatchelder.com/code/coverage

View Slide


View Slide

Sentry
Exception Reporting
github.com/getsentry/sentry

View Slide

Because Tests Aren’t Enough

View Slide


View Slide

Simple Continuous Integration
jenkins-ci.org

View Slide

A Workﬂow With CI
Review
Integration
Deploy
Failed Build
Reporting
Rollback
Commit

View Slide

Selenium
api/*
Unit
Integration
disqus-web
admin/* analytics/*
JavaScript Package
Simple Build Distribution
embed/*
forums/* db/*
Performance

View Slide

Code Review Done Right
phabricator.org

View Slide

Highly Integrated Review Process

View Slide


View Slide

Gargoyle
Selectively Enable Features in Code
github.com/disqus/gargoyle

View Slide


View Slide

from gargoyle import gargoyle
def my_view(request):
if gargoyle.is_active('my_awesome_switch',
request):
# do something awesome
else:
# do something less awesome

View Slide

Zumanji
Performance Testing
github.com/disqus/zumanji

View Slide


View Slide

Still a W.I.P.

View Slide

Takeaways

View Slide

Your Needs Will Vary

View Slide

The Process is Evolving

View Slide

Culture is Key

View Slide

Go Write Tests!

View Slide

Questions?

View Slide

Obligatory Slide Full of Useful Links
๏ Sentry (exception reporting)
github.com/dcramer/sentry
๏ Jenkins CI (continuous integration)
jenkins-ci.org
๏ Phabricator (code review)
phabricator.org
๏ Gargoyle (feature switches)
github.com/disqus/gargoyle
๏ Zumanji (performance tests)
github.com/disqus/zumanji
๏ Mock
mock.readthedocs.org
๏ Nose (test runner)
nose.readthedocs.org
๏ nose-quickunit (test discovery)
github.com/dcramer/nose-quickunit
๏ django-nose (Django integration)
github.com/jbalogh/django-nose
๏ Coverage.py (code coverage)
nedbatchelder.com/code/coverage/

View Slide

Lessons in Testing - DjangoCon 2012

Lessons in Testing - DjangoCon 2012

David Cramer

More Decks by David Cramer

Other Decks in Technology

Featured

Transcript