Slide 1

Slide 1 text

Lessons in Testing David Cramer twitter.com/zeeg Wednesday, September 5, 12

Slide 2

Slide 2 text

Who am I? Wednesday, September 5, 12

Slide 3

Slide 3 text

I lead Infrastructure at DISQUS Wednesday, September 5, 12

Slide 4

Slide 4 text

Wednesday, September 5, 12

Slide 5

Slide 5 text

DISQUS sees a lot of tra c Google Analytics: Aug 2 2012 - Sep 1 2012 Wednesday, September 5, 12

Slide 6

Slide 6 text

Lots of Django (and things like Flask) Wednesday, September 5, 12

Slide 7

Slide 7 text

Growing Startup (less than 20 engineers) Wednesday, September 5, 12

Slide 8

Slide 8 text

Wednesday, September 5, 12

Slide 9

Slide 9 text

We are also terrible at testing (but we try not to be) Wednesday, September 5, 12

Slide 10

Slide 10 text

This presentation is about Lessons and Aspirations Wednesday, September 5, 12

Slide 11

Slide 11 text

Lessons Mostly Learned Wednesday, September 5, 12

Slide 12

Slide 12 text

Lesson #1: No Want Likes Writing Tests Wednesday, September 5, 12

Slide 13

Slide 13 text

Time Consuming to Write Wednesday, September 5, 12

Slide 14

Slide 14 text

Simple Feature: 10 lines of code from django.db import transaction class AutocommitMiddleware(object): def process_view(self, request, *a, **kw): if request.method == 'POST': enter_transaction_management(True) def process_response(self, request, response): leave_transaction_management() return response def process_exception(self, *a, **kw): leave_transaction_management() Wednesday, September 5, 12

Slide 15

Slide 15 text

with precision testing class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12

Slide 16

Slide 16 text

turns into 36 lines of tests import mock from mock_django.http import WsgiHttpRequest from django.http import HttpResponse from disqus.middleware.autocommit import AutocommitMiddleware from disqus.tests import DisqusTest class AutocommitMiddlewareTest(DisqusTest): @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_changes_to_transactional_on_post(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) enter_transaction_management.assert_called_once_with(True, using='default') @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_leaves_autocommit_on_get(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'GET' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) self.assertFalse(enter_transaction_management.called) @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_response(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' response = HttpResponse() middleware = AutocommitMiddleware() middleware.process_response(request, response) leave_transaction_management.assert_called_once_with(using='default') @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_exception(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' exception = Exception() middleware = AutocommitMiddleware() middleware.process_exception(request, exception) leave_transaction_management.assert_called_once_with(using='default') Wednesday, September 5, 12

Slide 17

Slide 17 text

~50% of the time was writing tests (often more) Wednesday, September 5, 12

Slide 18

Slide 18 text

Legacy (Untested) Code is Expensive Wednesday, September 5, 12

Slide 19

Slide 19 text

Add Tests for New Code Wednesday, September 5, 12

Slide 20

Slide 20 text

Add Tests for Regressions Wednesday, September 5, 12

Slide 21

Slide 21 text

You Pick: Fast or Accurate (you only get to have one) Wednesday, September 5, 12

Slide 22

Slide 22 text

Spend more time writing tests or more time running them Wednesday, September 5, 12

Slide 23

Slide 23 text

You Can Buy Faster Hardware Wednesday, September 5, 12

Slide 24

Slide 24 text

class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12

Slide 25

Slide 25 text

class AutocommitMiddlewareTest: def setUp(self): assert 'AutocommitMiddleware' in INSTALLED_APPS def test_something_useful(self): with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, September 5, 12

Slide 26

Slide 26 text

Higher Level Tests are Slower But More “Correct” Wednesday, September 5, 12

Slide 27

Slide 27 text

Unit vs Integration Tests Wednesday, September 5, 12

Slide 28

Slide 28 text

@mock.patch('enter_transaction_management') def test_etm_called(self, etm): request = WsgiHttpRequest(method='POST') middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12

Slide 29

Slide 29 text

with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, September 5, 12

Slide 30

Slide 30 text

Which is better? (you should have both) Wednesday, September 5, 12

Slide 31

Slide 31 text

Mocking is Fragile Wednesday, September 5, 12

Slide 32

Slide 32 text

# what happens if the path changes? @mock.patch('enter_transaction_management') def test_something_useful(self, etm): Wednesday, September 5, 12

Slide 33

Slide 33 text

# what if we change the arguments passed? etm.assert_called_once_with(True) Wednesday, September 5, 12

Slide 34

Slide 34 text

Useful for Testing External Services (Twitter, Facebook, Disqus) Wednesday, September 5, 12

Slide 35

Slide 35 text

Capture Live Data for Mocking Wednesday, September 5, 12

Slide 36

Slide 36 text

class TwitterTest: @mock.patch('twitter.Twitter.getresponse') def test_something_useful(self, gr): gr.return_value = TWITTER_RESPONSE res = Twitter().foo() assert res == [1, 2, 3] Wednesday, September 5, 12

Slide 37

Slide 37 text

Assume APIs Don’t Change (it’s mostly true) Wednesday, September 5, 12

Slide 38

Slide 38 text

Test The Lifecycle of Requests Wednesday, September 5, 12

Slide 39

Slide 39 text

Selenium Kind of Works (but it’s extremely brittle) Wednesday, September 5, 12

Slide 40

Slide 40 text

Lesson #2: Don’t Admit Defeat Wednesday, September 5, 12

Slide 41

Slide 41 text

Start with a Goal Wednesday, September 5, 12

Slide 42

Slide 42 text

Goal #1: Write Testable Code Wednesday, September 5, 12

Slide 43

Slide 43 text

A lot of Code is Hard to Test (but it doesn’t have to be) Wednesday, September 5, 12

Slide 44

Slide 44 text

def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, September 5, 12

Slide 45

Slide 45 text

Break Up Your Code Wednesday, September 5, 12

Slide 46

Slide 46 text

def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, September 5, 12

Slide 47

Slide 47 text

Goal #2: Start Writing Tests Wednesday, September 5, 12

Slide 48

Slide 48 text

Make Testing Easy Wednesday, September 5, 12

Slide 49

Slide 49 text

class AutocommitMiddlewareTest(DisqusTest): def test_changes_to_transactional_on_post(self): # adding a simple assert helper makes this # test much easier to write with self.assertInTransaction(): # changing this to an integration test # doesn’t require an understanding of # Django middleware internals self.client.post('/') Wednesday, September 5, 12

Slide 50

Slide 50 text

Create Structure Wednesday, September 5, 12

Slide 51

Slide 51 text

tests !"" __init__.py !"" integration # !"" __init__.py # !"" disqus # # !"" __init__.py # # !"" db # # # !"" __init__.py # # # !"" annotations # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" cachedbymodel # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" disqusmanager # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" iterablequeryset # # # # !"" __init__.py # # # # %"" tests.py Wednesday, September 5, 12

Slide 52

Slide 52 text

Document Best Practices Wednesday, September 5, 12

Slide 53

Slide 53 text

Wednesday, September 5, 12

Slide 54

Slide 54 text

Prevent Mistakes Wednesday, September 5, 12

Slide 55

Slide 55 text

NOSE_PLUGINS = [ 'disqus.tests.plugins.redis.RedisPlugin', 'disqus.tests.plugins.sockets.ErroringSocketWhitelistPlugin', 'disqus.tests.plugins.impermium.MockImpermiumPlugin', 'disqus.tests.plugins.mail.FixMailHostnamePlugin', 'disqus.tests.plugins.compat.UnitTestPlugin', 'disqus.tests.plugins.tags.TagSelector', 'disqus.tests.plugins.timing.TimingPlugin', 'disqus.tests.plugins.themes.DefaultThemePlugin', 'disqus.tests.plugins.apps.AddInstalledApps', 'disqus.tests.plugins.selenium.ScreenshotFailurePlugin', 'disqus.tests.plugins.tasks.SeppukuTasksPlugin', 'disqus.tests.plugins.django.DjangoSetUpPlugin', ] Wednesday, September 5, 12

Slide 56

Slide 56 text

Goal #3: Continuously Run Tests (via automation) Wednesday, September 5, 12

Slide 57

Slide 57 text

Wednesday, September 5, 12

Slide 58

Slide 58 text

Drive It Into Your Culture Wednesday, September 5, 12

Slide 59

Slide 59 text

Development Workflow Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, September 5, 12

Slide 60

Slide 60 text

Use Code Review Wednesday, September 5, 12

Slide 61

Slide 61 text

Wednesday, September 5, 12

Slide 62

Slide 62 text

Test Throughout The Process Wednesday, September 5, 12

Slide 63

Slide 63 text

Test Lifecycle Pre-Review Tests CI Smoke Tests Failed Build Commit Wednesday, September 5, 12

Slide 64

Slide 64 text

Tools of the Trade Wednesday, September 5, 12

Slide 65

Slide 65 text

Use the Right Tools (if they don’t exist, build them) Wednesday, September 5, 12

Slide 66

Slide 66 text

Nose A Better Test Runner nose.readthedocs.org Wednesday, September 5, 12

Slide 67

Slide 67 text

Plugins for (Almost) Everything Wednesday, September 5, 12

Slide 68

Slide 68 text

# dump XML results nosetests --with-xunit # drop into a PDB on errors and failures nosetests --pdb --pdb-failure Wednesday, September 5, 12

Slide 69

Slide 69 text

Coverage.py Find Untested Code nedbatchelder.com/code/coverage Wednesday, September 5, 12

Slide 70

Slide 70 text

Wednesday, September 5, 12

Slide 71

Slide 71 text

Wednesday, September 5, 12

Slide 72

Slide 72 text

Sentry Exception Reporting github.com/getsentry/sentry Wednesday, September 5, 12

Slide 73

Slide 73 text

Because Tests Aren’t Enough Wednesday, September 5, 12

Slide 74

Slide 74 text

Wednesday, September 5, 12

Slide 75

Slide 75 text

Wednesday, September 5, 12

Slide 76

Slide 76 text

Simple Continuous Integration jenkins-ci.org Wednesday, September 5, 12

Slide 77

Slide 77 text

A Workflow With CI Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, September 5, 12

Slide 78

Slide 78 text

Selenium api/* Unit Integration disqus-web admin/* analytics/* JavaScript Package Simple Build Distribution embed/* forums/* db/* Performance Wednesday, September 5, 12

Slide 79

Slide 79 text

Code Review Done Right phabricator.org Wednesday, September 5, 12

Slide 80

Slide 80 text

Highly Integrated Review Process Wednesday, September 5, 12

Slide 81

Slide 81 text

Wednesday, September 5, 12

Slide 82

Slide 82 text

Wednesday, September 5, 12

Slide 83

Slide 83 text

Gargoyle Selectively Enable Features in Code github.com/disqus/gargoyle Wednesday, September 5, 12

Slide 84

Slide 84 text

Wednesday, September 5, 12

Slide 85

Slide 85 text

from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('my_awesome_switch', request): # do something awesome else: # do something less awesome Wednesday, September 5, 12

Slide 86

Slide 86 text

Zumanji Performance Testing github.com/disqus/zumanji Wednesday, September 5, 12

Slide 87

Slide 87 text

Wednesday, September 5, 12

Slide 88

Slide 88 text

Wednesday, September 5, 12

Slide 89

Slide 89 text

Still a W.I.P. Wednesday, September 5, 12

Slide 90

Slide 90 text

Takeaways Wednesday, September 5, 12

Slide 91

Slide 91 text

Your Needs Will Vary Wednesday, September 5, 12

Slide 92

Slide 92 text

The Process is Evolving Wednesday, September 5, 12

Slide 93

Slide 93 text

Culture is Key Wednesday, September 5, 12

Slide 94

Slide 94 text

Go Write Tests! Wednesday, September 5, 12

Slide 95

Slide 95 text

Questions? Wednesday, September 5, 12

Slide 96

Slide 96 text

Obligatory Slide Full of Useful Links ๏ Sentry (exception reporting) github.com/dcramer/sentry ๏ Jenkins CI (continuous integration) jenkins-ci.org ๏ Phabricator (code review) phabricator.org ๏ Gargoyle (feature switches) github.com/disqus/gargoyle ๏ Zumanji (performance tests) github.com/disqus/zumanji ๏ Mock mock.readthedocs.org ๏ Nose (test runner) nose.readthedocs.org ๏ nose-quickunit (test discovery) github.com/dcramer/nose-quickunit ๏ django-nose (Django integration) github.com/jbalogh/django-nose ๏ Coverage.py (code coverage) nedbatchelder.com/code/coverage/ Wednesday, September 5, 12