Slide 1

Slide 1 text

Lessons in Testing David Cramer twitter.com/zeeg Wednesday, July 4, 12

Slide 2

Slide 2 text

Who am I? Wednesday, July 4, 12

Slide 3

Slide 3 text

I lead Infrastructure at DISQUS Wednesday, July 4, 12

Slide 4

Slide 4 text

Wednesday, July 4, 12

Slide 5

Slide 5 text

DISQUS sees a lot of tra c Google Analytics: May 30 2012 - June 28 2012 Wednesday, July 4, 12

Slide 6

Slide 6 text

One of the largest consumers of Django Wednesday, July 4, 12

Slide 7

Slide 7 text

... and Flask Wednesday, July 4, 12

Slide 8

Slide 8 text

We are a Growing Startup (less than 20 engineers) Wednesday, July 4, 12

Slide 9

Slide 9 text

Wednesday, July 4, 12

Slide 10

Slide 10 text

We are also terrible at testing (but we try not to be) Wednesday, July 4, 12

Slide 11

Slide 11 text

This presentation is about Lessons and Aspirations Wednesday, July 4, 12

Slide 12

Slide 12 text

Lessons Mostly Learned Wednesday, July 4, 12

Slide 13

Slide 13 text

Lesson #1: No Want Likes Writing Tests Wednesday, July 4, 12

Slide 14

Slide 14 text

Time Consuming to Write Wednesday, July 4, 12

Slide 15

Slide 15 text

Simple Feature: 10 lines of code from django.db import transaction class AutocommitMiddleware(object): def process_view(self, request, *a, **kw): if request.method == 'POST': enter_transaction_management(True) def process_response(self, request, response): leave_transaction_management() return response def process_exception(self, *a, **kw): leave_transaction_management() Wednesday, July 4, 12

Slide 16

Slide 16 text

with precision testing class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, July 4, 12

Slide 17

Slide 17 text

turns into 36 lines of tests import mock from mock_django.http import WsgiHttpRequest from django.http import HttpResponse from disqus.middleware.autocommit import AutocommitMiddleware from disqus.tests import DisqusTest class AutocommitMiddlewareTest(DisqusTest): @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_changes_to_transactional_on_post(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) enter_transaction_management.assert_called_once_with(True, using='default') @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_leaves_autocommit_on_get(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'GET' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) self.assertFalse(enter_transaction_management.called) @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_response(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' response = HttpResponse() middleware = AutocommitMiddleware() middleware.process_response(request, response) leave_transaction_management.assert_called_once_with(using='default') @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_exception(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' exception = Exception() middleware = AutocommitMiddleware() middleware.process_exception(request, exception) leave_transaction_management.assert_called_once_with(using='default') Wednesday, July 4, 12

Slide 18

Slide 18 text

~50% of the time was writing tests (sometimes more) Wednesday, July 4, 12

Slide 19

Slide 19 text

Legacy (Untested) Code is Expensive Wednesday, July 4, 12

Slide 20

Slide 20 text

Add Tests for New Code Wednesday, July 4, 12

Slide 21

Slide 21 text

Add Tests for Regressions Wednesday, July 4, 12

Slide 22

Slide 22 text

Slow or Inaccurate (there can be only one) Wednesday, July 4, 12

Slide 23

Slide 23 text

Spend more time writing tests or more time running them Wednesday, July 4, 12

Slide 24

Slide 24 text

We Cater to Cost to Run Wednesday, July 4, 12

Slide 25

Slide 25 text

Hardware Costs Decrease (Engineers Don’t) Wednesday, July 4, 12

Slide 26

Slide 26 text

Interface Contracts Yield Inaccuracy Wednesday, July 4, 12

Slide 27

Slide 27 text

@mock.patch('enter_transaction_management') def test_something_useful(self, etm): # ... etm.assert_called_once_with(True) ๏ Are we calling enter_transaction_management correctly Wednesday, July 4, 12

Slide 28

Slide 28 text

middleware = AutocommitMiddleware() middleware.process_view(request) ๏ Is this the correct interface for process_view? Wednesday, July 4, 12

Slide 29

Slide 29 text

Bound to Uncontrolled Interfaces (what happens when they change?) Wednesday, July 4, 12

Slide 30

Slide 30 text

class AutocommitMiddlewareTest: def setUp(self): assert 'AutocommitMiddleware' in INSTALLED_APPS def test_something_useful(self): with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, July 4, 12

Slide 31

Slide 31 text

Higher Level Tests are Slower But More “Correct” Wednesday, July 4, 12

Slide 32

Slide 32 text

No “Right” Way Wednesday, July 4, 12

Slide 33

Slide 33 text

Unit vs Integration Tests Wednesday, July 4, 12

Slide 34

Slide 34 text

@mock.patch('enter_transaction_management') def test_etm_called(self, etm): request = WsgiHttpRequest(method='POST') middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, July 4, 12

Slide 35

Slide 35 text

with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, July 4, 12

Slide 36

Slide 36 text

Which is better? (you can have both) Wednesday, July 4, 12

Slide 37

Slide 37 text

Mocking is Fragile Wednesday, July 4, 12

Slide 38

Slide 38 text

# what happens if the path changes? @mock.patch('enter_transaction_management') def test_something_useful(self, etm): Wednesday, July 4, 12

Slide 39

Slide 39 text

# what if we change the arguments passed? etm.assert_called_once_with(True) Wednesday, July 4, 12

Slide 40

Slide 40 text

Useful for Testing External Services (Twitter, Facebook, Disqus) Wednesday, July 4, 12

Slide 41

Slide 41 text

Limit What You Test Wednesday, July 4, 12

Slide 42

Slide 42 text

Record Live Data for Mocking Wednesday, July 4, 12

Slide 43

Slide 43 text

class TwitterTest: # This is not the Twitter API you are # looking for @mock.patch('twitter.Twitter.getresponse') def test_something_useful(self, gr): gr.return_value = TWITTER_RESPONSE res = Twitter().foo() assert res == [1, 2, 3] Wednesday, July 4, 12

Slide 44

Slide 44 text

Assume APIs Don’t Change (it’s mostly true) Wednesday, July 4, 12

Slide 45

Slide 45 text

Guarantees with High Level Smoke Tests Wednesday, July 4, 12

Slide 46

Slide 46 text

Test The Lifecycle of Requests Wednesday, July 4, 12

Slide 47

Slide 47 text

Selenium Kind of Works Wednesday, July 4, 12

Slide 48

Slide 48 text

Lesson #2: Don’t Admit Defeat Wednesday, July 4, 12

Slide 49

Slide 49 text

Start with a Goal Wednesday, July 4, 12

Slide 50

Slide 50 text

Goal #1: Write Testable Code Wednesday, July 4, 12

Slide 51

Slide 51 text

A lot of Software is Hard to Test (but yours doesn’t have to be) Wednesday, July 4, 12

Slide 52

Slide 52 text

def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, July 4, 12

Slide 53

Slide 53 text

Break Up Your Code Wednesday, July 4, 12

Slide 54

Slide 54 text

def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, July 4, 12

Slide 55

Slide 55 text

Goal #2: Start Writing Tests Wednesday, July 4, 12

Slide 56

Slide 56 text

Make Testing Easy Wednesday, July 4, 12

Slide 57

Slide 57 text

class AutocommitMiddlewareTest(DisqusTest): def test_changes_to_transactional_on_post(self): # adding a simple assert helper makes this # test much easier to write with self.assertInTransaction(): # changing this to an integration test # doesn’t require an understanding of # Django middleware internals self.client.post('/') Wednesday, July 4, 12

Slide 58

Slide 58 text

Create Structure Wednesday, July 4, 12

Slide 59

Slide 59 text

tests !"" __init__.py !"" integration # !"" __init__.py # !"" disqus # # !"" __init__.py # # !"" db # # # !"" __init__.py # # # !"" annotations # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" cachedbymodel # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" disqusmanager # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" iterablequeryset # # # # !"" __init__.py # # # # %"" tests.py Wednesday, July 4, 12

Slide 60

Slide 60 text

Document Best Practices Wednesday, July 4, 12

Slide 61

Slide 61 text

Wednesday, July 4, 12

Slide 62

Slide 62 text

Goal #3: Continuously Run Tests (via automation) Wednesday, July 4, 12

Slide 63

Slide 63 text

Wednesday, July 4, 12

Slide 64

Slide 64 text

Drive It Into Your Culture Wednesday, July 4, 12

Slide 65

Slide 65 text

Development Workflow Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, July 4, 12

Slide 66

Slide 66 text

Use Code Review Wednesday, July 4, 12

Slide 67

Slide 67 text

Wednesday, July 4, 12

Slide 68

Slide 68 text

Test Throughout The Process Wednesday, July 4, 12

Slide 69

Slide 69 text

Test Lifecycle Pre-Review Tests CI Smoke Tests Failed Build Commit Wednesday, July 4, 12

Slide 70

Slide 70 text

Tools of the Trade Wednesday, July 4, 12

Slide 71

Slide 71 text

Use the Right Tools (if they don’t exist, build them) Wednesday, July 4, 12

Slide 72

Slide 72 text

Nose A Better Test Runner Wednesday, July 4, 12

Slide 73

Slide 73 text

# dump XML results nosetests --with-xunit # drop into a PDB on errors and failures nosetests --pdb --pdb-failure Wednesday, July 4, 12

Slide 74

Slide 74 text

Works With Your Existing Tests Wednesday, July 4, 12

Slide 75

Slide 75 text

XML Reporting Wednesday, July 4, 12

Slide 76

Slide 76 text

Code Coverage Wednesday, July 4, 12

Slide 77

Slide 77 text

Test Debugging (PDB) Wednesday, July 4, 12

Slide 78

Slide 78 text

Timings Wednesday, July 4, 12

Slide 79

Slide 79 text

Plugins for (Almost) Everything Wednesday, July 4, 12

Slide 80

Slide 80 text

Coverage.py Record Code Coverage Wednesday, July 4, 12

Slide 81

Slide 81 text

# use ``coverage run`` instead of ``python`` coverage run setup.py test # generate html reports coverage html --include=disqus/sexyapi/* # realize how terrible your tests are open htmlcov/index.html Wednesday, July 4, 12

Slide 82

Slide 82 text

Wednesday, July 4, 12

Slide 83

Slide 83 text

Wednesday, July 4, 12

Slide 84

Slide 84 text

Sentry Exception Reporting Wednesday, July 4, 12

Slide 85

Slide 85 text

Because Tests Aren’t Enough Wednesday, July 4, 12

Slide 86

Slide 86 text

Wednesday, July 4, 12

Slide 87

Slide 87 text

Wednesday, July 4, 12

Slide 88

Slide 88 text

Easily Grab Context for Testing Wednesday, July 4, 12

Slide 89

Slide 89 text

Wednesday, July 4, 12

Slide 90

Slide 90 text

Simple Continuous Integration Wednesday, July 4, 12

Slide 91

Slide 91 text

Wednesday, July 4, 12

Slide 92

Slide 92 text

Test Every Commit Wednesday, July 4, 12

Slide 93

Slide 93 text

Pipeline Jobs for Best Results Wednesday, July 4, 12

Slide 94

Slide 94 text

Selenium api/* Unit Integration disqus-web admin/* analytics/* JavaScript Package Simple Build Distribution embed/* forums/* db/* Wednesday, July 4, 12

Slide 95

Slide 95 text

Code Review Done Right Wednesday, July 4, 12

Slide 96

Slide 96 text

Highly Integrated Review Process Wednesday, July 4, 12

Slide 97

Slide 97 text

Wednesday, July 4, 12

Slide 98

Slide 98 text

Automated Partial Test Suite Wednesday, July 4, 12

Slide 99

Slide 99 text

Wednesday, July 4, 12

Slide 100

Slide 100 text

Feedback Loop Wednesday, July 4, 12

Slide 101

Slide 101 text

Wednesday, July 4, 12

Slide 102

Slide 102 text

Gargoyle Selectively Enable Features in Code Wednesday, July 4, 12

Slide 103

Slide 103 text

Wednesday, July 4, 12

Slide 104

Slide 104 text

Silently Launch Features Eases Performance and Load Testing Wednesday, July 4, 12

Slide 105

Slide 105 text

We Dark Launched Realtime (minimal e ort for capacity testing) Wednesday, July 4, 12

Slide 106

Slide 106 text

It Also Failed Several Times (without a ecting anything) Wednesday, July 4, 12

Slide 107

Slide 107 text

Takeaways Wednesday, July 4, 12

Slide 108

Slide 108 text

Your Needs Will Vary Wednesday, July 4, 12

Slide 109

Slide 109 text

The Process is Evolving Wednesday, July 4, 12

Slide 110

Slide 110 text

Culture is Key Wednesday, July 4, 12

Slide 111

Slide 111 text

Just Do It? Wednesday, July 4, 12

Slide 112

Slide 112 text

Questions? Wednesday, July 4, 12

Slide 113

Slide 113 text

Obligatory Slide Full of Useful Links ๏ Sentry (exception reporting) github.com/dcramer/sentry ๏ Jenkins CI (continuous integration) jenkins-ci.org ๏ Phabricator (code review) phabricator.org ๏ Gargoyle (feature switches) github.com/disqus/gargoyle ๏ Mock mock.readthedocs.org ๏ Nose (test runner) nose.readthedocs.org ๏ nose-quickunit (test discovery) github.com/dcramer/nose-quickunit ๏ django-nose (Django integration) github.com/jbalogh/django-nose ๏ Coverage.py (code coverage) nedbatchelder.com/code/coverage/ Wednesday, July 4, 12