Lessons in Testing - EuroPython

Talk given at EuroPython 2012

Transcript

1. Lessons in Testing David Cramer twitter.com/zeeg Wednesday, July 4, 12
2. Who am I? Wednesday, July 4, 12
3. I lead Infrastructure at DISQUS Wednesday, July 4, 12
4. Wednesday, July 4, 12
5. DISQUS sees a lot of tra c Google Analytics: May 30 2012 - June 28 2012 Wednesday, July 4, 12
6. One of the largest consumers of Django Wednesday, July 4, 12
7. ... and Flask Wednesday, July 4, 12
8. We are a Growing Startup (less than 20 engineers) Wednesday, July 4, 12
9. Wednesday, July 4, 12
10. We are also terrible at testing (but we try not to be) Wednesday, July 4, 12
11. This presentation is about Lessons and Aspirations Wednesday, July 4, 12
12. Lessons Mostly Learned Wednesday, July 4, 12
13. Lesson #1: No Want Likes Writing Tests Wednesday, July 4, 12
14. Time Consuming to Write Wednesday, July 4, 12
15. Simple Feature: 10 lines of code from django.db import transaction class AutocommitMiddleware(object): def process_view(self, request, *a, **kw): if request.method == 'POST': enter_transaction_management(True) def process_response(self, request, response): leave_transaction_management() return response def process_exception(self, *a, **kw): leave_transaction_management() Wednesday, July 4, 12
16. with precision testing class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, July 4, 12
17. turns into 36 lines of tests import mock from mock_django.http import WsgiHttpRequest from django.http import HttpResponse from disqus.middleware.autocommit import AutocommitMiddleware from disqus.tests import DisqusTest class AutocommitMiddlewareTest(DisqusTest): @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_changes_to_transactional_on_post(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) enter_transaction_management.assert_called_once_with(True, using='default') @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_leaves_autocommit_on_get(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'GET' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) self.assertFalse(enter_transaction_management.called) @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_response(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' response = HttpResponse() middleware = AutocommitMiddleware() middleware.process_response(request, response) leave_transaction_management.assert_called_once_with(using='default') @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_exception(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' exception = Exception() middleware = AutocommitMiddleware() middleware.process_exception(request, exception) leave_transaction_management.assert_called_once_with(using='default') Wednesday, July 4, 12
18. ~50% of the time was writing tests (sometimes more) Wednesday, July 4, 12
19. Legacy (Untested) Code is Expensive Wednesday, July 4, 12
20. Add Tests for New Code Wednesday, July 4, 12
21. Add Tests for Regressions Wednesday, July 4, 12
22. Slow or Inaccurate (there can be only one) Wednesday, July 4, 12
23. Spend more time writing tests or more time running them Wednesday, July 4, 12
24. We Cater to Cost to Run Wednesday, July 4, 12
25. Hardware Costs Decrease (Engineers Don’t) Wednesday, July 4, 12
26. Interface Contracts Yield Inaccuracy Wednesday, July 4, 12
27. @mock.patch('enter_transaction_management') def test_something_useful(self, etm): # ... etm.assert_called_once_with(True) ๏ Are we calling enter_transaction_management correctly Wednesday, July 4, 12
28. middleware = AutocommitMiddleware() middleware.process_view(request) ๏ Is this the correct interface for process_view? Wednesday, July 4, 12
29. Bound to Uncontrolled Interfaces (what happens when they change?) Wednesday, July 4, 12
30. class AutocommitMiddlewareTest: def setUp(self): assert 'AutocommitMiddleware' in INSTALLED_APPS def test_something_useful(self): with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, July 4, 12
31. Higher Level Tests are Slower But More “Correct” Wednesday, July 4, 12
32. No “Right” Way Wednesday, July 4, 12
33. Unit vs Integration Tests Wednesday, July 4, 12
34. @mock.patch('enter_transaction_management') def test_etm_called(self, etm): request = WsgiHttpRequest(method='POST') middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, July 4, 12
35. with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, July 4, 12
36. Which is better? (you can have both) Wednesday, July 4, 12
37. Mocking is Fragile Wednesday, July 4, 12
38. # what happens if the path changes? @mock.patch('enter_transaction_management') def test_something_useful(self, etm): Wednesday, July 4, 12
39. # what if we change the arguments passed? etm.assert_called_once_with(True) Wednesday, July 4, 12
40. Useful for Testing External Services (Twitter, Facebook, Disqus) Wednesday, July 4, 12
41. Limit What You Test Wednesday, July 4, 12
42. Record Live Data for Mocking Wednesday, July 4, 12
43. class TwitterTest: # This is not the Twitter API you are # looking for @mock.patch('twitter.Twitter.getresponse') def test_something_useful(self, gr): gr.return_value = TWITTER_RESPONSE res = Twitter().foo() assert res == [1, 2, 3] Wednesday, July 4, 12
44. Assume APIs Don’t Change (it’s mostly true) Wednesday, July 4, 12
45. Guarantees with High Level Smoke Tests Wednesday, July 4, 12
46. Test The Lifecycle of Requests Wednesday, July 4, 12
47. Selenium Kind of Works Wednesday, July 4, 12
48. Lesson #2: Don’t Admit Defeat Wednesday, July 4, 12
49. Start with a Goal Wednesday, July 4, 12
50. Goal #1: Write Testable Code Wednesday, July 4, 12
51. A lot of Software is Hard to Test (but yours doesn’t have to be) Wednesday, July 4, 12
52. def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, July 4, 12
53. Break Up Your Code Wednesday, July 4, 12
54. def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, July 4, 12
55. Goal #2: Start Writing Tests Wednesday, July 4, 12
56. Make Testing Easy Wednesday, July 4, 12
57. class AutocommitMiddlewareTest(DisqusTest): def test_changes_to_transactional_on_post(self): # adding a simple assert helper makes this # test much easier to write with self.assertInTransaction(): # changing this to an integration test # doesn’t require an understanding of # Django middleware internals self.client.post('/') Wednesday, July 4, 12
58. Create Structure Wednesday, July 4, 12
59. tests !"" __init__.py !"" integration # !"" __init__.py # !"" disqus # # !"" __init__.py # # !"" db # # # !"" __init__.py # # # !"" annotations # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" cachedbymodel # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" disqusmanager # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" iterablequeryset # # # # !"" __init__.py # # # # %"" tests.py Wednesday, July 4, 12
60. Document Best Practices Wednesday, July 4, 12
61. Wednesday, July 4, 12
62. Goal #3: Continuously Run Tests (via automation) Wednesday, July 4, 12
63. Wednesday, July 4, 12
64. Drive It Into Your Culture Wednesday, July 4, 12
65. Development Workflow Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, July 4, 12
66. Use Code Review Wednesday, July 4, 12
67. Wednesday, July 4, 12
68. Test Throughout The Process Wednesday, July 4, 12
69. Test Lifecycle Pre-Review Tests CI Smoke Tests Failed Build Commit Wednesday, July 4, 12
70. Tools of the Trade Wednesday, July 4, 12
71. Use the Right Tools (if they don’t exist, build them) Wednesday, July 4, 12
72. Nose A Better Test Runner Wednesday, July 4, 12
73. # dump XML results nosetests --with-xunit # drop into a PDB on errors and failures nosetests --pdb --pdb-failure Wednesday, July 4, 12
74. Works With Your Existing Tests Wednesday, July 4, 12
75. XML Reporting Wednesday, July 4, 12
76. Code Coverage Wednesday, July 4, 12
77. Test Debugging (PDB) Wednesday, July 4, 12
78. Timings Wednesday, July 4, 12
79. Plugins for (Almost) Everything Wednesday, July 4, 12
80. Coverage.py Record Code Coverage Wednesday, July 4, 12
81. # use ``coverage run`` instead of ``python`` coverage run setup.py test # generate html reports coverage html --include=disqus/sexyapi/* # realize how terrible your tests are open htmlcov/index.html Wednesday, July 4, 12
82. Wednesday, July 4, 12
83. Wednesday, July 4, 12
84. Sentry Exception Reporting Wednesday, July 4, 12
85. Because Tests Aren’t Enough Wednesday, July 4, 12
86. Wednesday, July 4, 12
87. Wednesday, July 4, 12
88. Easily Grab Context for Testing Wednesday, July 4, 12
89. Wednesday, July 4, 12
90. Simple Continuous Integration Wednesday, July 4, 12
91. Wednesday, July 4, 12
92. Test Every Commit Wednesday, July 4, 12
93. Pipeline Jobs for Best Results Wednesday, July 4, 12
94. Selenium api/* Unit Integration disqus-web admin/* analytics/* JavaScript Package Simple Build Distribution embed/* forums/* db/* Wednesday, July 4, 12
95. Code Review Done Right Wednesday, July 4, 12
96. Highly Integrated Review Process Wednesday, July 4, 12
97. Wednesday, July 4, 12
98. Automated Partial Test Suite Wednesday, July 4, 12
99. Wednesday, July 4, 12
100. Feedback Loop Wednesday, July 4, 12
101. Wednesday, July 4, 12
102. Gargoyle Selectively Enable Features in Code Wednesday, July 4, 12
103. Wednesday, July 4, 12
104. Silently Launch Features Eases Performance and Load Testing Wednesday, July 4, 12
105. We Dark Launched Realtime (minimal e ort for capacity testing) Wednesday, July 4, 12
106. It Also Failed Several Times (without a ecting anything) Wednesday, July 4, 12
107. Takeaways Wednesday, July 4, 12
108. Your Needs Will Vary Wednesday, July 4, 12
109. The Process is Evolving Wednesday, July 4, 12
110. Culture is Key Wednesday, July 4, 12
111. Just Do It? Wednesday, July 4, 12
112. Questions? Wednesday, July 4, 12
113. Obligatory Slide Full of Useful Links ๏ Sentry (exception reporting) github.com/dcramer/sentry ๏ Jenkins CI (continuous integration) jenkins-ci.org ๏ Phabricator (code review) phabricator.org ๏ Gargoyle (feature switches) github.com/disqus/gargoyle ๏ Mock mock.readthedocs.org ๏ Nose (test runner) nose.readthedocs.org ๏ nose-quickunit (test discovery) github.com/dcramer/nose-quickunit ๏ django-nose (Django integration) github.com/jbalogh/django-nose ๏ Coverage.py (code coverage) nedbatchelder.com/code/coverage/ Wednesday, July 4, 12