Lessons in Testing - DjangoCon 2012

Transcript

1. Lessons in Testing David Cramer twitter.com/zeeg Wednesday, September 5, 12
2. Who am I? Wednesday, September 5, 12
3. I lead Infrastructure at DISQUS Wednesday, September 5, 12
4. Wednesday, September 5, 12
5. DISQUS sees a lot of tra c Google Analytics: Aug 2 2012 - Sep 1 2012 Wednesday, September 5, 12
6. Lots of Django (and things like Flask) Wednesday, September 5, 12
7. Growing Startup (less than 20 engineers) Wednesday, September 5, 12
8. Wednesday, September 5, 12
9. We are also terrible at testing (but we try not to be) Wednesday, September 5, 12
10. This presentation is about Lessons and Aspirations Wednesday, September 5, 12
11. Lessons Mostly Learned Wednesday, September 5, 12
12. Lesson #1: No Want Likes Writing Tests Wednesday, September 5, 12
13. Time Consuming to Write Wednesday, September 5, 12
14. Simple Feature: 10 lines of code from django.db import transaction class AutocommitMiddleware(object): def process_view(self, request, *a, **kw): if request.method == 'POST': enter_transaction_management(True) def process_response(self, request, response): leave_transaction_management() return response def process_exception(self, *a, **kw): leave_transaction_management() Wednesday, September 5, 12
15. with precision testing class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12
16. turns into 36 lines of tests import mock from mock_django.http import WsgiHttpRequest from django.http import HttpResponse from disqus.middleware.autocommit import AutocommitMiddleware from disqus.tests import DisqusTest class AutocommitMiddlewareTest(DisqusTest): @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_changes_to_transactional_on_post(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) enter_transaction_management.assert_called_once_with(True, using='default') @mock.patch('disqus.middleware.autocommit.transaction.enter_transaction_management') def test_leaves_autocommit_on_get(self, enter_transaction_management): request = WsgiHttpRequest() request.method = 'GET' middleware = AutocommitMiddleware() middleware.process_view(request, lambda: "", [], {}) self.assertFalse(enter_transaction_management.called) @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_response(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' response = HttpResponse() middleware = AutocommitMiddleware() middleware.process_response(request, response) leave_transaction_management.assert_called_once_with(using='default') @mock.patch('disqus.middleware.autocommit.transaction.leave_transaction_management') def test_changes_to_autocommit_on_exception(self, leave_transaction_management): request = WsgiHttpRequest() request.method = 'GET' exception = Exception() middleware = AutocommitMiddleware() middleware.process_exception(request, exception) leave_transaction_management.assert_called_once_with(using='default') Wednesday, September 5, 12
17. ~50% of the time was writing tests (often more) Wednesday, September 5, 12
18. Legacy (Untested) Code is Expensive Wednesday, September 5, 12
19. Add Tests for New Code Wednesday, September 5, 12
20. Add Tests for Regressions Wednesday, September 5, 12
21. You Pick: Fast or Accurate (you only get to have one) Wednesday, September 5, 12
22. Spend more time writing tests or more time running them Wednesday, September 5, 12
23. You Can Buy Faster Hardware Wednesday, September 5, 12
24. class AutocommitMiddlewareTest: @mock.patch('enter_transaction_management') def test_something_precise(self, etm): request = WsgiHttpRequest() request.method = 'POST' middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12
25. class AutocommitMiddlewareTest: def setUp(self): assert 'AutocommitMiddleware' in INSTALLED_APPS def test_something_useful(self): with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, September 5, 12
26. Higher Level Tests are Slower But More “Correct” Wednesday, September 5, 12
27. Unit vs Integration Tests Wednesday, September 5, 12
28. @mock.patch('enter_transaction_management') def test_etm_called(self, etm): request = WsgiHttpRequest(method='POST') middleware = AutocommitMiddleware() middleware.process_view(request) etm.assert_called_once_with(True) Wednesday, September 5, 12
29. with self.assertInTransaction(): # use Django’s integrated test client self.client.post('/') Wednesday, September 5, 12
30. Which is better? (you should have both) Wednesday, September 5, 12
31. Mocking is Fragile Wednesday, September 5, 12
32. # what happens if the path changes? @mock.patch('enter_transaction_management') def test_something_useful(self, etm): Wednesday, September 5, 12
33. # what if we change the arguments passed? etm.assert_called_once_with(True) Wednesday, September 5, 12
34. Useful for Testing External Services (Twitter, Facebook, Disqus) Wednesday, September 5, 12
35. Capture Live Data for Mocking Wednesday, September 5, 12
36. class TwitterTest: @mock.patch('twitter.Twitter.getresponse') def test_something_useful(self, gr): gr.return_value = TWITTER_RESPONSE res = Twitter().foo() assert res == [1, 2, 3] Wednesday, September 5, 12
37. Assume APIs Don’t Change (it’s mostly true) Wednesday, September 5, 12
38. Test The Lifecycle of Requests Wednesday, September 5, 12
39. Selenium Kind of Works (but it’s extremely brittle) Wednesday, September 5, 12
40. Lesson #2: Don’t Admit Defeat Wednesday, September 5, 12
41. Start with a Goal Wednesday, September 5, 12
42. Goal #1: Write Testable Code Wednesday, September 5, 12
43. A lot of Code is Hard to Test (but it doesn’t have to be) Wednesday, September 5, 12
44. def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, September 5, 12
45. Break Up Your Code Wednesday, September 5, 12
46. def static_media(request, path): """ Serve static files below a given point in the directory structure. """ from django.utils.http import http_date from django.views.static import was_modified_since import mimetypes import os.path import posixpath import stat import urllib document_root = os.path.join(settings.STATIC_ROOT) path = posixpath.normpath(urllib.unquote(path)) path = path.lstrip('/') newpath = '' for part in path.split('/'): if not part: # Strip empty path components. continue drive, part = os.path.splitdrive(part) head, part = os.path.split(part) if part in (os.curdir, os.pardir): # Strip '.' and '..' in path. continue newpath = os.path.join(newpath, part).replace('\\', '/') if newpath and path != newpath: return HttpResponseRedirect(newpath) fullpath = os.path.join(document_root, newpath) if os.path.isdir(fullpath): raise Http404("Directory indexes are not allowed here.") if not os.path.exists(fullpath): raise Http404('"%s" does not exist' % fullpath) # Respect the If-Modified-Since header. statobj = os.stat(fullpath) mimetype = mimetypes.guess_type(fullpath)[0] or 'application/octet-stream' if not was_modified_since(request.META.get('HTTP_IF_MODIFIED_SINCE'), statobj[stat.ST_MTIME], statobj[stat.ST_SIZE]): return HttpResponseNotModified(mimetype=mimetype) contents = open(fullpath, 'rb').read() response = HttpResponse(contents, mimetype=mimetype) response["Last-Modified"] = http_date(statobj[stat.ST_MTIME]) response["Content-Length"] = len(contents) return response Wednesday, September 5, 12
47. Goal #2: Start Writing Tests Wednesday, September 5, 12
48. Make Testing Easy Wednesday, September 5, 12
49. class AutocommitMiddlewareTest(DisqusTest): def test_changes_to_transactional_on_post(self): # adding a simple assert helper makes this # test much easier to write with self.assertInTransaction(): # changing this to an integration test # doesn’t require an understanding of # Django middleware internals self.client.post('/') Wednesday, September 5, 12
50. Create Structure Wednesday, September 5, 12
51. tests !"" __init__.py !"" integration # !"" __init__.py # !"" disqus # # !"" __init__.py # # !"" db # # # !"" __init__.py # # # !"" annotations # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" cachedbymodel # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" disqusmanager # # # # !"" __init__.py # # # # !"" models.py # # # # %"" tests.py # # # !"" iterablequeryset # # # # !"" __init__.py # # # # %"" tests.py Wednesday, September 5, 12
52. Document Best Practices Wednesday, September 5, 12
53. Wednesday, September 5, 12
54. Prevent Mistakes Wednesday, September 5, 12
55. NOSE_PLUGINS = [ 'disqus.tests.plugins.redis.RedisPlugin', 'disqus.tests.plugins.sockets.ErroringSocketWhitelistPlugin', 'disqus.tests.plugins.impermium.MockImpermiumPlugin', 'disqus.tests.plugins.mail.FixMailHostnamePlugin', 'disqus.tests.plugins.compat.UnitTestPlugin', 'disqus.tests.plugins.tags.TagSelector', 'disqus.tests.plugins.timing.TimingPlugin', 'disqus.tests.plugins.themes.DefaultThemePlugin', 'disqus.tests.plugins.apps.AddInstalledApps', 'disqus.tests.plugins.selenium.ScreenshotFailurePlugin', 'disqus.tests.plugins.tasks.SeppukuTasksPlugin', 'disqus.tests.plugins.django.DjangoSetUpPlugin', ] Wednesday, September 5, 12
56. Goal #3: Continuously Run Tests (via automation) Wednesday, September 5, 12
57. Wednesday, September 5, 12
58. Drive It Into Your Culture Wednesday, September 5, 12
59. Development Workflow Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, September 5, 12
60. Use Code Review Wednesday, September 5, 12
61. Wednesday, September 5, 12
62. Test Throughout The Process Wednesday, September 5, 12
63. Test Lifecycle Pre-Review Tests CI Smoke Tests Failed Build Commit Wednesday, September 5, 12
64. Tools of the Trade Wednesday, September 5, 12
65. Use the Right Tools (if they don’t exist, build them) Wednesday, September 5, 12
66. Nose A Better Test Runner nose.readthedocs.org Wednesday, September 5, 12
67. Plugins for (Almost) Everything Wednesday, September 5, 12
68. # dump XML results nosetests --with-xunit # drop into a PDB on errors and failures nosetests --pdb --pdb-failure Wednesday, September 5, 12
69. Coverage.py Find Untested Code nedbatchelder.com/code/coverage Wednesday, September 5, 12
70. Wednesday, September 5, 12
71. Wednesday, September 5, 12
72. Sentry Exception Reporting github.com/getsentry/sentry Wednesday, September 5, 12
73. Because Tests Aren’t Enough Wednesday, September 5, 12
74. Wednesday, September 5, 12
75. Wednesday, September 5, 12
76. Simple Continuous Integration jenkins-ci.org Wednesday, September 5, 12
77. A Workflow With CI Review Integration Deploy Failed Build Reporting Rollback Commit Wednesday, September 5, 12
78. Selenium api/* Unit Integration disqus-web admin/* analytics/* JavaScript Package Simple Build Distribution embed/* forums/* db/* Performance Wednesday, September 5, 12
79. Code Review Done Right phabricator.org Wednesday, September 5, 12
80. Highly Integrated Review Process Wednesday, September 5, 12
81. Wednesday, September 5, 12
82. Wednesday, September 5, 12
83. Gargoyle Selectively Enable Features in Code github.com/disqus/gargoyle Wednesday, September 5, 12
84. Wednesday, September 5, 12
85. from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('my_awesome_switch', request): # do something awesome else: # do something less awesome Wednesday, September 5, 12
86. Zumanji Performance Testing github.com/disqus/zumanji Wednesday, September 5, 12
87. Wednesday, September 5, 12
88. Wednesday, September 5, 12
89. Still a W.I.P. Wednesday, September 5, 12
90. Takeaways Wednesday, September 5, 12
91. Your Needs Will Vary Wednesday, September 5, 12
92. The Process is Evolving Wednesday, September 5, 12
93. Culture is Key Wednesday, September 5, 12
94. Go Write Tests! Wednesday, September 5, 12
95. Questions? Wednesday, September 5, 12
96. Obligatory Slide Full of Useful Links ๏ Sentry (exception reporting) github.com/dcramer/sentry ๏ Jenkins CI (continuous integration) jenkins-ci.org ๏ Phabricator (code review) phabricator.org ๏ Gargoyle (feature switches) github.com/disqus/gargoyle ๏ Zumanji (performance tests) github.com/disqus/zumanji ๏ Mock mock.readthedocs.org ๏ Nose (test runner) nose.readthedocs.org ๏ nose-quickunit (test discovery) github.com/dcramer/nose-quickunit ๏ django-nose (Django integration) github.com/jbalogh/django-nose ๏ Coverage.py (code coverage) nedbatchelder.com/code/coverage/ Wednesday, September 5, 12