Test Everything

Testing in Django is easy if you're testing models against your database. What happens when you need to test other-systems, like a search engine, or an API? This tutorial will cover how I built SphinxTestCase, ESTestCase and redisutils to allow us to maintain test coverage on our web sites.

Transcript

1. Test Anything and Everything dave dash DJANGOCON 2011 1
2. me (the djangonaut) sr web developer @ mozilla helped move mozilla webdev to Django worked on Firefox Add-ons Technical lead for Firefox Input I work on internal libraries, tools, and some smaller sites. i blog @ davedash.com i email @ dd@davedash.com i tweet @davedash 2
3. mozilla webdev pronounced mo-zilla 40+ people 100+ web sites we happily churn out PEP-8 compliant code Django shop since early 2010 we create and support most web sites *.mozilla.com / *.mozilla.org Mozilla Developer Network Firefox Add-ons Firefox Support Mozilla.org and more... actively hiring, if this sort of thing appeals to you 3
4. What we’ll cover philosophy of testing ./manage.py test Writing TestCases Code coverage Testing difficult things in the django test suite • • • • • 4
5. What we won’t cover Systems testing Failure testing In browser-testing Load testing • • • • 5
6. Follow along Etherpad - http://mzl.la/django-test +1 any interesting topics List any testing issues or general questions you might have • • • 6
7. Introduce yourself Name Involvement with Django Testing experience What you want to get out of this morning • • • • 7
8. Agenda I’ll start a fight Testing @ Mozilla Interactive stuff How Django’s testing works How we can make testing better Testing beyond the DB Gotchas 8
9. please... Ask questions. This is 3 hours... I can’t fill it all with slides. Correct me if I’m wrong. • • 9
10. Questions? 10
11. when to test? 11
12. never 12
13. but... you told us to test 13
14. you convinced us it was good 14
15. “Instead of writing tests I try to be extremely careful in coding, and keep the code size small so I continue to understand it.” -maciej ceglowski, pinboard.in 15
16. it all depends 16
17. what am I doing? 17
18. know your app 18
19. but... 19
20. when to test 20
21. when others use your code 21
22. libraries 22
23. projects? 23
24. big sites 24
25. teams 25
26. pull requests 26
27. no tests, no merge 27
28. failed tests... no merge 28
29. Testing Django @ Mozilla 29
30. 2009/2010 30
31. Python Revolution at Mozilla 31
32. previously... 32
33. Banging our heads 33
34. PHP apps with no test framework 34
35. Django’s Test framework == Appealing 35
36. today... 36
37. Firefox Add-ons: 2500+ tests 37
38. Firefox Support: 1100 Tests 38
39. Coverage: 90%+ 39
40. Tons of smaller Django sites 40
41. All expected to be well tested 41
42. At least 80-90% 42
43. Some things aren’t worth testing 43
44. What does 5000 tests buy you? 44
45. Problems 45
46. Too many tests per file 46
47. Tests take too long to run 47
48. Failing tests don’t get fixed 48
49. avoid testing external systems 49
50. Vagrant 50
51. vagrant up
52. regular django testing
53. cd ~/fxinput_nohacks ./manage.py test
55. How testing works in Django 55
56. The Test Runner 56
57. Creates Database 57
58. No Data... to start 58
59. Finds tests 59
60. in models.py and tests.py 60
61. runs the tests 61
62. runs non-Class tests 62
63. runs any doctests 63
65. Class tests 65
66. Class Setup 66
67. for each test... 67
68. Setup 68
69. Load a fixture 69
71. Run Test 71
72. Teardown 72
73. after all tests in a class... 73
74. class teardown 74
76. then we get our report card 76
77. “F” means bad 77
78. “.” means good 78
79. “E” doesn’t mean effort 79
80. Now that we know how it works... 80
81. WE CAN HACK IT 81
82. hacking testing 82
83. 2,500 tests is a lot 83
84. Multiple Files 84
86. be messy! 86
88. test_utils 88
89. be real messy! 89
93. django-nose + test_utils 93
94. organize your mess 94
95. load a set of fixtures once 95
96. run all the test-cases 96
97. DRY... for fixtures 97
98. 98 class TestCase1(TestCase): fixtures = ['a'] class TestCase2(TestCase): fixtures = ['a', 'b'] class TestCase3(TestCase): fixtures = ['a'] class TestCase4(TestCase): fixtures = ['a', 'c'] class TestCase5(TestCase): fixtures = ['a'] class TestCase6(TestCase): fixtures = ['a', 'c']
99. django-nose + test_utils 99
100. fixture misers 100
101. model maker 101
102. Useful testing tools 102
103. nose (via django-nose) 103
104. nose finds test 104
106. it has nice tools 106
107. raise SkipTest 107
109. eq_(x, y, “x ain’t y”) 109
110. --failed 110
112. nose has plugins 112
113. 113 ................................S....................................................F................................................................ ====================================================================== FAIL: Verify the "next" pagination link appears and directs the user to the ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 ---------------------------------------------------------------------- Ran 150 tests in 32.504s FAILED (SKIP=1, failures=1)
114. nicedots 114
115. 115 apps/input/tests/test_middleware.py:MiddlewareTests ........... apps/input/tests/test_redirects.py:RedirectTests ... apps/myadmin/tests.py:ViewTestCase ...... apps/search/tests/test_client.py:SearchTest .............. apps/search/tests/test_client.py .. ====================================================================== FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 apps/search/tests/test_dashboard.py:TestDashboard .. apps/search/tests/test_dashboard.py:TestHelpers ........ apps/search/tests/test_dashboard.py:TestMobileDashboard . apps/search/tests/test_elastic.py:TestElastic ..
116. nose-progressive 116
118. do not mix those two 118
119. coverage 119
120. coverage ./manage.py test 120
121. coverage report -m 121
122. 122 Name Stmts Miss Cover Missing ---------------------------------------------------------------- apps/search/__init__ 0 0 100% apps/search/client 193 4 98% 96-97, 134-137 apps/search/context_processors 5 0 100% apps/search/cron 13 13 0% 1-24 apps/search/forms 70 2 97% 131, 136 apps/search/helpers 136 1 99% 154 apps/search/models 0 0 100% apps/search/tasks 8 0 100% apps/search/tests/__init__ 26 1 96% 27 apps/search/tests/test_client 75 0 100% apps/search/tests/test_dashboard 105 1 99% 58 apps/search/tests/test_elastic 15 0 100% apps/search/tests/test_views 193 0 100% apps/search/urls 3 0 100% apps/search/utils 10 0 100% apps/search/views 142 1 99% 166 ---------------------------------------------------------------- TOTAL 994 23 98%
123. jenkins! 123
124. formerly hudson 124
125. jenkins.mozilla.org 125
126. runs after each commit 126
127. Testing Everything 127
128. not 100% coverage 128
129. 80-90% is okay 129
130. “everything?” 130
131. Test your entire site 131
132. not just your database 132
133. test your search engines 133
134. your weird REST API 134
135. that kind of “everything” 135
136. good coverage on tricky things 136
137. some coverage on everything 137
138. code coverage isn’t everything 138
139. subclass me 139
140. APIs 140
141. Search 141
142. key-value stores 142
143. anything... 143
144. you can do cool things... 144
145. assert! 145
146. results are in order 146
147. 147 class RankingTest(SphinxTestCase): """This test assures that we don't regress our rankings.""" fixtures = ('base/users', 'base/addon_1833_yoono', 'base/addon_9825_fastestfox', 'base/addon_5579', 'base/addon_personas-plus', ) def test_twitter(self): """ Search for twitter should yield Yoono before FastestFox since Yoono has "twitter" in it's name field. """ r = query('twitter') eq_(r[0].id, 1833) eq_(r[1].id, 9825) def test_cool(self): """Search for cool should return CoolIris before PersonasPlus.""" r = query('cool') eq_(r[0].slug, 'cooliris') eq_(r[1].slug, 'personas-plus')
148. FireFox == fire fox 148
149. saves headaches 149
150. Mock 150
151. Mock Redis 151
152. 152 if not connections: # don't set this repeatedly for alias, backend in settings.REDIS_BACKENDS.items(): _, server, params = parse_backend_uri(backend) try: socket_timeout = float(params.pop('socket_timeout')) except (KeyError, ValueError): socket_timeout = None password = params.pop('password', None) if ':' in server: host, port = server.split(':') try: port = int(port) except (ValueError, TypeError): port = 6379 else: host = 'localhost' port = 6379 connections[alias] = redislib.Redis(host=host, port=port, db=0, password=password, socket_timeout=socket_timeout) def mock_redis(): ret = dict(connections) for key in connections: connections[key] = MockRedis() return ret
153. Redis <> testing requirement 153
154. the mock client 154
155. doesn’t do everything 155
156. does what I’m testing for 156
157. could get built out more... 157
158. Setup & Teardown 158
159. Django’s setup 159
160. Loads fixtures 160
161. Gives you a clean slate 161
162. make clean slates 162
163. search, ldap and others 163
164. destroy the data-store 164
165. load data-fixtures 165
166. wrap this in a subclass of TestCase 166
167. 167 class SphinxTestCase(TestCase): """ This test case type can setUp and tearDown the sphinx daemon. Use this when testing any feature that requires sphinx. """ fixtures = ['users.json', 'search/documents.json', 'posts.json', 'questions.json'] @classmethod def setup_class(cls): super(SphinxTestCase, cls).setup_class() if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER: raise SkipTest() os.environ['DJANGO_ENVIRONMENT'] = 'test' if os.path.exists(settings.TEST_SPHINX_PATH): shutil.rmtree(settings.TEST_SPHINX_PATH) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc')) reindex() start_sphinx() time.sleep(1) @classmethod def teardown_class(cls): stop_sphinx() super(SphinxTestCase, cls).teardown_class()
168. your coworkers will thank you 168
169. “But I don’t work on search” 169
170. raise SkipTest 170
171. comes with nose 171
172. you can sniff settings 172
173. Built this for Sphinx Search 173
174. Elastic Search 174
175. LDAP 175
176. Extras 176
177. fixture-magic 177
178. dj custom-dump addon 3615 178
179. the Model Maker pattern 179
180. be careful of dates 180
181. use PDB for snooping 181
182. use ipython+pdb 182
183. alias i from IPython.Shell import IPShellEmbed as IPSh; IPSh(argv='') () > ~/.pdbrc 183
184. I’m looking for new coworkers Mozilla is a fun place to work It’s more exciting than these slides We are a developer friendly python shop We build things and share them • • • • 184
185. Perks desks computers chairs (if you want) work remotely fantastic coworkers <= seriously • • • • • 185