Test Everything

Testing in Django is easy if you're testing models against your database. What happens when you need to test other-systems, like a search engine, or an API? This tutorial will cover how I built SphinxTestCase, ESTestCase and redisutils to allow us to maintain test coverage on our web sites.

Transcript

1. Test Anything and Everything
   dave dash
   DJANGOCON 2011
   1
   

   View Slide

2. me (the djangonaut)
   sr web developer @ mozilla
   helped move mozilla webdev to Django
   worked on Firefox Add-ons
   Technical lead for Firefox Input
   I work on internal libraries, tools, and some
   smaller sites.
   i blog @ davedash.com
   i email @ dd@davedash.com
   i tweet @davedash
   2
   

   View Slide

3. mozilla webdev
   pronounced mo-zilla
   40+ people
   100+ web sites
   we happily churn out PEP-8 compliant code
   Django shop since early 2010
   we create and support most web sites
   *.mozilla.com / *.mozilla.org
   Mozilla Developer Network
   Firefox Add-ons
   Firefox Support
   Mozilla.org
   and more...
   actively hiring, if this sort of thing appeals to
   you
   3
   

   View Slide

4. What we’ll cover
   philosophy of testing
   ./manage.py test
   Writing TestCases
   Code coverage
   Testing difficult things in the django test suite
   •
   •
   •
   •
   •
   4
   

   View Slide

5. What we won’t cover
   Systems testing
   Failure testing
   In browser-testing
   Load testing
   •
   •
   •
   •
   5
   

   View Slide

6. Follow along
   Etherpad - http://mzl.la/django-test
   +1 any interesting topics
   List any testing issues or general questions you might have
   •
   •
   •
   6
   

   View Slide

7. Introduce yourself
   Name
   Involvement with Django
   Testing experience
   What you want to get out of this morning
   •
   •
   •
   •
   7
   

   View Slide

8. Agenda
   I’ll start a fight
   Testing @ Mozilla
   Interactive stuff
   How Django’s testing works
   How we can make testing better
   Testing beyond the DB
   Gotchas
   8
   

   View Slide

9. please...
   Ask questions. This is 3 hours... I can’t fill it all with slides.
   Correct me if I’m wrong.
   •
   •
   9
   

   View Slide

10. Questions?
    10
    

    View Slide

11. when to test?
    11
    

    View Slide

12. never
    12
    

    View Slide

13. but... you told us to test
    13
    

    View Slide

14. you convinced us it was good
    14
    

    View Slide

15. “Instead of writing tests I try to be
    extremely careful in coding, and
    keep the code size small so I
    continue to understand it.”
    -maciej ceglowski, pinboard.in
    15
    

    View Slide

16. it all depends
    16
    

    View Slide

17. what am I doing?
    17
    

    View Slide

18. know your app
    18
    

    View Slide

19. but...
    19
    

    View Slide

20. when to test
    20
    

    View Slide

21. when others use your code
    21
    

    View Slide

22. libraries
    22
    

    View Slide

23. projects?
    23
    

    View Slide

24. big sites
    24
    

    View Slide

25. teams
    25
    

    View Slide

26. pull requests
    26
    

    View Slide

27. no tests, no merge
    27
    

    View Slide

28. failed tests... no merge
    28
    

    View Slide

29. Testing Django @ Mozilla
    29
    

    View Slide

30. 2009/2010
    30
    

    View Slide

31. Python Revolution at Mozilla
    31
    

    View Slide

32. previously...
    32
    

    View Slide

33. Banging our heads
    33
    

    View Slide

34. PHP apps with no test framework
    34
    

    View Slide

35. Django’s Test framework
    ==
    Appealing
    35
    

    View Slide

36. today...
    36
    

    View Slide

37. Firefox Add-ons: 2500+ tests
    37
    

    View Slide

38. Firefox Support: 1100 Tests
    38
    

    View Slide

39. Coverage: 90%+
    39
    

    View Slide

40. Tons of smaller Django sites
    40
    

    View Slide

41. All expected to be well tested
    41
    

    View Slide

42. At least 80-90%
    42
    

    View Slide

43. Some things aren’t worth testing
    43
    

    View Slide

44. What does 5000 tests buy you?
    44
    

    View Slide

45. Problems
    45
    

    View Slide

46. Too many tests per file
    46
    

    View Slide

47. Tests take too long to run
    47
    

    View Slide

48. Failing tests don’t get fixed
    48
    

    View Slide

49. avoid testing external systems
    49
    

    View Slide

50. Vagrant
    50
    

    View Slide

51. vagrant up
    

    View Slide

52. regular django testing
    

    View Slide

53. cd ~/fxinput_nohacks
    ./manage.py test
    

    View Slide

54. View Slide

55. How testing works in Django
    55
    

    View Slide

56. The Test Runner
    56
    

    View Slide

57. Creates Database
    57
    

    View Slide

58. No Data... to start
    58
    

    View Slide

59. Finds tests
    59
    

    View Slide

60. in models.py and tests.py
    60
    

    View Slide

61. runs the tests
    61
    

    View Slide

62. runs non-Class tests
    62
    

    View Slide

63. runs any doctests
    63
    

    View Slide

64. View Slide

65. Class tests
    65
    

    View Slide

66. Class Setup
    66
    

    View Slide

67. for each test...
    67
    

    View Slide

68. Setup
    68
    

    View Slide

69. Load a fixture
    69
    

    View Slide

70. View Slide

71. Run Test
    71
    

    View Slide

72. Teardown
    72
    

    View Slide

73. after all tests in a class...
    73
    

    View Slide

74. class teardown
    74
    

    View Slide

75. View Slide

76. then we get our report card
    76
    

    View Slide

77. “F” means bad
    77
    

    View Slide

78. “.” means good
    78
    

    View Slide

79. “E” doesn’t mean effort
    79
    

    View Slide

80. Now that we know how it works...
    80
    

    View Slide

81. WE CAN HACK IT
    81
    

    View Slide

82. hacking testing
    82
    

    View Slide

83. 2,500 tests is a lot
    83
    

    View Slide

84. Multiple Files
    84
    

    View Slide

85. View Slide

86. be messy!
    86
    

    View Slide

87. View Slide

88. test_utils
    88
    

    View Slide

89. be real messy!
    89
    

    View Slide

90. View Slide

91. View Slide

92. View Slide

93. django-nose + test_utils
    93
    

    View Slide

94. organize your mess
    94
    

    View Slide

95. load a set of fixtures once
    95
    

    View Slide

96. run all the test-cases
    96
    

    View Slide

97. DRY... for fixtures
    97
    

    View Slide

98. 98
    class TestCase1(TestCase):
    fixtures = ['a']
    class TestCase2(TestCase):
    fixtures = ['a', 'b']
    class TestCase3(TestCase):
    fixtures = ['a']
    class TestCase4(TestCase):
    fixtures = ['a', 'c']
    class TestCase5(TestCase):
    fixtures = ['a']
    class TestCase6(TestCase):
    fixtures = ['a', 'c']
    

    View Slide

99. django-nose + test_utils
    99
    

    View Slide

100. fixture misers
     100
     

     View Slide

101. model maker
     101
     

     View Slide

102. Useful testing tools
     102
     

     View Slide

103. nose (via django-nose)
     103
     

     View Slide

104. nose finds test
     104
     

     View Slide

105. View Slide

106. it has nice tools
     106
     

     View Slide

107. raise SkipTest
     107
     

     View Slide

108. View Slide

109. eq_(x, y, “x ain’t y”)
     109
     

     View Slide

110. --failed
     110
     

     View Slide

111. View Slide

112. nose has plugins
     112
     

     View Slide

113. 113
     ................................S....................................................F................................................................
     ======================================================================
     FAIL: Verify the "next" pagination link appears and directs the user to the
     ----------------------------------------------------------------------
     Traceback (most recent call last):
     File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in
     test_beta_pagination_link
     eq_(len(pag_link), 1)
     File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
     assert a == b, msg or "%r != %r" % (a, b)
     AssertionError: 0 != 1
     ----------------------------------------------------------------------
     Ran 150 tests in 32.504s
     FAILED (SKIP=1, failures=1)
     

     View Slide

114. nicedots
     114
     

     View Slide

115. 115
     apps/input/tests/test_middleware.py:MiddlewareTests
     ...........
     apps/input/tests/test_redirects.py:RedirectTests
     ...
     apps/myadmin/tests.py:ViewTestCase
     ......
     apps/search/tests/test_client.py:SearchTest
     ..............
     apps/search/tests/test_client.py
     ..
     ======================================================================
     FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link
     ----------------------------------------------------------------------
     Traceback (most recent call last):
     File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link
     eq_(len(pag_link), 1)
     File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
     assert a == b, msg or "%r != %r" % (a, b)
     AssertionError: 0 != 1
     apps/search/tests/test_dashboard.py:TestDashboard
     ..
     apps/search/tests/test_dashboard.py:TestHelpers
     ........
     apps/search/tests/test_dashboard.py:TestMobileDashboard
     .
     apps/search/tests/test_elastic.py:TestElastic
     ..
     

     View Slide

116. nose-progressive
     116
     

     View Slide

117. View Slide

118. do not mix those two
     118
     

     View Slide

119. coverage
     119
     

     View Slide

120. coverage ./manage.py test
     120
     

     View Slide

121. coverage report -m
     121
     

     View Slide

122. 122
     Name Stmts Miss Cover Missing
     ----------------------------------------------------------------
     apps/search/__init__ 0 0 100%
     apps/search/client 193 4 98% 96-97,
     134-137
     apps/search/context_processors 5 0 100%
     apps/search/cron 13 13 0% 1-24
     apps/search/forms 70 2 97% 131,
     136
     apps/search/helpers 136 1 99% 154
     apps/search/models 0 0 100%
     apps/search/tasks 8 0 100%
     apps/search/tests/__init__ 26 1 96% 27
     apps/search/tests/test_client 75 0 100%
     apps/search/tests/test_dashboard 105 1 99% 58
     apps/search/tests/test_elastic 15 0 100%
     apps/search/tests/test_views 193 0 100%
     apps/search/urls 3 0 100%
     apps/search/utils 10 0 100%
     apps/search/views 142 1 99% 166
     ----------------------------------------------------------------
     TOTAL 994 23 98%
     

     View Slide

123. jenkins!
     123
     

     View Slide

124. formerly hudson
     124
     

     View Slide

125. jenkins.mozilla.org
     125
     

     View Slide

126. runs after each commit
     126
     

     View Slide

127. Testing Everything
     127
     

     View Slide

128. not 100% coverage
     128
     

     View Slide

129. 80-90% is okay
     129
     

     View Slide

130. “everything?”
     130
     

     View Slide

131. Test your entire site
     131
     

     View Slide

132. not just your database
     132
     

     View Slide

133. test your search engines
     133
     

     View Slide

134. your weird REST API
     134
     

     View Slide

135. that kind of “everything”
     135
     

     View Slide

136. good coverage on tricky things
     136
     

     View Slide

137. some coverage on everything
     137
     

     View Slide

138. code coverage isn’t everything
     138
     

     View Slide

139. subclass me
     139
     

     View Slide

140. APIs
     140
     

     View Slide

141. Search
     141
     

     View Slide

142. key-value stores
     142
     

     View Slide

143. anything...
     143
     

     View Slide

144. you can do cool things...
     144
     

     View Slide

145. assert!
     145
     

     View Slide

146. results are in order
     146
     

     View Slide

147. 147
     class RankingTest(SphinxTestCase):
     """This test assures that we don't regress our rankings."""
     fixtures = ('base/users',
     'base/addon_1833_yoono',
     'base/addon_9825_fastestfox',
     'base/addon_5579',
     'base/addon_personas-plus',
     )
     def test_twitter(self):
     """
     Search for twitter should yield Yoono before FastestFox since Yoono has
     "twitter" in it's name field.
     """
     r = query('twitter')
     eq_(r[0].id, 1833)
     eq_(r[1].id, 9825)
     def test_cool(self):
     """Search for cool should return CoolIris before PersonasPlus."""
     r = query('cool')
     eq_(r[0].slug, 'cooliris')
     eq_(r[1].slug, 'personas-plus')
     

     View Slide

148. FireFox == fire fox
     148
     

     View Slide

149. saves headaches
     149
     

     View Slide

150. Mock
     150
     

     View Slide

151. Mock Redis
     151
     

     View Slide

152. 152
     if not connections: # don't set this repeatedly
     for alias, backend in settings.REDIS_BACKENDS.items():
     _, server, params = parse_backend_uri(backend)
     try:
     socket_timeout = float(params.pop('socket_timeout'))
     except (KeyError, ValueError):
     socket_timeout = None
     password = params.pop('password', None)
     if ':' in server:
     host, port = server.split(':')
     try:
     port = int(port)
     except (ValueError, TypeError):
     port = 6379
     else:
     host = 'localhost'
     port = 6379
     connections[alias] = redislib.Redis(host=host, port=port, db=0,
     password=password,
     socket_timeout=socket_timeout)
     def mock_redis():
     ret = dict(connections)
     for key in connections:
     connections[key] = MockRedis()
     return ret
     

     View Slide

153. Redis <> testing requirement
     153
     

     View Slide

154. the mock client
     154
     

     View Slide

155. doesn’t do everything
     155
     

     View Slide

156. does what I’m testing for
     156
     

     View Slide

157. could get built out more...
     157
     

     View Slide

158. Setup & Teardown
     158
     

     View Slide

159. Django’s setup
     159
     

     View Slide

160. Loads fixtures
     160
     

     View Slide

161. Gives you a clean slate
     161
     

     View Slide

162. make clean slates
     162
     

     View Slide

163. search, ldap and others
     163
     

     View Slide

164. destroy the data-store
     164
     

     View Slide

165. load data-fixtures
     165
     

     View Slide

166. wrap this in a subclass of TestCase
     166
     

     View Slide

167. 167
     class SphinxTestCase(TestCase):
     """
     This test case type can setUp and tearDown the sphinx daemon. Use this
     when testing any feature that requires sphinx.
     """
     fixtures = ['users.json', 'search/documents.json',
     'posts.json', 'questions.json']
     @classmethod
     def setup_class(cls):
     super(SphinxTestCase, cls).setup_class()
     if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER:
     raise SkipTest()
     os.environ['DJANGO_ENVIRONMENT'] = 'test'
     if os.path.exists(settings.TEST_SPHINX_PATH):
     shutil.rmtree(settings.TEST_SPHINX_PATH)
     os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data'))
     os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log'))
     os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc'))
     reindex()
     start_sphinx()
     time.sleep(1)
     @classmethod
     def teardown_class(cls):
     stop_sphinx()
     super(SphinxTestCase, cls).teardown_class()
     

     View Slide

168. your coworkers will thank you
     168
     

     View Slide

169. “But I don’t work on search”
     169
     

     View Slide

170. raise SkipTest
     170
     

     View Slide

171. comes with nose
     171
     

     View Slide

172. you can sniff settings
     172
     

     View Slide

173. Built this for Sphinx Search
     173
     

     View Slide

174. Elastic Search
     174
     

     View Slide

175. LDAP
     175
     

     View Slide

176. Extras
     176
     

     View Slide

177. fixture-magic
     177
     

     View Slide

178. dj custom-dump addon 3615
     178
     

     View Slide

179. the Model Maker pattern
     179
     

     View Slide

180. be careful of dates
     180
     

     View Slide

181. use PDB for snooping
     181
     

     View Slide

182. use ipython+pdb
     182
     

     View Slide

183. alias i from IPython.Shell import
     IPShellEmbed as IPSh; IPSh(argv='')
     () > ~/.pdbrc
     183
     

     View Slide

184. I’m looking for new coworkers
     Mozilla is a fun place to work
     It’s more exciting than these slides
     We are a developer friendly python shop
     We build things and share them
     •
     •
     •
     •
     184
     

     View Slide

185. Perks
     desks
     computers
     chairs (if you want)
     work remotely
     fantastic coworkers <= seriously
     •
     •
     •
     •
     •
     185
     

     View Slide