Test Everything

E3c6ff6229e3fe28f6dd008d8dc5ad04?s=47 Dave Dash
September 11, 2011

Test Everything

Testing in Django is easy if you're testing models against your database. What happens when you need to test other-systems, like a search engine, or an API? This tutorial will cover how I built SphinxTestCase, ESTestCase and redisutils to allow us to maintain test coverage on our web sites.

E3c6ff6229e3fe28f6dd008d8dc5ad04?s=128

Dave Dash

September 11, 2011
Tweet

Transcript

  1. Test Anything and Everything dave dash DJANGOCON 2011 1

  2. me (the djangonaut) sr web developer @ mozilla helped move

    mozilla webdev to Django worked on Firefox Add-ons Technical lead for Firefox Input I work on internal libraries, tools, and some smaller sites. i blog @ davedash.com i email @ dd@davedash.com i tweet @davedash 2
  3. mozilla webdev pronounced mo-zilla 40+ people 100+ web sites we

    happily churn out PEP-8 compliant code Django shop since early 2010 we create and support most web sites *.mozilla.com / *.mozilla.org Mozilla Developer Network Firefox Add-ons Firefox Support Mozilla.org and more... actively hiring, if this sort of thing appeals to you 3
  4. What we’ll cover philosophy of testing ./manage.py test Writing TestCases

    Code coverage Testing difficult things in the django test suite • • • • • 4
  5. What we won’t cover Systems testing Failure testing In browser-testing

    Load testing • • • • 5
  6. Follow along Etherpad - http://mzl.la/django-test +1 any interesting topics List

    any testing issues or general questions you might have • • • 6
  7. Introduce yourself Name Involvement with Django Testing experience What you

    want to get out of this morning • • • • 7
  8. Agenda I’ll start a fight Testing @ Mozilla Interactive stuff

    How Django’s testing works How we can make testing better Testing beyond the DB Gotchas 8
  9. please... Ask questions. This is 3 hours... I can’t fill

    it all with slides. Correct me if I’m wrong. • • 9
  10. Questions? 10

  11. when to test? 11

  12. never 12

  13. but... you told us to test 13

  14. you convinced us it was good 14

  15. “Instead of writing tests I try to be extremely careful

    in coding, and keep the code size small so I continue to understand it.” -maciej ceglowski, pinboard.in 15
  16. it all depends 16

  17. what am I doing? 17

  18. know your app 18

  19. but... 19

  20. when to test 20

  21. when others use your code 21

  22. libraries 22

  23. projects? 23

  24. big sites 24

  25. teams 25

  26. pull requests 26

  27. no tests, no merge 27

  28. failed tests... no merge 28

  29. Testing Django @ Mozilla 29

  30. 2009/2010 30

  31. Python Revolution at Mozilla 31

  32. previously... 32

  33. Banging our heads 33

  34. PHP apps with no test framework 34

  35. Django’s Test framework == Appealing 35

  36. today... 36

  37. Firefox Add-ons: 2500+ tests 37

  38. Firefox Support: 1100 Tests 38

  39. Coverage: 90%+ 39

  40. Tons of smaller Django sites 40

  41. All expected to be well tested 41

  42. At least 80-90% 42

  43. Some things aren’t worth testing 43

  44. What does 5000 tests buy you? 44

  45. Problems 45

  46. Too many tests per file 46

  47. Tests take too long to run 47

  48. Failing tests don’t get fixed 48

  49. avoid testing external systems 49

  50. Vagrant 50

  51. vagrant up

  52. regular django testing

  53. cd ~/fxinput_nohacks ./manage.py test

  54. None
  55. How testing works in Django 55

  56. The Test Runner 56

  57. Creates Database 57

  58. No Data... to start 58

  59. Finds tests 59

  60. in models.py and tests.py 60

  61. runs the tests 61

  62. runs non-Class tests 62

  63. runs any doctests 63

  64. 64

  65. Class tests 65

  66. Class Setup 66

  67. for each test... 67

  68. Setup 68

  69. Load a fixture 69

  70. 70

  71. Run Test 71

  72. Teardown 72

  73. after all tests in a class... 73

  74. class teardown 74

  75. 75

  76. then we get our report card 76

  77. “F” means bad 77

  78. “.” means good 78

  79. “E” doesn’t mean effort 79

  80. Now that we know how it works... 80

  81. WE CAN HACK IT 81

  82. hacking testing 82

  83. 2,500 tests is a lot 83

  84. Multiple Files 84

  85. 85

  86. be messy! 86

  87. 87

  88. test_utils 88

  89. be real messy! 89

  90. 90

  91. 91

  92. 92

  93. django-nose + test_utils 93

  94. organize your mess 94

  95. load a set of fixtures once 95

  96. run all the test-cases 96

  97. DRY... for fixtures 97

  98. 98 class TestCase1(TestCase): fixtures = ['a'] class TestCase2(TestCase): fixtures =

    ['a', 'b'] class TestCase3(TestCase): fixtures = ['a'] class TestCase4(TestCase): fixtures = ['a', 'c'] class TestCase5(TestCase): fixtures = ['a'] class TestCase6(TestCase): fixtures = ['a', 'c']
  99. django-nose + test_utils 99

  100. fixture misers 100

  101. model maker 101

  102. Useful testing tools 102

  103. nose (via django-nose) 103

  104. nose finds test 104

  105. 105

  106. it has nice tools 106

  107. raise SkipTest 107

  108. 108

  109. eq_(x, y, “x ain’t y”) 109

  110. --failed 110

  111. 111

  112. nose has plugins 112

  113. 113 ................................S....................................................F................................................................ ====================================================================== FAIL: Verify the "next" pagination link appears

    and directs the user to the ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 ---------------------------------------------------------------------- Ran 150 tests in 32.504s FAILED (SKIP=1, failures=1)
  114. nicedots 114

  115. 115 apps/input/tests/test_middleware.py:MiddlewareTests ........... apps/input/tests/test_redirects.py:RedirectTests ... apps/myadmin/tests.py:ViewTestCase ...... apps/search/tests/test_client.py:SearchTest .............. apps/search/tests/test_client.py

    .. ====================================================================== FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link eq_(len(pag_link), 1) File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_ assert a == b, msg or "%r != %r" % (a, b) AssertionError: 0 != 1 apps/search/tests/test_dashboard.py:TestDashboard .. apps/search/tests/test_dashboard.py:TestHelpers ........ apps/search/tests/test_dashboard.py:TestMobileDashboard . apps/search/tests/test_elastic.py:TestElastic ..
  116. nose-progressive 116

  117. 117

  118. do not mix those two 118

  119. coverage 119

  120. coverage ./manage.py test 120

  121. coverage report -m 121

  122. 122 Name Stmts Miss Cover Missing ---------------------------------------------------------------- apps/search/__init__ 0 0

    100% apps/search/client 193 4 98% 96-97, 134-137 apps/search/context_processors 5 0 100% apps/search/cron 13 13 0% 1-24 apps/search/forms 70 2 97% 131, 136 apps/search/helpers 136 1 99% 154 apps/search/models 0 0 100% apps/search/tasks 8 0 100% apps/search/tests/__init__ 26 1 96% 27 apps/search/tests/test_client 75 0 100% apps/search/tests/test_dashboard 105 1 99% 58 apps/search/tests/test_elastic 15 0 100% apps/search/tests/test_views 193 0 100% apps/search/urls 3 0 100% apps/search/utils 10 0 100% apps/search/views 142 1 99% 166 ---------------------------------------------------------------- TOTAL 994 23 98%
  123. jenkins! 123

  124. formerly hudson 124

  125. jenkins.mozilla.org 125

  126. runs after each commit 126

  127. Testing Everything 127

  128. not 100% coverage 128

  129. 80-90% is okay 129

  130. “everything?” 130

  131. Test your entire site 131

  132. not just your database 132

  133. test your search engines 133

  134. your weird REST API 134

  135. that kind of “everything” 135

  136. good coverage on tricky things 136

  137. some coverage on everything 137

  138. code coverage isn’t everything 138

  139. subclass me 139

  140. APIs 140

  141. Search 141

  142. key-value stores 142

  143. anything... 143

  144. you can do cool things... 144

  145. assert! 145

  146. results are in order 146

  147. 147 class RankingTest(SphinxTestCase): """This test assures that we don't regress

    our rankings.""" fixtures = ('base/users', 'base/addon_1833_yoono', 'base/addon_9825_fastestfox', 'base/addon_5579', 'base/addon_personas-plus', ) def test_twitter(self): """ Search for twitter should yield Yoono before FastestFox since Yoono has "twitter" in it's name field. """ r = query('twitter') eq_(r[0].id, 1833) eq_(r[1].id, 9825) def test_cool(self): """Search for cool should return CoolIris before PersonasPlus.""" r = query('cool') eq_(r[0].slug, 'cooliris') eq_(r[1].slug, 'personas-plus')
  148. FireFox == fire fox 148

  149. saves headaches 149

  150. Mock 150

  151. Mock Redis 151

  152. 152 if not connections: # don't set this repeatedly for

    alias, backend in settings.REDIS_BACKENDS.items(): _, server, params = parse_backend_uri(backend) try: socket_timeout = float(params.pop('socket_timeout')) except (KeyError, ValueError): socket_timeout = None password = params.pop('password', None) if ':' in server: host, port = server.split(':') try: port = int(port) except (ValueError, TypeError): port = 6379 else: host = 'localhost' port = 6379 connections[alias] = redislib.Redis(host=host, port=port, db=0, password=password, socket_timeout=socket_timeout) def mock_redis(): ret = dict(connections) for key in connections: connections[key] = MockRedis() return ret
  153. Redis <> testing requirement 153

  154. the mock client 154

  155. doesn’t do everything 155

  156. does what I’m testing for 156

  157. could get built out more... 157

  158. Setup & Teardown 158

  159. Django’s setup 159

  160. Loads fixtures 160

  161. Gives you a clean slate 161

  162. make clean slates 162

  163. search, ldap and others 163

  164. destroy the data-store 164

  165. load data-fixtures 165

  166. wrap this in a subclass of TestCase 166

  167. 167 class SphinxTestCase(TestCase): """ This test case type can setUp

    and tearDown the sphinx daemon. Use this when testing any feature that requires sphinx. """ fixtures = ['users.json', 'search/documents.json', 'posts.json', 'questions.json'] @classmethod def setup_class(cls): super(SphinxTestCase, cls).setup_class() if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER: raise SkipTest() os.environ['DJANGO_ENVIRONMENT'] = 'test' if os.path.exists(settings.TEST_SPHINX_PATH): shutil.rmtree(settings.TEST_SPHINX_PATH) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log')) os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc')) reindex() start_sphinx() time.sleep(1) @classmethod def teardown_class(cls): stop_sphinx() super(SphinxTestCase, cls).teardown_class()
  168. your coworkers will thank you 168

  169. “But I don’t work on search” 169

  170. raise SkipTest 170

  171. comes with nose 171

  172. you can sniff settings 172

  173. Built this for Sphinx Search 173

  174. Elastic Search 174

  175. LDAP 175

  176. Extras 176

  177. fixture-magic 177

  178. dj custom-dump addon 3615 178

  179. the Model Maker pattern 179

  180. be careful of dates 180

  181. use PDB for snooping 181

  182. use ipython+pdb 182

  183. alias i from IPython.Shell import IPShellEmbed as IPSh; IPSh(argv='') ()

    > ~/.pdbrc 183
  184. I’m looking for new coworkers Mozilla is a fun place

    to work It’s more exciting than these slides We are a developer friendly python shop We build things and share them • • • • 184
  185. Perks desks computers chairs (if you want) work remotely fantastic

    coworkers <= seriously • • • • • 185