Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test Everything

Dave Dash
September 11, 2011

Test Everything

Testing in Django is easy if you're testing models against your database. What happens when you need to test other-systems, like a search engine, or an API? This tutorial will cover how I built SphinxTestCase, ESTestCase and redisutils to allow us to maintain test coverage on our web sites.

Dave Dash

September 11, 2011
Tweet

More Decks by Dave Dash

Other Decks in Programming

Transcript

  1. Test Anything and Everything
    dave dash
    DJANGOCON 2011
    1

    View Slide

  2. me (the djangonaut)
    sr web developer @ mozilla
    helped move mozilla webdev to Django
    worked on Firefox Add-ons
    Technical lead for Firefox Input
    I work on internal libraries, tools, and some
    smaller sites.
    i blog @ davedash.com
    i email @ [email protected]
    i tweet @davedash
    2

    View Slide

  3. mozilla webdev
    pronounced mo-zilla
    40+ people
    100+ web sites
    we happily churn out PEP-8 compliant code
    Django shop since early 2010
    we create and support most web sites
    *.mozilla.com / *.mozilla.org
    Mozilla Developer Network
    Firefox Add-ons
    Firefox Support
    Mozilla.org
    and more...
    actively hiring, if this sort of thing appeals to
    you
    3

    View Slide

  4. What we’ll cover
    philosophy of testing
    ./manage.py test
    Writing TestCases
    Code coverage
    Testing difficult things in the django test suite





    4

    View Slide

  5. What we won’t cover
    Systems testing
    Failure testing
    In browser-testing
    Load testing




    5

    View Slide

  6. Follow along
    Etherpad - http://mzl.la/django-test
    +1 any interesting topics
    List any testing issues or general questions you might have



    6

    View Slide

  7. Introduce yourself
    Name
    Involvement with Django
    Testing experience
    What you want to get out of this morning




    7

    View Slide

  8. Agenda
    I’ll start a fight
    Testing @ Mozilla
    Interactive stuff
    How Django’s testing works
    How we can make testing better
    Testing beyond the DB
    Gotchas
    8

    View Slide

  9. please...
    Ask questions. This is 3 hours... I can’t fill it all with slides.
    Correct me if I’m wrong.


    9

    View Slide

  10. Questions?
    10

    View Slide

  11. when to test?
    11

    View Slide

  12. never
    12

    View Slide

  13. but... you told us to test
    13

    View Slide

  14. you convinced us it was good
    14

    View Slide

  15. “Instead of writing tests I try to be
    extremely careful in coding, and
    keep the code size small so I
    continue to understand it.”
    -maciej ceglowski, pinboard.in
    15

    View Slide

  16. it all depends
    16

    View Slide

  17. what am I doing?
    17

    View Slide

  18. know your app
    18

    View Slide

  19. but...
    19

    View Slide

  20. when to test
    20

    View Slide

  21. when others use your code
    21

    View Slide

  22. libraries
    22

    View Slide

  23. projects?
    23

    View Slide

  24. big sites
    24

    View Slide

  25. teams
    25

    View Slide

  26. pull requests
    26

    View Slide

  27. no tests, no merge
    27

    View Slide

  28. failed tests... no merge
    28

    View Slide

  29. Testing Django @ Mozilla
    29

    View Slide

  30. 2009/2010
    30

    View Slide

  31. Python Revolution at Mozilla
    31

    View Slide

  32. previously...
    32

    View Slide

  33. Banging our heads
    33

    View Slide

  34. PHP apps with no test framework
    34

    View Slide

  35. Django’s Test framework
    ==
    Appealing
    35

    View Slide

  36. today...
    36

    View Slide

  37. Firefox Add-ons: 2500+ tests
    37

    View Slide

  38. Firefox Support: 1100 Tests
    38

    View Slide

  39. Coverage: 90%+
    39

    View Slide

  40. Tons of smaller Django sites
    40

    View Slide

  41. All expected to be well tested
    41

    View Slide

  42. At least 80-90%
    42

    View Slide

  43. Some things aren’t worth testing
    43

    View Slide

  44. What does 5000 tests buy you?
    44

    View Slide

  45. Problems
    45

    View Slide

  46. Too many tests per file
    46

    View Slide

  47. Tests take too long to run
    47

    View Slide

  48. Failing tests don’t get fixed
    48

    View Slide

  49. avoid testing external systems
    49

    View Slide

  50. Vagrant
    50

    View Slide

  51. vagrant up

    View Slide

  52. regular django testing

    View Slide

  53. cd ~/fxinput_nohacks
    ./manage.py test

    View Slide

  54. View Slide

  55. How testing works in Django
    55

    View Slide

  56. The Test Runner
    56

    View Slide

  57. Creates Database
    57

    View Slide

  58. No Data... to start
    58

    View Slide

  59. Finds tests
    59

    View Slide

  60. in models.py and tests.py
    60

    View Slide

  61. runs the tests
    61

    View Slide

  62. runs non-Class tests
    62

    View Slide

  63. runs any doctests
    63

    View Slide

  64. 64

    View Slide

  65. Class tests
    65

    View Slide

  66. Class Setup
    66

    View Slide

  67. for each test...
    67

    View Slide

  68. Setup
    68

    View Slide

  69. Load a fixture
    69

    View Slide

  70. 70

    View Slide

  71. Run Test
    71

    View Slide

  72. Teardown
    72

    View Slide

  73. after all tests in a class...
    73

    View Slide

  74. class teardown
    74

    View Slide

  75. 75

    View Slide

  76. then we get our report card
    76

    View Slide

  77. “F” means bad
    77

    View Slide

  78. “.” means good
    78

    View Slide

  79. “E” doesn’t mean effort
    79

    View Slide

  80. Now that we know how it works...
    80

    View Slide

  81. WE CAN HACK IT
    81

    View Slide

  82. hacking testing
    82

    View Slide

  83. 2,500 tests is a lot
    83

    View Slide

  84. Multiple Files
    84

    View Slide

  85. 85

    View Slide

  86. be messy!
    86

    View Slide

  87. 87

    View Slide

  88. test_utils
    88

    View Slide

  89. be real messy!
    89

    View Slide

  90. 90

    View Slide

  91. 91

    View Slide

  92. 92

    View Slide

  93. django-nose + test_utils
    93

    View Slide

  94. organize your mess
    94

    View Slide

  95. load a set of fixtures once
    95

    View Slide

  96. run all the test-cases
    96

    View Slide

  97. DRY... for fixtures
    97

    View Slide

  98. 98
    class TestCase1(TestCase):
    fixtures = ['a']
    class TestCase2(TestCase):
    fixtures = ['a', 'b']
    class TestCase3(TestCase):
    fixtures = ['a']
    class TestCase4(TestCase):
    fixtures = ['a', 'c']
    class TestCase5(TestCase):
    fixtures = ['a']
    class TestCase6(TestCase):
    fixtures = ['a', 'c']

    View Slide

  99. django-nose + test_utils
    99

    View Slide

  100. fixture misers
    100

    View Slide

  101. model maker
    101

    View Slide

  102. Useful testing tools
    102

    View Slide

  103. nose (via django-nose)
    103

    View Slide

  104. nose finds test
    104

    View Slide

  105. 105

    View Slide

  106. it has nice tools
    106

    View Slide

  107. raise SkipTest
    107

    View Slide

  108. 108

    View Slide

  109. eq_(x, y, “x ain’t y”)
    109

    View Slide

  110. --failed
    110

    View Slide

  111. 111

    View Slide

  112. nose has plugins
    112

    View Slide

  113. 113
    ................................S....................................................F................................................................
    ======================================================================
    FAIL: Verify the "next" pagination link appears and directs the user to the
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in
    test_beta_pagination_link
    eq_(len(pag_link), 1)
    File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
    AssertionError: 0 != 1
    ----------------------------------------------------------------------
    Ran 150 tests in 32.504s
    FAILED (SKIP=1, failures=1)

    View Slide

  114. nicedots
    114

    View Slide

  115. 115
    apps/input/tests/test_middleware.py:MiddlewareTests
    ...........
    apps/input/tests/test_redirects.py:RedirectTests
    ...
    apps/myadmin/tests.py:ViewTestCase
    ......
    apps/search/tests/test_client.py:SearchTest
    ..............
    apps/search/tests/test_client.py
    ..
    ======================================================================
    FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link
    eq_(len(pag_link), 1)
    File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
    AssertionError: 0 != 1
    apps/search/tests/test_dashboard.py:TestDashboard
    ..
    apps/search/tests/test_dashboard.py:TestHelpers
    ........
    apps/search/tests/test_dashboard.py:TestMobileDashboard
    .
    apps/search/tests/test_elastic.py:TestElastic
    ..

    View Slide

  116. nose-progressive
    116

    View Slide

  117. 117

    View Slide

  118. do not mix those two
    118

    View Slide

  119. coverage
    119

    View Slide

  120. coverage ./manage.py test
    120

    View Slide

  121. coverage report -m
    121

    View Slide

  122. 122
    Name Stmts Miss Cover Missing
    ----------------------------------------------------------------
    apps/search/__init__ 0 0 100%
    apps/search/client 193 4 98% 96-97,
    134-137
    apps/search/context_processors 5 0 100%
    apps/search/cron 13 13 0% 1-24
    apps/search/forms 70 2 97% 131,
    136
    apps/search/helpers 136 1 99% 154
    apps/search/models 0 0 100%
    apps/search/tasks 8 0 100%
    apps/search/tests/__init__ 26 1 96% 27
    apps/search/tests/test_client 75 0 100%
    apps/search/tests/test_dashboard 105 1 99% 58
    apps/search/tests/test_elastic 15 0 100%
    apps/search/tests/test_views 193 0 100%
    apps/search/urls 3 0 100%
    apps/search/utils 10 0 100%
    apps/search/views 142 1 99% 166
    ----------------------------------------------------------------
    TOTAL 994 23 98%

    View Slide

  123. jenkins!
    123

    View Slide

  124. formerly hudson
    124

    View Slide

  125. jenkins.mozilla.org
    125

    View Slide

  126. runs after each commit
    126

    View Slide

  127. Testing Everything
    127

    View Slide

  128. not 100% coverage
    128

    View Slide

  129. 80-90% is okay
    129

    View Slide

  130. “everything?”
    130

    View Slide

  131. Test your entire site
    131

    View Slide

  132. not just your database
    132

    View Slide

  133. test your search engines
    133

    View Slide

  134. your weird REST API
    134

    View Slide

  135. that kind of “everything”
    135

    View Slide

  136. good coverage on tricky things
    136

    View Slide

  137. some coverage on everything
    137

    View Slide

  138. code coverage isn’t everything
    138

    View Slide

  139. subclass me
    139

    View Slide

  140. APIs
    140

    View Slide

  141. Search
    141

    View Slide

  142. key-value stores
    142

    View Slide

  143. anything...
    143

    View Slide

  144. you can do cool things...
    144

    View Slide

  145. assert!
    145

    View Slide

  146. results are in order
    146

    View Slide

  147. 147
    class RankingTest(SphinxTestCase):
    """This test assures that we don't regress our rankings."""
    fixtures = ('base/users',
    'base/addon_1833_yoono',
    'base/addon_9825_fastestfox',
    'base/addon_5579',
    'base/addon_personas-plus',
    )
    def test_twitter(self):
    """
    Search for twitter should yield Yoono before FastestFox since Yoono has
    "twitter" in it's name field.
    """
    r = query('twitter')
    eq_(r[0].id, 1833)
    eq_(r[1].id, 9825)
    def test_cool(self):
    """Search for cool should return CoolIris before PersonasPlus."""
    r = query('cool')
    eq_(r[0].slug, 'cooliris')
    eq_(r[1].slug, 'personas-plus')

    View Slide

  148. FireFox == fire fox
    148

    View Slide

  149. saves headaches
    149

    View Slide

  150. Mock
    150

    View Slide

  151. Mock Redis
    151

    View Slide

  152. 152
    if not connections: # don't set this repeatedly
    for alias, backend in settings.REDIS_BACKENDS.items():
    _, server, params = parse_backend_uri(backend)
    try:
    socket_timeout = float(params.pop('socket_timeout'))
    except (KeyError, ValueError):
    socket_timeout = None
    password = params.pop('password', None)
    if ':' in server:
    host, port = server.split(':')
    try:
    port = int(port)
    except (ValueError, TypeError):
    port = 6379
    else:
    host = 'localhost'
    port = 6379
    connections[alias] = redislib.Redis(host=host, port=port, db=0,
    password=password,
    socket_timeout=socket_timeout)
    def mock_redis():
    ret = dict(connections)
    for key in connections:
    connections[key] = MockRedis()
    return ret

    View Slide

  153. Redis <> testing requirement
    153

    View Slide

  154. the mock client
    154

    View Slide

  155. doesn’t do everything
    155

    View Slide

  156. does what I’m testing for
    156

    View Slide

  157. could get built out more...
    157

    View Slide

  158. Setup & Teardown
    158

    View Slide

  159. Django’s setup
    159

    View Slide

  160. Loads fixtures
    160

    View Slide

  161. Gives you a clean slate
    161

    View Slide

  162. make clean slates
    162

    View Slide

  163. search, ldap and others
    163

    View Slide

  164. destroy the data-store
    164

    View Slide

  165. load data-fixtures
    165

    View Slide

  166. wrap this in a subclass of TestCase
    166

    View Slide

  167. 167
    class SphinxTestCase(TestCase):
    """
    This test case type can setUp and tearDown the sphinx daemon. Use this
    when testing any feature that requires sphinx.
    """
    fixtures = ['users.json', 'search/documents.json',
    'posts.json', 'questions.json']
    @classmethod
    def setup_class(cls):
    super(SphinxTestCase, cls).setup_class()
    if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER:
    raise SkipTest()
    os.environ['DJANGO_ENVIRONMENT'] = 'test'
    if os.path.exists(settings.TEST_SPHINX_PATH):
    shutil.rmtree(settings.TEST_SPHINX_PATH)
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data'))
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log'))
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc'))
    reindex()
    start_sphinx()
    time.sleep(1)
    @classmethod
    def teardown_class(cls):
    stop_sphinx()
    super(SphinxTestCase, cls).teardown_class()

    View Slide

  168. your coworkers will thank you
    168

    View Slide

  169. “But I don’t work on search”
    169

    View Slide

  170. raise SkipTest
    170

    View Slide

  171. comes with nose
    171

    View Slide

  172. you can sniff settings
    172

    View Slide

  173. Built this for Sphinx Search
    173

    View Slide

  174. Elastic Search
    174

    View Slide

  175. LDAP
    175

    View Slide

  176. Extras
    176

    View Slide

  177. fixture-magic
    177

    View Slide

  178. dj custom-dump addon 3615
    178

    View Slide

  179. the Model Maker pattern
    179

    View Slide

  180. be careful of dates
    180

    View Slide

  181. use PDB for snooping
    181

    View Slide

  182. use ipython+pdb
    182

    View Slide

  183. alias i from IPython.Shell import
    IPShellEmbed as IPSh; IPSh(argv='')
    () > ~/.pdbrc
    183

    View Slide

  184. I’m looking for new coworkers
    Mozilla is a fun place to work
    It’s more exciting than these slides
    We are a developer friendly python shop
    We build things and share them




    184

    View Slide

  185. Perks
    desks
    computers
    chairs (if you want)
    work remotely
    fantastic coworkers <= seriously





    185

    View Slide