Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Test Everything

Dave Dash
September 11, 2011

Test Everything

Testing in Django is easy if you're testing models against your database. What happens when you need to test other-systems, like a search engine, or an API? This tutorial will cover how I built SphinxTestCase, ESTestCase and redisutils to allow us to maintain test coverage on our web sites.

Dave Dash

September 11, 2011
Tweet

More Decks by Dave Dash

Other Decks in Programming

Transcript

  1. Test Anything and Everything
    dave dash
    DJANGOCON 2011
    1

    View full-size slide

  2. me (the djangonaut)
    sr web developer @ mozilla
    helped move mozilla webdev to Django
    worked on Firefox Add-ons
    Technical lead for Firefox Input
    I work on internal libraries, tools, and some
    smaller sites.
    i blog @ davedash.com
    i email @ [email protected]
    i tweet @davedash
    2

    View full-size slide

  3. mozilla webdev
    pronounced mo-zilla
    40+ people
    100+ web sites
    we happily churn out PEP-8 compliant code
    Django shop since early 2010
    we create and support most web sites
    *.mozilla.com / *.mozilla.org
    Mozilla Developer Network
    Firefox Add-ons
    Firefox Support
    Mozilla.org
    and more...
    actively hiring, if this sort of thing appeals to
    you
    3

    View full-size slide

  4. What we’ll cover
    philosophy of testing
    ./manage.py test
    Writing TestCases
    Code coverage
    Testing difficult things in the django test suite





    4

    View full-size slide

  5. What we won’t cover
    Systems testing
    Failure testing
    In browser-testing
    Load testing




    5

    View full-size slide

  6. Follow along
    Etherpad - http://mzl.la/django-test
    +1 any interesting topics
    List any testing issues or general questions you might have



    6

    View full-size slide

  7. Introduce yourself
    Name
    Involvement with Django
    Testing experience
    What you want to get out of this morning




    7

    View full-size slide

  8. Agenda
    I’ll start a fight
    Testing @ Mozilla
    Interactive stuff
    How Django’s testing works
    How we can make testing better
    Testing beyond the DB
    Gotchas
    8

    View full-size slide

  9. please...
    Ask questions. This is 3 hours... I can’t fill it all with slides.
    Correct me if I’m wrong.


    9

    View full-size slide

  10. Questions?
    10

    View full-size slide

  11. when to test?
    11

    View full-size slide

  12. but... you told us to test
    13

    View full-size slide

  13. you convinced us it was good
    14

    View full-size slide

  14. “Instead of writing tests I try to be
    extremely careful in coding, and
    keep the code size small so I
    continue to understand it.”
    -maciej ceglowski, pinboard.in
    15

    View full-size slide

  15. it all depends
    16

    View full-size slide

  16. what am I doing?
    17

    View full-size slide

  17. know your app
    18

    View full-size slide

  18. when to test
    20

    View full-size slide

  19. when others use your code
    21

    View full-size slide

  20. pull requests
    26

    View full-size slide

  21. no tests, no merge
    27

    View full-size slide

  22. failed tests... no merge
    28

    View full-size slide

  23. Testing Django @ Mozilla
    29

    View full-size slide

  24. Python Revolution at Mozilla
    31

    View full-size slide

  25. previously...
    32

    View full-size slide

  26. Banging our heads
    33

    View full-size slide

  27. PHP apps with no test framework
    34

    View full-size slide

  28. Django’s Test framework
    ==
    Appealing
    35

    View full-size slide

  29. Firefox Add-ons: 2500+ tests
    37

    View full-size slide

  30. Firefox Support: 1100 Tests
    38

    View full-size slide

  31. Coverage: 90%+
    39

    View full-size slide

  32. Tons of smaller Django sites
    40

    View full-size slide

  33. All expected to be well tested
    41

    View full-size slide

  34. At least 80-90%
    42

    View full-size slide

  35. Some things aren’t worth testing
    43

    View full-size slide

  36. What does 5000 tests buy you?
    44

    View full-size slide

  37. Too many tests per file
    46

    View full-size slide

  38. Tests take too long to run
    47

    View full-size slide

  39. Failing tests don’t get fixed
    48

    View full-size slide

  40. avoid testing external systems
    49

    View full-size slide

  41. regular django testing

    View full-size slide

  42. cd ~/fxinput_nohacks
    ./manage.py test

    View full-size slide

  43. How testing works in Django
    55

    View full-size slide

  44. The Test Runner
    56

    View full-size slide

  45. Creates Database
    57

    View full-size slide

  46. No Data... to start
    58

    View full-size slide

  47. Finds tests
    59

    View full-size slide

  48. in models.py and tests.py
    60

    View full-size slide

  49. runs the tests
    61

    View full-size slide

  50. runs non-Class tests
    62

    View full-size slide

  51. runs any doctests
    63

    View full-size slide

  52. Class tests
    65

    View full-size slide

  53. Class Setup
    66

    View full-size slide

  54. for each test...
    67

    View full-size slide

  55. Load a fixture
    69

    View full-size slide

  56. after all tests in a class...
    73

    View full-size slide

  57. class teardown
    74

    View full-size slide

  58. then we get our report card
    76

    View full-size slide

  59. “F” means bad
    77

    View full-size slide

  60. “.” means good
    78

    View full-size slide

  61. “E” doesn’t mean effort
    79

    View full-size slide

  62. Now that we know how it works...
    80

    View full-size slide

  63. WE CAN HACK IT
    81

    View full-size slide

  64. hacking testing
    82

    View full-size slide

  65. 2,500 tests is a lot
    83

    View full-size slide

  66. Multiple Files
    84

    View full-size slide

  67. test_utils
    88

    View full-size slide

  68. be real messy!
    89

    View full-size slide

  69. django-nose + test_utils
    93

    View full-size slide

  70. organize your mess
    94

    View full-size slide

  71. load a set of fixtures once
    95

    View full-size slide

  72. run all the test-cases
    96

    View full-size slide

  73. DRY... for fixtures
    97

    View full-size slide

  74. 98
    class TestCase1(TestCase):
    fixtures = ['a']
    class TestCase2(TestCase):
    fixtures = ['a', 'b']
    class TestCase3(TestCase):
    fixtures = ['a']
    class TestCase4(TestCase):
    fixtures = ['a', 'c']
    class TestCase5(TestCase):
    fixtures = ['a']
    class TestCase6(TestCase):
    fixtures = ['a', 'c']

    View full-size slide

  75. django-nose + test_utils
    99

    View full-size slide

  76. fixture misers
    100

    View full-size slide

  77. model maker
    101

    View full-size slide

  78. Useful testing tools
    102

    View full-size slide

  79. nose (via django-nose)
    103

    View full-size slide

  80. nose finds test
    104

    View full-size slide

  81. it has nice tools
    106

    View full-size slide

  82. raise SkipTest
    107

    View full-size slide

  83. eq_(x, y, “x ain’t y”)
    109

    View full-size slide

  84. nose has plugins
    112

    View full-size slide

  85. 113
    ................................S....................................................F................................................................
    ======================================================================
    FAIL: Verify the "next" pagination link appears and directs the user to the
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in
    test_beta_pagination_link
    eq_(len(pag_link), 1)
    File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
    AssertionError: 0 != 1
    ----------------------------------------------------------------------
    Ran 150 tests in 32.504s
    FAILED (SKIP=1, failures=1)

    View full-size slide

  86. 115
    apps/input/tests/test_middleware.py:MiddlewareTests
    ...........
    apps/input/tests/test_redirects.py:RedirectTests
    ...
    apps/myadmin/tests.py:ViewTestCase
    ......
    apps/search/tests/test_client.py:SearchTest
    ..............
    apps/search/tests/test_client.py
    ..
    ======================================================================
    FAIL: apps/search/tests/test_dashboard.py:TestDashboard.test_beta_pagination_link
    ----------------------------------------------------------------------
    Traceback (most recent call last):
    File "/Users/dash/Projects/input/reporter/apps/search/tests/test_dashboard.py", line 57, in test_beta_pagination_link
    eq_(len(pag_link), 1)
    File "/Users/dash/Projects/input/reporter/vendor/packages/nose/nose/tools.py", line 31, in eq_
    assert a == b, msg or "%r != %r" % (a, b)
    AssertionError: 0 != 1
    apps/search/tests/test_dashboard.py:TestDashboard
    ..
    apps/search/tests/test_dashboard.py:TestHelpers
    ........
    apps/search/tests/test_dashboard.py:TestMobileDashboard
    .
    apps/search/tests/test_elastic.py:TestElastic
    ..

    View full-size slide

  87. nose-progressive
    116

    View full-size slide

  88. do not mix those two
    118

    View full-size slide

  89. coverage ./manage.py test
    120

    View full-size slide

  90. coverage report -m
    121

    View full-size slide

  91. 122
    Name Stmts Miss Cover Missing
    ----------------------------------------------------------------
    apps/search/__init__ 0 0 100%
    apps/search/client 193 4 98% 96-97,
    134-137
    apps/search/context_processors 5 0 100%
    apps/search/cron 13 13 0% 1-24
    apps/search/forms 70 2 97% 131,
    136
    apps/search/helpers 136 1 99% 154
    apps/search/models 0 0 100%
    apps/search/tasks 8 0 100%
    apps/search/tests/__init__ 26 1 96% 27
    apps/search/tests/test_client 75 0 100%
    apps/search/tests/test_dashboard 105 1 99% 58
    apps/search/tests/test_elastic 15 0 100%
    apps/search/tests/test_views 193 0 100%
    apps/search/urls 3 0 100%
    apps/search/utils 10 0 100%
    apps/search/views 142 1 99% 166
    ----------------------------------------------------------------
    TOTAL 994 23 98%

    View full-size slide

  92. formerly hudson
    124

    View full-size slide

  93. jenkins.mozilla.org
    125

    View full-size slide

  94. runs after each commit
    126

    View full-size slide

  95. Testing Everything
    127

    View full-size slide

  96. not 100% coverage
    128

    View full-size slide

  97. 80-90% is okay
    129

    View full-size slide

  98. “everything?”
    130

    View full-size slide

  99. Test your entire site
    131

    View full-size slide

  100. not just your database
    132

    View full-size slide

  101. test your search engines
    133

    View full-size slide

  102. your weird REST API
    134

    View full-size slide

  103. that kind of “everything”
    135

    View full-size slide

  104. good coverage on tricky things
    136

    View full-size slide

  105. some coverage on everything
    137

    View full-size slide

  106. code coverage isn’t everything
    138

    View full-size slide

  107. subclass me
    139

    View full-size slide

  108. key-value stores
    142

    View full-size slide

  109. anything...
    143

    View full-size slide

  110. you can do cool things...
    144

    View full-size slide

  111. results are in order
    146

    View full-size slide

  112. 147
    class RankingTest(SphinxTestCase):
    """This test assures that we don't regress our rankings."""
    fixtures = ('base/users',
    'base/addon_1833_yoono',
    'base/addon_9825_fastestfox',
    'base/addon_5579',
    'base/addon_personas-plus',
    )
    def test_twitter(self):
    """
    Search for twitter should yield Yoono before FastestFox since Yoono has
    "twitter" in it's name field.
    """
    r = query('twitter')
    eq_(r[0].id, 1833)
    eq_(r[1].id, 9825)
    def test_cool(self):
    """Search for cool should return CoolIris before PersonasPlus."""
    r = query('cool')
    eq_(r[0].slug, 'cooliris')
    eq_(r[1].slug, 'personas-plus')

    View full-size slide

  113. FireFox == fire fox
    148

    View full-size slide

  114. saves headaches
    149

    View full-size slide

  115. Mock Redis
    151

    View full-size slide

  116. 152
    if not connections: # don't set this repeatedly
    for alias, backend in settings.REDIS_BACKENDS.items():
    _, server, params = parse_backend_uri(backend)
    try:
    socket_timeout = float(params.pop('socket_timeout'))
    except (KeyError, ValueError):
    socket_timeout = None
    password = params.pop('password', None)
    if ':' in server:
    host, port = server.split(':')
    try:
    port = int(port)
    except (ValueError, TypeError):
    port = 6379
    else:
    host = 'localhost'
    port = 6379
    connections[alias] = redislib.Redis(host=host, port=port, db=0,
    password=password,
    socket_timeout=socket_timeout)
    def mock_redis():
    ret = dict(connections)
    for key in connections:
    connections[key] = MockRedis()
    return ret

    View full-size slide

  117. Redis <> testing requirement
    153

    View full-size slide

  118. the mock client
    154

    View full-size slide

  119. doesn’t do everything
    155

    View full-size slide

  120. does what I’m testing for
    156

    View full-size slide

  121. could get built out more...
    157

    View full-size slide

  122. Setup & Teardown
    158

    View full-size slide

  123. Django’s setup
    159

    View full-size slide

  124. Loads fixtures
    160

    View full-size slide

  125. Gives you a clean slate
    161

    View full-size slide

  126. make clean slates
    162

    View full-size slide

  127. search, ldap and others
    163

    View full-size slide

  128. destroy the data-store
    164

    View full-size slide

  129. load data-fixtures
    165

    View full-size slide

  130. wrap this in a subclass of TestCase
    166

    View full-size slide

  131. 167
    class SphinxTestCase(TestCase):
    """
    This test case type can setUp and tearDown the sphinx daemon. Use this
    when testing any feature that requires sphinx.
    """
    fixtures = ['users.json', 'search/documents.json',
    'posts.json', 'questions.json']
    @classmethod
    def setup_class(cls):
    super(SphinxTestCase, cls).setup_class()
    if not settings.SPHINX_SEARCHD or not settings.SPHINX_INDEXER:
    raise SkipTest()
    os.environ['DJANGO_ENVIRONMENT'] = 'test'
    if os.path.exists(settings.TEST_SPHINX_PATH):
    shutil.rmtree(settings.TEST_SPHINX_PATH)
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'data'))
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'log'))
    os.makedirs(os.path.join(settings.TEST_SPHINX_PATH, 'etc'))
    reindex()
    start_sphinx()
    time.sleep(1)
    @classmethod
    def teardown_class(cls):
    stop_sphinx()
    super(SphinxTestCase, cls).teardown_class()

    View full-size slide

  132. your coworkers will thank you
    168

    View full-size slide

  133. “But I don’t work on search”
    169

    View full-size slide

  134. raise SkipTest
    170

    View full-size slide

  135. comes with nose
    171

    View full-size slide

  136. you can sniff settings
    172

    View full-size slide

  137. Built this for Sphinx Search
    173

    View full-size slide

  138. Elastic Search
    174

    View full-size slide

  139. fixture-magic
    177

    View full-size slide

  140. dj custom-dump addon 3615
    178

    View full-size slide

  141. the Model Maker pattern
    179

    View full-size slide

  142. be careful of dates
    180

    View full-size slide

  143. use PDB for snooping
    181

    View full-size slide

  144. use ipython+pdb
    182

    View full-size slide

  145. alias i from IPython.Shell import
    IPShellEmbed as IPSh; IPSh(argv='')
    () > ~/.pdbrc
    183

    View full-size slide

  146. I’m looking for new coworkers
    Mozilla is a fun place to work
    It’s more exciting than these slides
    We are a developer friendly python shop
    We build things and share them




    184

    View full-size slide

  147. Perks
    desks
    computers
    chairs (if you want)
    work remotely
    fantastic coworkers <= seriously





    185

    View full-size slide