Testing at Scale

Transcript

1. TESTING
   3600 TESTS, AND COUNTING
   

   View Slide

2. TESTING AT SCALE
   WHAT ARE GOOD TESTS?
   

   View Slide

3. TESTS SHOULD BE AS SIMPLE AS POSSIBLE
   Don’t write test code that needs it’s own tests!
   Be clear - not clever.
   DRY is great for code, just not for test code.
   

   View Slide

4. TESTS SHOULD RUN AS QUICKLY AS POSSIBLE
   Encourage developers to run tests frequently.
   Sometimes running subset of tests may not be
   enough.
   Instill a culture of reliance on tests.
   

   View Slide

5. TESTS SHOULD AVOID COUPLING WITH OTHER TESTS
   TestCases should not extend TestCases in other
   python modules.
   Tests should not use helper methods which exist in
   other python modules.
   

   View Slide

6. TESTS SHOULD COMMUNICATE INTENT
   Tests are explicitly clear about what they are testing.
   Related test methods reside in the same TestCase.
   Tests should not rely on documentation to explain what
   they are doing.
   Developers reading the tests should not have to leave
   the TestCase code to determine what the test is doing.
   

   View Slide

7. CANONICAL FORM
   THE STRUCTURE OF A TEST
   

   View Slide

8. SET UP PRE-CONDITIONS
   Create the environment needed for the test to run.
   • Create model instances.
   • Set up the state required for the test.
   • Create mock objects.
   • Patch methods, functions and objects.
   

   View Slide

9. PERFORM OPERATION UNDER TEST
   Run test using the environment established in the pre-
   conditions.
   • Modify state.
   • Record return values.
   

   View Slide

10. MAKE ASSERTIONS
    Make assertions about the return values or side effects.
    • Check return values against expected values.
    • Test that the state has been modified as expected.
    

    View Slide

11. View Slide

12. DO ABSOLUTELY NOTHING ELSE
    • Don’t modify the test conditions and test assertions again.
    • Don’t extend your test further.
    • Don’t create test methods that chain together main tests.
    • Write a new test method, and copy the pre-conditions.
    

    View Slide

13. BENEFITS OF FOLLOWING CANONICAL FORM
    Each test method tests exactly one path through the code under test.
    The set of TestCases will define a clear contract with the developer.
    For any test that fails, the cause will be easy to pinpoint because the expected results is clearly
    focused.
    

    View Slide

14. FACTORIES
    SO MUCH BETTER THAN FIXTURES
    

    View Slide

15. DJANGO APP called factory_boy
    Third-party Django application.
    Based on popular Rails library called factory_girl.
    Based on your own Django models.
    Replacement for fixtures.
    

    View Slide

16. Only define sub-factories for data that needs to be created.
    Model relations with null=True or blank=True then this isn’t generally a business need.
    ONLY CREATE DATA BASED ON BUSINESS NEEDS
    

    View Slide

17. Factories should only be responsible for creating data, not setting up pre-conditions for tests.
    FACTORIES CREATE DATA ONLY
    

    View Slide

18. CODE COVERAGE
    TOOL, NOT A METRIC
    

    View Slide

19. COVERAGE.PY
    Treat the results of coverage.py as a
    litmus test.
    

    View Slide

20. COVERAGE.PY
    Treat the results of coverage.py as a
    litmus test.
    Use it to find paths in code that you
    missed in your tests.
    

    View Slide

21. Run tests in memory db,
    using multiprocess.
    NOSE, DJANGO-NOSE, AND MULTIPROCESS
    

    View Slide

22. IN-MEMORY SQLite DATABASE
    Databases in memory are extremely fast.
    Generally, back-end agnostic.
    More complicated when using database backend specific code.
    

    View Slide

23. NOSE + MULTIPROCESS
    Run test suite in multiple processes.
    Split tests, not TestCases so your tests need to be sure to ensure tearDown() happens.
    

    View Slide

24. GUIDELINES
    BETTER THAN FLAKEY TESTS
    

    View Slide

25. 1
    Test should be as simple as possible.
    

    View Slide

26. AVOID COMPLEXITY IN TESTS
    Harder for a reader of the test code to
    understand what is going on.
    The test code adds functionality which itself
    should be tested.
    

    View Slide

27. 2
    Each test method tests one thing, and one thing only.
    

    View Slide

28. TEST ONE THING
    Six assertions testing six different behaviors.
    Pre-conditions set four times.
    Should have split into separate test methods.
    Don’t do what you see here!
    

    View Slide

29. 3
    Only set up the minimum needed pre-conditions for
    your test.
    

    View Slide

30. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    

    View Slide

31. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    

    View Slide

32. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    

    View Slide

33. User
    UserProfile
    28 x Business Accounts
    Site
    5x Business
    1x Business (Personal)
    Currency
    Country
    Customer
    Vendor
    StandardAccount
    Account
    IncomeAccount
    ExpenseAccount
    Product
    PaymentTerm
    Tax (GST)
    Tax (PST)
    Tax (TEST)
    Invoice
    InvoiceItem (1)
    InvoiceItem (2)
    Bill
    BillItem (1)
    BillItem (2)
    MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    Loaded 2x test fixtures.
    Modified ownership for 6 businesses.
    

    View Slide

34. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    All this database record manipulation to test something that doesn’t even rely on the
    pre-conditions!
    

    View Slide

35. 4
    Create your pre-conditions explicitly - don’t use shared
    helper methods outside your module.
    

    View Slide

36. DRY DOESN’T APPLY WHEN TESTING
    Shared helper methods tend to grow over time.
    Shared helper methods try to be generic enough for everyone to use.
    Impose a lookup burden on readers of the test code.
    Tend to become complex over time (do your tests need tests)?
    

    View Slide

37. AVOID UNNECESSARY COUPLING IN YOUR TESTS
    When you refactor test code, you risk breaking unrelated tests.
    You no longer have direct control over your pre-conditions.
    Anyone can affect the confidence of your tests.
    

    View Slide

38. 5
    Name your TestCase methods to indicate what they
    actually test.
    

    View Slide

39. DON’T DEFINE DOCSTRINGS IN TESTS
    Docstrings in test methods make it
    hard to track down tests in output.
    Use simple python comments instead.
    

    View Slide

40. TEST METHOD NAMES DESCRIBE THE TEST
    Makes it easy to identify what the test is actually testing.
    Gives clues about what is wrong when tests fail.
    

    View Slide

41. 6
    Use factories, not fixtures.
    

    View Slide

42. FIXTURES ARE HORRIBLE
    Fixtures are hard to maintain as your project’s
    data models change.
    Tend to get re-used in tests rather than
    generating fixtures for each TestCase.
    

    View Slide

43. FACTORIES ARE WAY BETTER
    Factories generate data using the Django
    ORM.
    Give us hooks into data preparation.
    FactoryBoy lets us define data across
    relationships using familiar `double-
    underscore` notation.
    We can define associations in FactoryBoy to
    mimic our business logic.
    

    View Slide

44. 7
    Use django.tests.TestCase instead of unittest2.TestCase.
    

    View Slide

45. TESTS PASS, BUT SUITE FAILS
    Django TestCase’s run tests in a database
    transaction which gets rolled-back.
    History shows you can’t rely on developers to
    tearDown properly.
    * Only applies to the `default` database.
    

    View Slide

46. 8
    Create mixins, not shared TestCases.
    

    View Slide

47. MIXINS ARE NOT INCLUDED IN TEST DISCOVERY
    When a TestCase extends a BaseTestCase, it
    will run all tests, even the ones in the
    BaseTestCase.
    Leads to the same tests being run many
    times unnecessarily.
    Use a mixin - it will not be run by the test
    runner.
    

    View Slide

48. 9
    Segment your tests.
    

    View Slide

49. USE nose AND django-nose
    Decorate your tests and TestCases with @attr().
    Segment your tests based on type of tests.
    Selectively choose which type of tests to run using -a on the command line.
    

    View Slide

50. 10
    Don’t use setupClass() or tearDownClass().
    

    View Slide

51. CLEANUP IS A PAIN
    Can not guarantee that all the data is cleaned up - especially when using django signals.
    If we want to use multi-processing for tests, we need to be sure test methods can be run in any
    order.
    Over the entire test suite, did not give us enough gains to overcome the potential pitfalls.
    Using multi-processing for tests gave us much better wins.
    

    View Slide

52. Questions?
    @ashchristopher
    ash.christopher@gmail.com
    http://www.github.com/ashchristopher
    

    View Slide