Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing at Scale

Testing at Scale

Ash Christopher

June 13, 2013
Tweet

More Decks by Ash Christopher

Other Decks in Technology

Transcript

  1. TESTING
    3600 TESTS, AND COUNTING

    View Slide

  2. TESTING AT SCALE
    WHAT ARE GOOD TESTS?

    View Slide

  3. TESTS SHOULD BE AS SIMPLE AS POSSIBLE
    Don’t write test code that needs it’s own tests!
    Be clear - not clever.
    DRY is great for code, just not for test code.

    View Slide

  4. TESTS SHOULD RUN AS QUICKLY AS POSSIBLE
    Encourage developers to run tests frequently.
    Sometimes running subset of tests may not be
    enough.
    Instill a culture of reliance on tests.

    View Slide

  5. TESTS SHOULD AVOID COUPLING WITH OTHER TESTS
    TestCases should not extend TestCases in other
    python modules.
    Tests should not use helper methods which exist in
    other python modules.

    View Slide

  6. TESTS SHOULD COMMUNICATE INTENT
    Tests are explicitly clear about what they are testing.
    Related test methods reside in the same TestCase.
    Tests should not rely on documentation to explain what
    they are doing.
    Developers reading the tests should not have to leave
    the TestCase code to determine what the test is doing.

    View Slide

  7. CANONICAL FORM
    THE STRUCTURE OF A TEST

    View Slide

  8. SET UP PRE-CONDITIONS
    Create the environment needed for the test to run.
    • Create model instances.
    • Set up the state required for the test.
    • Create mock objects.
    • Patch methods, functions and objects.

    View Slide

  9. PERFORM OPERATION UNDER TEST
    Run test using the environment established in the pre-
    conditions.
    • Modify state.
    • Record return values.

    View Slide

  10. MAKE ASSERTIONS
    Make assertions about the return values or side effects.
    • Check return values against expected values.
    • Test that the state has been modified as expected.

    View Slide

  11. View Slide

  12. DO ABSOLUTELY NOTHING ELSE
    • Don’t modify the test conditions and test assertions again.
    • Don’t extend your test further.
    • Don’t create test methods that chain together main tests.
    • Write a new test method, and copy the pre-conditions.

    View Slide

  13. BENEFITS OF FOLLOWING CANONICAL FORM
    Each test method tests exactly one path through the code under test.
    The set of TestCases will define a clear contract with the developer.
    For any test that fails, the cause will be easy to pinpoint because the expected results is clearly
    focused.

    View Slide

  14. FACTORIES
    SO MUCH BETTER THAN FIXTURES

    View Slide

  15. DJANGO APP called factory_boy
    Third-party Django application.
    Based on popular Rails library called factory_girl.
    Based on your own Django models.
    Replacement for fixtures.

    View Slide

  16. Only define sub-factories for data that needs to be created.
    Model relations with null=True or blank=True then this isn’t generally a business need.
    ONLY CREATE DATA BASED ON BUSINESS NEEDS

    View Slide

  17. Factories should only be responsible for creating data, not setting up pre-conditions for tests.
    FACTORIES CREATE DATA ONLY

    View Slide

  18. CODE COVERAGE
    TOOL, NOT A METRIC

    View Slide

  19. COVERAGE.PY
    Treat the results of coverage.py as a
    litmus test.

    View Slide

  20. COVERAGE.PY
    Treat the results of coverage.py as a
    litmus test.
    Use it to find paths in code that you
    missed in your tests.

    View Slide

  21. Run tests in memory db,
    using multiprocess.
    NOSE, DJANGO-NOSE, AND MULTIPROCESS

    View Slide

  22. IN-MEMORY SQLite DATABASE
    Databases in memory are extremely fast.
    Generally, back-end agnostic.
    More complicated when using database backend specific code.

    View Slide

  23. NOSE + MULTIPROCESS
    Run test suite in multiple processes.
    Split tests, not TestCases so your tests need to be sure to ensure tearDown() happens.

    View Slide

  24. GUIDELINES
    BETTER THAN FLAKEY TESTS

    View Slide

  25. 1
    Test should be as simple as possible.

    View Slide

  26. AVOID COMPLEXITY IN TESTS
    Harder for a reader of the test code to
    understand what is going on.
    The test code adds functionality which itself
    should be tested.

    View Slide

  27. 2
    Each test method tests one thing, and one thing only.

    View Slide

  28. TEST ONE THING
    Six assertions testing six different behaviors.
    Pre-conditions set four times.
    Should have split into separate test methods.
    Don’t do what you see here!

    View Slide

  29. 3
    Only set up the minimum needed pre-conditions for
    your test.

    View Slide

  30. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

    View Slide

  31. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

    View Slide

  32. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

    View Slide

  33. User
    UserProfile
    28 x Business Accounts
    Site
    5x Business
    1x Business (Personal)
    Currency
    Country
    Customer
    Vendor
    StandardAccount
    Account
    IncomeAccount
    ExpenseAccount
    Product
    PaymentTerm
    Tax (GST)
    Tax (PST)
    Tax (TEST)
    Invoice
    InvoiceItem (1)
    InvoiceItem (2)
    Bill
    BillItem (1)
    BillItem (2)
    MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    Loaded 2x test fixtures.
    Modified ownership for 6 businesses.

    View Slide

  34. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
    All this database record manipulation to test something that doesn’t even rely on the
    pre-conditions!

    View Slide

  35. 4
    Create your pre-conditions explicitly - don’t use shared
    helper methods outside your module.

    View Slide

  36. DRY DOESN’T APPLY WHEN TESTING
    Shared helper methods tend to grow over time.
    Shared helper methods try to be generic enough for everyone to use.
    Impose a lookup burden on readers of the test code.
    Tend to become complex over time (do your tests need tests)?

    View Slide

  37. AVOID UNNECESSARY COUPLING IN YOUR TESTS
    When you refactor test code, you risk breaking unrelated tests.
    You no longer have direct control over your pre-conditions.
    Anyone can affect the confidence of your tests.

    View Slide

  38. 5
    Name your TestCase methods to indicate what they
    actually test.

    View Slide

  39. DON’T DEFINE DOCSTRINGS IN TESTS
    Docstrings in test methods make it
    hard to track down tests in output.
    Use simple python comments instead.

    View Slide

  40. TEST METHOD NAMES DESCRIBE THE TEST
    Makes it easy to identify what the test is actually testing.
    Gives clues about what is wrong when tests fail.

    View Slide

  41. 6
    Use factories, not fixtures.

    View Slide

  42. FIXTURES ARE HORRIBLE
    Fixtures are hard to maintain as your project’s
    data models change.
    Tend to get re-used in tests rather than
    generating fixtures for each TestCase.

    View Slide

  43. FACTORIES ARE WAY BETTER
    Factories generate data using the Django
    ORM.
    Give us hooks into data preparation.
    FactoryBoy lets us define data across
    relationships using familiar `double-
    underscore` notation.
    We can define associations in FactoryBoy to
    mimic our business logic.

    View Slide

  44. 7
    Use django.tests.TestCase instead of unittest2.TestCase.

    View Slide

  45. TESTS PASS, BUT SUITE FAILS
    Django TestCase’s run tests in a database
    transaction which gets rolled-back.
    History shows you can’t rely on developers to
    tearDown properly.
    * Only applies to the `default` database.

    View Slide

  46. 8
    Create mixins, not shared TestCases.

    View Slide

  47. MIXINS ARE NOT INCLUDED IN TEST DISCOVERY
    When a TestCase extends a BaseTestCase, it
    will run all tests, even the ones in the
    BaseTestCase.
    Leads to the same tests being run many
    times unnecessarily.
    Use a mixin - it will not be run by the test
    runner.

    View Slide

  48. 9
    Segment your tests.

    View Slide

  49. USE nose AND django-nose
    Decorate your tests and TestCases with @attr().
    Segment your tests based on type of tests.
    Selectively choose which type of tests to run using -a on the command line.

    View Slide

  50. 10
    Don’t use setupClass() or tearDownClass().

    View Slide

  51. CLEANUP IS A PAIN
    Can not guarantee that all the data is cleaned up - especially when using django signals.
    If we want to use multi-processing for tests, we need to be sure test methods can be run in any
    order.
    Over the entire test suite, did not give us enough gains to overcome the potential pitfalls.
    Using multi-processing for tests gave us much better wins.

    View Slide

  52. Questions?
    @ashchristopher
    [email protected]
    http://www.github.com/ashchristopher

    View Slide