Testing at Scale

Transcript

1. TESTING 3600 TESTS, AND COUNTING
2. TESTING AT SCALE WHAT ARE GOOD TESTS?
3. TESTS SHOULD BE AS SIMPLE AS POSSIBLE Don’t write test code that needs it’s own tests! Be clear - not clever. DRY is great for code, just not for test code.
4. TESTS SHOULD RUN AS QUICKLY AS POSSIBLE Encourage developers to run tests frequently. Sometimes running subset of tests may not be enough. Instill a culture of reliance on tests.
5. TESTS SHOULD AVOID COUPLING WITH OTHER TESTS TestCases should not extend TestCases in other python modules. Tests should not use helper methods which exist in other python modules.
6. TESTS SHOULD COMMUNICATE INTENT Tests are explicitly clear about what they are testing. Related test methods reside in the same TestCase. Tests should not rely on documentation to explain what they are doing. Developers reading the tests should not have to leave the TestCase code to determine what the test is doing.
7. CANONICAL FORM THE STRUCTURE OF A TEST
8. SET UP PRE-CONDITIONS Create the environment needed for the test to run. • Create model instances. • Set up the state required for the test. • Create mock objects. • Patch methods, functions and objects.
9. PERFORM OPERATION UNDER TEST Run test using the environment established in the pre- conditions. • Modify state. • Record return values.
10. MAKE ASSERTIONS Make assertions about the return values or side effects. • Check return values against expected values. • Test that the state has been modified as expected.
12. DO ABSOLUTELY NOTHING ELSE • Don’t modify the test conditions and test assertions again. • Don’t extend your test further. • Don’t create test methods that chain together main tests. • Write a new test method, and copy the pre-conditions.
13. BENEFITS OF FOLLOWING CANONICAL FORM Each test method tests exactly one path through the code under test. The set of TestCases will define a clear contract with the developer. For any test that fails, the cause will be easy to pinpoint because the expected results is clearly focused.
14. FACTORIES SO MUCH BETTER THAN FIXTURES
15. DJANGO APP called factory_boy Third-party Django application. Based on popular Rails library called factory_girl. Based on your own Django models. Replacement for fixtures.
16. Only define sub-factories for data that needs to be created. Model relations with null=True or blank=True then this isn’t generally a business need. ONLY CREATE DATA BASED ON BUSINESS NEEDS
17. Factories should only be responsible for creating data, not setting up pre-conditions for tests. FACTORIES CREATE DATA ONLY
18. CODE COVERAGE TOOL, NOT A METRIC
19. COVERAGE.PY Treat the results of coverage.py as a litmus test.
20. COVERAGE.PY Treat the results of coverage.py as a litmus test. Use it to find paths in code that you missed in your tests.
21. Run tests in memory db, using multiprocess. NOSE, DJANGO-NOSE, AND MULTIPROCESS
22. IN-MEMORY SQLite DATABASE Databases in memory are extremely fast. Generally, back-end agnostic. More complicated when using database backend specific code.
23. NOSE + MULTIPROCESS Run test suite in multiple processes. Split tests, not TestCases so your tests need to be sure to ensure tearDown() happens.
24. GUIDELINES BETTER THAN FLAKEY TESTS
25. 1 Test should be as simple as possible.
26. AVOID COMPLEXITY IN TESTS Harder for a reader of the test code to understand what is going on. The test code adds functionality which itself should be tested.
27. 2 Each test method tests one thing, and one thing only.
28. TEST ONE THING Six assertions testing six different behaviors. Pre-conditions set four times. Should have split into separate test methods. Don’t do what you see here!
29. 3 Only set up the minimum needed pre-conditions for your test.
30. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
31. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
32. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST
33. User UserProfile 28 x Business Accounts Site 5x Business 1x Business (Personal) Currency Country Customer Vendor StandardAccount Account IncomeAccount ExpenseAccount Product PaymentTerm Tax (GST) Tax (PST) Tax (TEST) Invoice InvoiceItem (1) InvoiceItem (2) Bill BillItem (1) BillItem (2) MINIMIZE PRECONDITIONS NEEDED FOR THE TEST Loaded 2x test fixtures. Modified ownership for 6 businesses.
34. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST All this database record manipulation to test something that doesn’t even rely on the pre-conditions!
35. 4 Create your pre-conditions explicitly - don’t use shared helper methods outside your module.
36. DRY DOESN’T APPLY WHEN TESTING Shared helper methods tend to grow over time. Shared helper methods try to be generic enough for everyone to use. Impose a lookup burden on readers of the test code. Tend to become complex over time (do your tests need tests)?
37. AVOID UNNECESSARY COUPLING IN YOUR TESTS When you refactor test code, you risk breaking unrelated tests. You no longer have direct control over your pre-conditions. Anyone can affect the confidence of your tests.
38. 5 Name your TestCase methods to indicate what they actually test.
39. DON’T DEFINE DOCSTRINGS IN TESTS Docstrings in test methods make it hard to track down tests in output. Use simple python comments instead.
40. TEST METHOD NAMES DESCRIBE THE TEST Makes it easy to identify what the test is actually testing. Gives clues about what is wrong when tests fail.
41. 6 Use factories, not fixtures.
42. FIXTURES ARE HORRIBLE Fixtures are hard to maintain as your project’s data models change. Tend to get re-used in tests rather than generating fixtures for each TestCase.
43. FACTORIES ARE WAY BETTER Factories generate data using the Django ORM. Give us hooks into data preparation. FactoryBoy lets us define data across relationships using familiar `double- underscore` notation. We can define associations in FactoryBoy to mimic our business logic.
44. 7 Use django.tests.TestCase instead of unittest2.TestCase.
45. TESTS PASS, BUT SUITE FAILS Django TestCase’s run tests in a database transaction which gets rolled-back. History shows you can’t rely on developers to tearDown properly. * Only applies to the `default` database.
46. 8 Create mixins, not shared TestCases.
47. MIXINS ARE NOT INCLUDED IN TEST DISCOVERY When a TestCase extends a BaseTestCase, it will run all tests, even the ones in the BaseTestCase. Leads to the same tests being run many times unnecessarily. Use a mixin - it will not be run by the test runner.
48. 9 Segment your tests.
49. USE nose AND django-nose Decorate your tests and TestCases with @attr(<attribute>). Segment your tests based on type of tests. Selectively choose which type of tests to run using -a<attribute> on the command line.
50. 10 Don’t use setupClass() or tearDownClass().
51. CLEANUP IS A PAIN Can not guarantee that all the data is cleaned up - especially when using django signals. If we want to use multi-processing for tests, we need to be sure test methods can be run in any order. Over the entire test suite, did not give us enough gains to overcome the potential pitfalls. Using multi-processing for tests gave us much better wins.
52. Questions? @ashchristopher ash.christopher@gmail.com http://www.github.com/ashchristopher