Tweet
Tweet

Slide 1

Slide 1 text

TESTING 3600 TESTS, AND COUNTING

Slide 2

Slide 2 text

TESTING AT SCALE WHAT ARE GOOD TESTS?

Slide 3

Slide 3 text

TESTS SHOULD BE AS SIMPLE AS POSSIBLE Don’t write test code that needs it’s own tests! Be clear - not clever. DRY is great for code, just not for test code.

Slide 4

Slide 4 text

TESTS SHOULD RUN AS QUICKLY AS POSSIBLE Encourage developers to run tests frequently. Sometimes running subset of tests may not be enough. Instill a culture of reliance on tests.

Slide 5

Slide 5 text

TESTS SHOULD AVOID COUPLING WITH OTHER TESTS TestCases should not extend TestCases in other python modules. Tests should not use helper methods which exist in other python modules.

Slide 6

Slide 6 text

TESTS SHOULD COMMUNICATE INTENT Tests are explicitly clear about what they are testing. Related test methods reside in the same TestCase. Tests should not rely on documentation to explain what they are doing. Developers reading the tests should not have to leave the TestCase code to determine what the test is doing.

Slide 7

Slide 7 text

CANONICAL FORM THE STRUCTURE OF A TEST

Slide 8

Slide 8 text

SET UP PRE-CONDITIONS Create the environment needed for the test to run. • Create model instances. • Set up the state required for the test. • Create mock objects. • Patch methods, functions and objects.

Slide 9

Slide 9 text

PERFORM OPERATION UNDER TEST Run test using the environment established in the pre- conditions. • Modify state. • Record return values.

Slide 10

Slide 10 text

MAKE ASSERTIONS Make assertions about the return values or side effects. • Check return values against expected values. • Test that the state has been modified as expected.

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

DO ABSOLUTELY NOTHING ELSE • Don’t modify the test conditions and test assertions again. • Don’t extend your test further. • Don’t create test methods that chain together main tests. • Write a new test method, and copy the pre-conditions.

Slide 13

Slide 13 text

BENEFITS OF FOLLOWING CANONICAL FORM Each test method tests exactly one path through the code under test. The set of TestCases will define a clear contract with the developer. For any test that fails, the cause will be easy to pinpoint because the expected results is clearly focused.

Slide 14

Slide 14 text

FACTORIES SO MUCH BETTER THAN FIXTURES

Slide 15

Slide 15 text

DJANGO APP called factory_boy Third-party Django application. Based on popular Rails library called factory_girl. Based on your own Django models. Replacement for fixtures.

Slide 16

Slide 16 text

Only define sub-factories for data that needs to be created. Model relations with null=True or blank=True then this isn’t generally a business need. ONLY CREATE DATA BASED ON BUSINESS NEEDS

Slide 17

Slide 17 text

Factories should only be responsible for creating data, not setting up pre-conditions for tests. FACTORIES CREATE DATA ONLY

Slide 18

Slide 18 text

CODE COVERAGE TOOL, NOT A METRIC

Slide 19

Slide 19 text

COVERAGE.PY Treat the results of coverage.py as a litmus test.

Slide 20

Slide 20 text

COVERAGE.PY Treat the results of coverage.py as a litmus test. Use it to find paths in code that you missed in your tests.

Slide 21

Slide 21 text

Run tests in memory db, using multiprocess. NOSE, DJANGO-NOSE, AND MULTIPROCESS

Slide 22

Slide 22 text

IN-MEMORY SQLite DATABASE Databases in memory are extremely fast. Generally, back-end agnostic. More complicated when using database backend specific code.

Slide 23

Slide 23 text

NOSE + MULTIPROCESS Run test suite in multiple processes. Split tests, not TestCases so your tests need to be sure to ensure tearDown() happens.

Slide 24

Slide 24 text

GUIDELINES BETTER THAN FLAKEY TESTS

Slide 25

Slide 25 text

1 Test should be as simple as possible.

Slide 26

Slide 26 text

AVOID COMPLEXITY IN TESTS Harder for a reader of the test code to understand what is going on. The test code adds functionality which itself should be tested.

Slide 27

Slide 27 text

2 Each test method tests one thing, and one thing only.

Slide 28

Slide 28 text

TEST ONE THING Six assertions testing six different behaviors. Pre-conditions set four times. Should have split into separate test methods. Don’t do what you see here!

Slide 29

Slide 29 text

3 Only set up the minimum needed pre-conditions for your test.

Slide 30

Slide 30 text

MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

Slide 31

Slide 31 text

MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

Slide 32

Slide 32 text

MINIMIZE PRECONDITIONS NEEDED FOR THE TEST

Slide 33

Slide 33 text

User UserProfile 28 x Business Accounts Site 5x Business 1x Business (Personal) Currency Country Customer Vendor StandardAccount Account IncomeAccount ExpenseAccount Product PaymentTerm Tax (GST) Tax (PST) Tax (TEST) Invoice InvoiceItem (1) InvoiceItem (2) Bill BillItem (1) BillItem (2) MINIMIZE PRECONDITIONS NEEDED FOR THE TEST Loaded 2x test fixtures. Modified ownership for 6 businesses.

Slide 34

Slide 34 text

MINIMIZE PRECONDITIONS NEEDED FOR THE TEST All this database record manipulation to test something that doesn’t even rely on the pre-conditions!

Slide 35

Slide 35 text

4 Create your pre-conditions explicitly - don’t use shared helper methods outside your module.

Slide 36

Slide 36 text

DRY DOESN’T APPLY WHEN TESTING Shared helper methods tend to grow over time. Shared helper methods try to be generic enough for everyone to use. Impose a lookup burden on readers of the test code. Tend to become complex over time (do your tests need tests)?

Slide 37

Slide 37 text

AVOID UNNECESSARY COUPLING IN YOUR TESTS When you refactor test code, you risk breaking unrelated tests. You no longer have direct control over your pre-conditions. Anyone can affect the confidence of your tests.

Slide 38

Slide 38 text

5 Name your TestCase methods to indicate what they actually test.

Slide 39

Slide 39 text

DON’T DEFINE DOCSTRINGS IN TESTS Docstrings in test methods make it hard to track down tests in output. Use simple python comments instead.

Slide 40

Slide 40 text

TEST METHOD NAMES DESCRIBE THE TEST Makes it easy to identify what the test is actually testing. Gives clues about what is wrong when tests fail.

Slide 41

Slide 41 text

6 Use factories, not fixtures.

Slide 42

Slide 42 text

FIXTURES ARE HORRIBLE Fixtures are hard to maintain as your project’s data models change. Tend to get re-used in tests rather than generating fixtures for each TestCase.

Slide 43

Slide 43 text

FACTORIES ARE WAY BETTER Factories generate data using the Django ORM. Give us hooks into data preparation. FactoryBoy lets us define data across relationships using familiar `double- underscore` notation. We can define associations in FactoryBoy to mimic our business logic.

Slide 44

Slide 44 text

7 Use django.tests.TestCase instead of unittest2.TestCase.

Slide 45

Slide 45 text

TESTS PASS, BUT SUITE FAILS Django TestCase’s run tests in a database transaction which gets rolled-back. History shows you can’t rely on developers to tearDown properly. * Only applies to the `default` database.

Slide 46

Slide 46 text

8 Create mixins, not shared TestCases.

Slide 47

Slide 47 text

MIXINS ARE NOT INCLUDED IN TEST DISCOVERY When a TestCase extends a BaseTestCase, it will run all tests, even the ones in the BaseTestCase. Leads to the same tests being run many times unnecessarily. Use a mixin - it will not be run by the test runner.

Slide 48

Slide 48 text

9 Segment your tests.

Slide 49

Slide 49 text

USE nose AND django-nose Decorate your tests and TestCases with @attr(). Segment your tests based on type of tests. Selectively choose which type of tests to run using -a on the command line.

Slide 50

Slide 50 text

10 Don’t use setupClass() or tearDownClass().

Slide 51

Slide 51 text

CLEANUP IS A PAIN Can not guarantee that all the data is cleaned up - especially when using django signals. If we want to use multi-processing for tests, we need to be sure test methods can be run in any order. Over the entire test suite, did not give us enough gains to overcome the potential pitfalls. Using multi-processing for tests gave us much better wins.

Slide 52

Slide 52 text

Questions? @ashchristopher [email protected] http://www.github.com/ashchristopher