Experiments with test setup by Iwan Vosloo

Ever since test driven development took hold, people have been experimenting with different ways to deal with setting up and tearing down test objects and data (amongst other things).

Focussing on this particular concern (set up and tear down of test objects), this talk provides an overview of how the mainstream python tools have developed over time in this regard: unittest, nose and py.test. A bit of a wider context is also given in terms of two ideas pioneered by tools in other languages: the annotations of TestNG and resources of Smalltalk's SUnit. Two ideas that address problems beyond the reach of the tools themselves are also introduced, namely the "object mother" and "builder pattern".

Against the backdrop of this overview, I also show some of our own experiments (as part of the Reahl project) to translate a combination of the object mother and the builder pattern into Python -- with surprising results.

The talk is aimed at people interested in improving the ways we can do set up for tests and people who are generally interested in how tests can be made easier to write and more useful. I hope to stimulate more thoughts around the topic against the backdrop of a slight overview. The talk is accessible to newcomers to this topic as well.

Transcript

1. PyConZA 2016 Experiments with test setup Iwan Vosloo
2. PyConZA 2016 Introduction Reahl https://goo.gl/3DMHFH data_table = DataTable(view, ….. )
3. PyConZA 2016 Context Explaining stuff
4. PyConZA 2016 Emergence Context Contents hide details Before we proceed Games and Numbers Maps, Game Theory and Computer-based Modelling Checkers Neural Nets ...
5. PyConZA 2016 Emergence Context Contents provide context Before we proceed Games and Numbers Maps, Game Theory and Computer-based Modelling Game Theory Emergence—A First Look Dynamic Models
6. PyConZA 2016 Context Contents provide context Before we proceed Games and Numbers Maps, Game Theory and Computer-based Modelling Emergence Game Theory States Tree of Moves Strategies
7. PyConZA 2016 Context Contents of a system widgets input & validation page flow reahl-web security advanced concepts bootstrap widget basics HTML-derived widgets layout event handling inputs fields (see also component) widget query arguments files
8. PyConZA 2016 Context Where tests come in widgets reahl-web widget basics Basic widgets are created by adding children widgets to them, and the result is rendered in HTML. Widgets are rendered only if their .visible property is True. Widgets can be created from factories which allow you to supply widget-specific args and/or kwargs before a view is available.
9. PyConZA 2016 Context Test conventions PURPOSE • The test must explain something CONTEXT • The test's topic must fit underneath a topic in the table of contents SCOPE • The test must only cover its own topic DECOUPLED from IMPLEMENTATION • The test must not reveal implementation details unrelated to its topic • Stub out stuff only when really necessary • Use stubs, not mocks (See http://martinfowler.com/articles/mocksArentStubs.html )
10. PyConZA 2016 Evolution tour The prehistory of xUnit C3 Crysler Comprehensive Compensation Kent Beck Ron Jeffreys Martin Fowler eXtreme Programming Agile Testing frameworks Refactoring TDD
11. PyConZA 2016 Evolution tour The Python lineage SUnit (SmallTalk) JUnit (Java) unittest (Python) zope. testing twisted Trial django .test other stuff nose pytest
12. PyConZA 2016 Evolution tour Definitions Fixture All the stuff your test is going to need to run A single test TestCase TestSuite A grouping of tests
13. PyConZA 2016 Evolution tour unittest class TestSequenceFunctions(unittest.TestCase): def setUp(self): self.seq = range(10) def test_shuffle(self): # make sure the shuffled sequence does not lose any elements random.shuffle(self.seq) self.seq.sort() self.assertEqual(self.seq, range(10)) def test_choice(self): element = random.choice(self.seq) self.assertTrue(element in self.seq)
14. PyConZA 2016 Evolution tour Is this really clear? TestCase TestSequence Functions setUp() test_shuffle() test_choice() list seq a “test case” a “test case” a method that creates “all the stuff comprising the fixture”
15. PyConZA 2016 Evolution tour Another perspective TestSequence Functions setUp() test_shuffle() test_choice() list seq a “test case” a “test case” a shared fixture
16. PyConZA 2016 Evolution tour Reusing setUp (take 1) TestSequence Functions list seq TestSomeOther RelatedStuff setUp() test_shuffle() test_choice()
17. PyConZA 2016 Evolution tour Reusing setUp (take 2) TestSequence Functions list seq TestSomeOther RelatedStuff setUp() test_shuffle() test_choice()
18. PyConZA 2016 Evolution tour The devil in the details class TestSomeOtherRelatedStuff(unittest.TestCase): def setUp(self): self.sf = TestSequenceFunctions('test_stuff') self.sf.setUp() def test_stuff2(self): assert self.sf.seq == list(range(10)) def tearDown(self): self.sf.tearDown()
19. PyConZA 2016 Evolution tour Reusing setUp (take 3) SequenceFunction Fixture list seq TestSomeOther RelatedStuff setUp() TestSequence Functions
20. PyConZA 2016 Evolution tour The other problem :TestCase :TestCase :TestCase :Database Connection
21. PyConZA 2016 Evolution tour Along came Nose my_project my_tests test_package_a test_package_b test_module_two.py test_module_one.py __init__.py class TestSomething(TestCase): @classmethod def setup_class(cls): # create a fixture def set_up(self): # some code def test_one_thing(self): # some test code… def tear_down(self): # some code @classmethod def teardown_class(cls): # tear down a fixture setup() teardown() setup() teardown()
22. PyConZA 2016 Evolution tour The trouble with nose fixtures my_project my_tests test_package_a test_package_b test_module_one.py test_module_one.py __init__.py You're constrained in how you group your tests into classes and modules by what resources you want to use in them. What if you want the test framework to run tests in parallel or change the order in which they are run?
23. PyConZA 2016 Evolution tour Flashback: organising tests widgets input & validation page flow reahl-web security advanced concepts bootstrap widget basics HTML-derived widgets layout event handling inputs fields (see also component) widget query arguments files
24. PyConZA 2016 Evolution tour The trouble with nose fixtures my_project my_tests test_package_a test_package_b test_module_one.py test_module_one.py __init__.py You're constrained in how you group your tests into classes and modules by what resources you want to use in them. What if you want the test framework to run tests in parallel or change the order in which they are run?
25. PyConZA 2016 Evolution tour What we want How tests are organised The order in which tests are run How test setup is reused The lifecycle & scope of what is set up
26. PyConZA 2016 Evolution tour Annotating things: TestNG public class MyTest { @Test public void widgetCanBeResized() { // your test code... } @BeforeMethod public void createWidget() { // some code…. } @BeforeClass public void startSeleniumBrowser() { // some code…. } @AfterMethod public void destroyWidget() { // some code…. } @AfterClass public void stopSeleniumBrowser() { // some code…. } }
27. PyConZA 2016 Evolution tour Test resources: SUnit TestCase TestResource setUp tearDown reset * * SomeServer
28. PyConZA 2016 Evolution tour Python testresources TestCase TestResource Manager make clean _reset * * BzrPopulated Branch * * class TestLog(testresources.ResourcedTestCase): resources = [('branch', BzrPopulatedBranch())] def test_log(self): show_log(self.branch, ...)
29. PyConZA 2016 Evolution tour Summary • Organising your tests • Fixture • Re-using the code that creates & disposes of a Fixture • Re-using an instance of the Fixture for several tests • Using annotations to add metadata to tests • Entire Fixtures vs individual TestResources
30. PyConZA 2016 Evolution tour On to pytest def test_something(some_object): # do something with some_object assert isinstance(some_object, MyObject) @pytest.fixture(scope='session') def some_object(): my_object = MyObject() # do some stuff to setup my_object yield my_object # do some stuff to tear down my_object @pytest.fixture def some_object(): return MyObject()
31. PyConZA 2016 Evolution tour Analysing pytest test function pytest.fixture (fixture function?) * * TestCase TestResource * *
32. PyConZA 2016 Evolution tour Analysing pytest TestCase TestResource * * Context * 1 contains uses * * * 1
33. PyConZA 2016 Beyond the tools A different kind of problem “upon settlement of an investment, commission is paid to a designated personal bank account of the applicable advisor” Person Advisor Role * * BankAccount designation Investor Account Investment ContactDetails FICADocument
34. PyConZA 2016 Beyond the tools def test_commission_payment_destination(investment_objects): """upon a new investment, commissions are paid to a designated personal bank account of the applicable advisor""" bank_account = investment_objects.get_advisor_commission_bank_account() investment = investment_objects.get_unsettled_investment() assert bank_account.balance.is_zero investment.settle() assert bank_account.balance == investment.commission_amount The "object mother"
35. PyConZA 2016 Beyond the tools class InvestmentObjects: """Test objects related to making investments""" def get_advisor_commission_bank_account(self): # code def get_unsettled_investment(self): # code def get_advisor(self): # code def get_advisor_with_invalid_license(self): # code def get_investor_with_outstanding_FICA(self): # code The "object mother" problem
36. PyConZA 2016 Beyond the tools The builder pattern PersonBuilder String Contact Detail BankAccount name contact details bank accounts withName(name) build() Person
37. PyConZA 2016 Beyond the tools The builder pattern Person john = new PersonBuilder().build(); Person jane = new PersonBuilder(). withName('Jane'). build(); Person person = new PersonBuilder(). withFicaOutstanding(). withoutContactDetails(). build();
38. PyConZA 2016 Beyond the tools The builder pattern class PersonBuilder { String name; List<ContactDetail> contactDetails; List<BankAccounts> bankAccounts; PersonBuilder() { name = 'John'; } PersonBuilder withName(String name) { this.name = name; return this; } Person build() { Person aPerson = new Person(name); // use all the data collected in the builder for the Person… return aPerson } }
39. PyConZA 2016 Beyond the tools A Pythonic builder? def new_person(name='John', contact_details=None, bank_accounts=None): if not contact_details: contact_details = [new_postal_address()] if not bank_accounts: bank_accounts = [new_bank_account()] person = Person(name=name) person.add_bank_accounts(bank_accounts) person.add_contact_details(contact_details) return person
40. PyConZA 2016 Beyond the tools A Pythonic builder? john = new_person() jane = new_person(name='Jane') person = new_person(fica_compliant=False, contact_details=[])
41. PyConZA 2016 Beyond the tools The best of both worlds class Fixture: def new_person(self, name='John', contact_details=None, bank_accounts=None): if contact_details is None: contact_details = [self.new_contact_detail()] person = Person(name=name) # etc def new_contact_detail(self): # etc def new_bank_account(self) # etc
42. PyConZA 2016 Our own experiment Reahl Fixtures class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) def new_bank_account(number='123', person=None): return BankAccount(number=number, owner=person or self.person) fixture = MyFixture(None) assert fixture.person is fixture.person assert fixture.bank_account.owner is fixture.person jane = fixture.new_person(name='Jane')
43. PyConZA 2016 Our own experiment Reahl Fixtures TestCase TestResource * * uses * * Fixture
44. PyConZA 2016 Our own experiment Reahl Fixtures class MyFixture(Fixture): @set_up def start_transaction(self): #... @tear_down def abort_transaction(self): #... def new_person(self, name='John'): return Person(name=name) with MyFixture(None) as fixture: person = fixure.person # change the database abort_transaction() start_transaction()
45. PyConZA 2016 Our own experiment Reahl Fixtures with pytest class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) @pytest.fixture def persons(): with MyFixture(None) as fixture: yield fixture def test_something(persons): assert persons.person.name == 'John'
46. PyConZA 2016 Thank you On groups.google.com: • reahl-discuss www.reahl.org iwan@reahl.org Slides: https://goo.gl/hM1zzj