Another angle on test infrastructure by Iwan Vosloo

Last year at PyConZA I presented a talk about different approaches to test setup and alluded to an idea that we have been playing with at Reahl: class based test Fixtures. We have since refined our approach and built it as an add-on that works with py.test.

In this talk I briefly introduce Fixtures again for people who did not see the last year's talk. I then show more detail about the types of problems we deal with while testing and how our fixture ideas solve these issues.

For example, in our tests the database schema is created from scratch at the beginning of a test run, but the database is kept clean by rolling back a transaction between each individual test. We can also create extra temporary database tables, just for a test run.

We integrate Selenium webdriver with our web server so that the web server runs in the same thread as the tests themselves, in the same database transaction. The effect of this is that when something breaks inside the web server, the test immediately breaks as well, and gives a sensible stack trace.

In our code, we use a global context much like what Flask does. This is a design often criticised because of the fact that it makes testing difficult. We use our fixtures to hide the associated complexities regarding the global context as well.

The talk covers some details about how we use Fixtures to provide this functionality and some thoughts on where we found it useful and which things we still struggle to deal with elegantly.

Transcript

1. PyConZA 2017 Another angle on test infrastructure Iwan Vosloo
2. PyConZA 2017 Reahl Fixtures TestFunctio n Fixture Person Bank Account https://2016.za.pycon.org/talks/26/
3. PyConZA 2017 Fixture basics class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) with MyFixture() as fixture: assert fixture.person.name == 'John' assert fixture.person is fixture.person
4. PyConZA 2017 Beyond defaults class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) with MyFixture() as fixture: jane = fixture.new_person(name='Jane') assert jane is not fixture.person
5. PyConZA 2017 Simplifying dependencies class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) def new_bank_account(number='123', person=None): return BankAccount(number=number, owner=person or self.person) with MyFixture() as fixture: assert fixture.bank_account.owner is fixture.person
6. PyConZA 2017 Set up & Tear down class MyFixture(Fixture): @set_up def start_server(self): ... with MyFixture() as fixture: assert fixture.bank_account.owner is fixture.person set_up() tear_down()
7. PyConZA 2017 pytest integration class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) @with_fixtures(MyFixture) def test_something(fixture): assert fixture.person.name == 'John' assert fixture.person is fixture.person
8. PyConZA 2017 Multiple Fixtures class MyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) @with_fixtures(MyFixture, AnotherFixture) def test_something(fixture, another): assert fixture.person.name == 'John' assert fixture.person is fixture.person login = another.new_login(fixture.person) ...
9. PyConZA 2017 Dependent Fixtures class PartyFixture(Fixture): def new_person(self, name='John'): return Person(name=name) @uses(parties=PartyFixture) class BankingFixture(Fixture) def new_bank_account(number='123', person=None): return BankAccount(number=number, owner=person or self.parties.person) @with_fixtures(BankingFixture) def test_something(banking): assert fixture.bank_account.owner is fixture.parties.person
10. PyConZA 2017 Fixture scope @scope('session') class Database(Fixture): def new_connection(self): ... @uses(database=Database) class PartyFixture(Fixture): def new_person(self, name='John'): # do stuff with self.database.connection # (which is only created once per session)
11. PyConZA 2017 Motivations • No magic - its all Python and imports • Each Fixture has responsibility for things that belong together • Dependencies on other Fixtures that have other responsibilities • The (other Fixture) dependencies of a Fixture is its own business • Able to re-use Fixtures without having to inherit
12. PyConZA 2017 Configuration and database «session» ReahlSystem SessionFixture @set_up(connect_db) @tear_down(disconnect_db) system_control config Configuration
13. PyConZA 2017 Isolating state @with_fixtures( ReahlSystemFixture ) def test_things( reahl_system ): reahl_system.config.some_setting = 'changed'
14. PyConZA 2017 Isolating state «session» ReahlSystem SessionFixture @set_up(connect_db) @tear_down(disconnect_db) system_control config «function» ReahlSystem Fixture system_control config Configuration Configuration copy of
15. PyConZA 2017 Controlling SqlAlchemy «session» ReahlSystem SessionFixture «function» ReahlSystem Fixture «function» SqlAlchemyFixture @set_up(start_transaction) @tear_down(abort_transaction) persistent_test_classes
16. PyConZA 2017 Using SqlAlchemy @with_fixtures( SqlAlchemyFixture ) def test_things( sql_alchemy ): Session.add( Person(name='Jane') ) assert Session.query( Person ).count() == 1
17. PyConZA 2017 Temporary tables @with_fixtures( SqlAlchemyFixture ) def test_things( sql_alchemy ): class MyTestObject( Base ): __tablename__ == 'my_object' name = Column( String, primary_key=True ) def __init__(self, name): self.name = name with sql_alchemy.persistent_test_classes( MyTestObject ): Session.add( MyTestObject(name='A') ) assert Session.query( MyTestObject ).count() == 1
18. PyConZA 2017 The web server web browser pytest process web server process database selenium
19. PyConZA 2017 Separate server transaction pytest process pytest transaction web browser web server web server transaction create_objects() commit() visit_web_page() modify_objects() http_get() commit() check_modified_ objects()
20. PyConZA 2017 Server-side breakage pytest process web browser web server visit_web_page() http_get() breaks http_500() check_web_page()
21. PyConZA 2017 Integrating the web server web browser database pytest code web server code selenium
22. PyConZA 2017 Fixtures for web «session» WebServerFixture @set_up(start_servers) @tear_down(stop_servers) reahl_server web_driver «function» WebFixture reahl_server driver_browser log_in «function» PartyAccount Fixture person system_account
23. PyConZA 2017 File uploads example
24. PyConZA 2017 File uploads example
25. PyConZA 2017 File uploads example
26. PyConZA 2017 Testing file uploads @with_fixtures(WebFixture, FileUploadInputFixture) def test_file_upload_input_basics(web_fixture, fixture): web_fixture.reahl_server.set_app(fixture.wsgi_app) browser = web_fixture.driver_browser assert not fixture.file_was_uploaded( fixture.file_to_upload1.name ) assert not fixture.file_was_uploaded( fixture.file_to_upload2.name ) # Upload one file browser.type(XPath.input_labelled('Choose file(s)'), fixture.file_to_upload1.name) browser.click(XPath.button_labelled('Upload')) assert fixture.file_was_uploaded( fixture.file_to_upload1.name ) assert not fixture.file_was_uploaded( fixture.file_to_upload2.name )
27. PyConZA 2017 Testing file uploads @with_fixtures(WebFixture, FileUploadInputFixture) def test_file_upload_input_basics(web_fixture, fixture): web_fixture.reahl_server.set_app(fixture.wsgi_app) browser = web_fixture.driver_browser assert not fixture.file_was_uploaded( fixture.file_to_upload1.name ) assert not fixture.file_was_uploaded( fixture.file_to_upload2.name ) # Upload one file browser.type(XPath.input_labelled('Choose file(s)'), fixture.file_to_upload1.name) browser.click(XPath.button_labelled('Upload')) assert fixture.file_was_uploaded( fixture.file_to_upload1.name ) assert not fixture.file_was_uploaded( fixture.file_to_upload2.name )
28. PyConZA 2017 File uploads example
29. PyConZA 2017 Beyond the rule assert not fixture.file_was_uploaded( fixture.file_to_upload1.name ) assert not fixture.uploaded_file_is_listed( fixture.file_to_upload1.name ) with web_fixture.reahl_server.in_background(wait_till_done_serving=False): # Upload will block, see fixture: browser.type(XPath.input_labelled('Choose file(s)'), fixture.file_to_upload1.name) assert fixture.progess_bar_is_displayed( fixture.file_to_upload1.name ) browser.click(XPath.button_labelled('Cancel')) assert not fixture.uploaded_file_is_listed( fixture.file_to_upload1.name ) assert not fixture.file_was_uploaded( fixture.file_to_upload1.name )
30. PyConZA 2017 The ExecutionContext • Who is logged in? • What locale to use to decide the current language etc. • Various system configuration settings. • The current database connection / transaction / Session • (The current request)
31. PyConZA 2017 Finding it ExecutionContext context = ExecutionContext.get_context() context.config.web.default_url_locale context.config.web.default_http_port
32. PyConZA 2017 Behind the scenes... @with_fixtures( SqlAlchemyFixture ) def test_things( sql_alchemy ): Session.add( Person(name='Jane') ) assert Session.query( Person ).count() == 1 context = ExecutionContext.get_context()
33. PyConZA 2017 Foundational Fixtures «session» ReahlSystem SessionFixture «function» ReahlSystem Fixture «function» SqlAlchemyFixture «session» WebServerFixture «function» WebFixture «function» PartyAccount Fixture «function» ContextAware Fixture
34. PyConZA 2017 Lingering issues • changing implementations of browsers • switching javascript on and off • testing browser GUI stuff • CSS effects and layout
35. PyConZA 2017 Thank you On groups.google.com: • reahl-discuss www.reahl.org iwan@reahl.org Slides: https://goo.gl/sGtmcE