Building an “Open” Test Automation Framework

Building an open test automation (TA) framework takes more than just selecting a TA framework that is “Open Source”.

As frameworks evolve they inevitably become more “closed”. Extensions, customizations and even simple support or “helper” libraries add to the TA developer’s learning curve. This can limit the useful life of the framework and all the automated tests that depend on it.

A TA framework that is seen as “closed” has an increased chance of being replaced after the original architects move on to other projects. Rebuilding a TA framework (and all of its tests) every few years comes at a high cost in terms of time and money.

This talk discusses strategies to designing a truly “Open" TA framework which could outlive even a total product refactor.

Presented at a Software Test & Automation - Montréal Meetup.

Transcript

1. Building an Open Test Automation Framework - November 28, 2017 Building an “Open” Test Automation Framework Building an open test automation (TA) framework takes more than just selecting a TA framework that is “Open Source”. As frameworks evolve they inevitably become more “closed”. Extensions, customizations and even simple support or “helper” libraries add to the TA developer’s learning curve. This can limit the useful life of the framework and all the automated tests that depend on it. A TA framework that is seen as “closed” has an increased chance of being replaced after the original architects move on to other projects. Rebuilding a TA framework (and all of its tests) every few years comes at a high cost in terms of time and money. This talk discusses strategies to designing a truly “Open" TA framework which could outlive even a total product refactor.
2. Building an Open Test Automation Framework - November 28, 2017 Agenda ▪ Introduction ▪ Open TA ▪ Inclusion ▪ Selection ▪ Documentation ▪ Abstraction ▪ Integration ▪ Conclusion
3. Building an Open Test Automation Framework - November 28, 2017 Introduction ▪ Ted Landis, CTS Architect at Lightspeed ▪ TA Manager at Lightspeed (3 years) ▪ held every (non-C-level) position in software development ▪ grew up in Central California ▪ “data centric” software design ▪ family, photography, camping ▪ cycle commute year round
4. Building an Open Test Automation Framework - November 28, 2017 Inclusion ▪ User Acceptance Scenarios ▪ Authored and Owned by PO, QA/TA and Dev ▪ Can be used for: ▪ Behaviour Driven Development (BDD) ▪ Acceptance Criteria ▪ Creates a “Live” product specification for the PO ▪ Automated (or not) by QA/TA/Dev ▪ Run early, run often ▪ “Culture of Quality”
5. User Story Acceptance Criteria Acceptance Criteria Example Scenario Example Scenario Example Scenario Example Scenario Example Scenario
6. Building an Open Test Automation Framework - November 28, 2017 Process Patterns of Specification By Example (SBE) Gojko Adzic - Author of “Specification by Example” ▪ Deriving scope from goals ▪ Specifying collaboratively ▪ Illustrating specifications using examples ▪ Refining the specifications ▪ Automating validation without changing the specifications ▪ Validating the system frequently ▪ Evolving living documentation
7. Building an Open Test Automation Framework - November 28, 2017 Selection ▪ Audience - who will automate? ▪ TA, QA, Dev ▪ Open technologies ▪ Selenium ▪ general purpose language e.g. java, python ▪ test runner with a plug-in architecture ▪ Less Open ▪ JavaScript widget library integrated test framework ▪ using the same language as the product
8. Python Automation DSL Page Objects API Objects Data Objects Gherkin Feature Scenarios Product Owners TA and Product Developers QA Analysts/Developers Step Definitions
9. Building an Open Test Automation Framework - November 28, 2017 Documentation ▪ Specification By Example (SBE) ▪ use Gherkin based scenarios ▪ proper Feature files with a “user value” clause ▪ keep documentation (tests) up-to-date ▪ Scenarios act as self-guided training ▪ open source the tool (if possible)
10. Building an Open Test Automation Framework - November 28, 2017 Abstraction ▪ Audit-able “code” ▪ Layers ▪ reduce the resistance to “internal” extensions ▪ refactored later by the core team ▪ WebDriver < Splinter < Field < Page < Step < Scenario ▪ API < API Object < Step < Scenario ▪ Data Objects ▪ Data Types: UI and API ▪ Schema support
11. Building an Open Test Automation Framework - November 28, 2017 item_list.items.first.description.click() find_by_xpath('//*[@id="i_table6_a"]/tr[2]/td[5]/div').click() Page TableRows LinkField find_element(by=By.XPATH, value='//*[@id="i_table6_a"]/tr[2]/td[5]/div').click() for item in item_list.items: if item.quantity > 0: assert item.id in purchase_order.items
12. Building an Open Test Automation Framework - November 28, 2017 Readable Step Definitions ▪ Two or three lines of “code” per step ▪ Step Re-use: ▪ encourages correctness and clarity ▪ keeps code alive and continuously reviewed ▪ encourages best practices ▪ concentrates break fixes ▪ Automation velocity increases as step library grows
13. Building an Open Test Automation Framework - November 28, 2017 Assert This.Product == That.Product Page Object that_page.product Data Object this.product API Object that_api.items that_api.inventory Data Object that.product Page Object this_page.product API Object this_api.products this_api.inven this_api that_api splinter selenium api wrapper requests
14. Building an Open Test Automation Framework - November 28, 2017 Integration ▪ Docker, Docker, Docker ▪ Test Runners ▪ Selenium Browsers and Grid ▪ Mock Services ▪ Test Data ▪ Metrics/Results Dashboards ▪ CI and CD ▪ Docker based
15. Building an Open Test Automation Framework - November 28, 2017 Conclusion ▪ Links: ▪ www.quora.com/What-is-the-biggest-challenge-in-UAT-User- Acceptance-testing/answer/J-B-Rainsberger ▪ www.manning.com/books/specification-by-example ▪ docs.pytest.org/en/latest/contents.html#toc ▪ Questions?