Teams and Systems for Successful Test Automation JaSST’22 Tohoku Invited lecture Hiroki Iseri 1 This presentation material is translated from ”テスト自動化の成功を支える チームと仕組み(JaSST'22 Tohoku)" by @tsueeemura (Takuya Suemura).

About this presentation ● Tips and considerations for successful test automation ● This slide will be available on Discord ● Target: ○ Target test level/type: All ■ General test automation, including unit testing to system testing ○ Target domain: General ■ Findings in this presentation are not dependent on specific domains such as web/embed/native apps 2

About me ● Current roles ○ Tech lead of test team at Toyota Motors ■ Responsible for test architecture, strategy, and advocating test automation ○ Technical committee member of JSTQB (Japan Software Testing Qualifications Board) ○ A chairman of the Test Design Contest U30 class ● Main Works ○ “Standard guide to system test automation,” “Android app testing techniques,” “Building maintainability in unit tests,” “How to build test architecture to lead test design.” I have implemented and operated various test automation systems in diverse roles such as developer, tester, and consultant. Today, I would like to organize and share the knowledge I have gained from these experiences. 3

Outline 4 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Process Test automation system Appropriate objective setting Testability Test validity test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Learn and improve skills Whole Team Establish responsibility Expand values Habituatio n Test design Maintain sustainability Motivate Improve through observation and control

What is successful test automation? 5

What is successful test automation? 6 = Achieve objectives with reasonable compensation

Objectives of the test automation 7 Business level ・Address the customer’s needs, seeds and complaints faster ・Add more value to the product Team level ・Faster development and testing feedback cycles ・Boost developer experience ・Improve quality and features Tester level ・Improve test efficiency, effectivity, and accuracy e.g. implement test that could not be done by manual testing

Key benefits of achieving objectives through test automation ● Improve test effectiveness ○ Implement tests that are difficult to perform manually ○ Help achieve essential objectives ● Improve testing efficiency ○ Save time and resources spent on testing ○ Run more tests for the same cost ● Improve test accuracy and reliability ○ Reduce manual errors ○ Address errors and instability ● Improve test maintainability ○ Easier to QA, maintain, modify, and reuse tests ○ Help maintain the integrity and traceability of tests 8

Main considerations for test automation ● Required resources ○ People: members, skills, and organization ○ Materials: equipment, tools, licenses, environment ○ Time: education, implementation, operation/maintenance, management ● Required efforts ○ Automate tests ■ Build and maintain test automation environments ■ Build and maintain automated tests ■ Deal with additional work associated with test automation (e.g., increased complexity, integration with other tests) ○ Improve test coverage and prepare test basis ■ Build and maintain testability to automate tests ■ Prepare and maintain a test basis for automated tests ○ Build automation infrastructure ■ Build and maintain infrastructure that supports automated tests ○ Set up and maintain management that supports test automation 9

What is successful automation? 10 = Achieve objectives with reasonable compensation

Activities that support test automation 11

What makes test automation successful? 12 = Achieve objectives with reasonable compensation 1. Make appropriate objectives 2. Improve cost- effectiveness

Outline 13 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Process Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Gain and improve skill Whole Team Establish responsibility Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control Common reasons for failed test automation: setting inappropriate objectives and goals

Decide objectives that will lead the team to success ● Depending on the needs, seeds, constraints, and circumstances of the team: ○ Achievable objectives may change ○ The risk and cost-efficiency of test automation may change ● Set objectives that are beneficial/feasible for the team at the time 14 Target: Develop a smartphone application for a server-client system. Needs: Detect regression of server service. Constraints: The UI has not yet been finalized and is likely to change frequently. Technology owned by the team: The server execution environment is expected to be virtualized. Effective and likely to be successful: Continuous detection of server API regressions Likely to fail: Reduce the cost of quality assurance of end-to-end behavior dependent on UI

● By broadening the scope of test automation activities... - More essential/effective objectives can be selected - More means/people available to achieve objectives Everyone’s perspective, including the business Objective: Rapid response to business requirements, increased product value Means: Collaboration with users, continuous improvement throughout the product lifecycle Developer + tester’s perspective Objective: Faster development, improve developer experience, optimize whole QA process Means: Improve testability, collaboration between testers and developers Set objectives for a wider range of activities 15 Tester’s perspective Objective: Improve cost-efficiency of a specific test execution process Means: Test execution

● More difficult objectives can be achieved by accumulating successful cases and techniques while continuing test automation efforts. Thus, sustainability is most important! Achieve objectives step by step 16 High technical level Takes time to see effect Low technical level Effect can be seen earlier - Continuous unit testing - Automated tests supported by application framework - Use of standard test tools - Test automation includes hardware - Need to develop testing tools from scratch

What is successful test automation? 17 = Achieve objectives with reasonable compensation 1. Set appropriate objectives 2. Improve cost- effectiveness

Improve cost-efficiency 18 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Process Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration with automation infrastructure Plan and approach Gain and improve skills Whole Team Establish responsibility Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control

Improve testability for test automation 19 Quality characteristic Characteristics for test automation Implementation Operability Can execute automated processes smoothly Simple setup procedure Observability Easy to observe outputs (including indirect outputs and side-effects) from automated tests Easy to recognize errors, appropriate logging design, less side effects (e.g. stateless), debugging feature for observation Controllability Automated tests can easily control direct/indirect inputs Easy-to-use API, Stubbing, Debugger support Decomposability Easy to isolate automatable target. Easy to identify the issues with the automated test Appropriate seams (e.g. Dependency injection), appropriate association design Simplicity Specs, architecture and procedure of the test target are simple Reduce redundancy of design and code Stability Fewer bugs and instability that block automated tests Fewer bugs and spec changes, immutable interfaces Understandability Specs, architectures and procedures of test target are easy to understand Specs, code, and documentation that are easy to understand Common reason for test automation to fail: lack of testability ※Definitions of quality characteristics: Software engineering: a practitioner's approach, Roger S Pressman, McGraw-Hill Education / Asia

Improve test validity 20 ● Design test cases that are worth automating and maintain their value ● Keys to achieving valuable and valid test cases: Test requirements analysis Work with a wide range of stakeholders (users, teams, PjM, PdM, etc.) to broadly uncover needs, seeds, and constraints for testing and identify requirements Test level/test type/test architecture design Develop a test strategy and define the role of the overall test so that the automated test can produce value Test design Design test cases that efficiently fulfills responsibility through appropriate approach Maintain test design Maintain test design so validity is kept throughout the development lifecycle as test requirements/constraints change Fundamentals / Prerequisites

Improve the internal quality of automated tests 21 ● Improve the internal quality of the test system and code through general software engineering and utilizing know-how Quality characteristics Necessary characteristics Maintainability Make automated tests maintainable e.g.Observability (easy to observe), testability (easy to validate), analyzability and fixability (easy to identify and fix the root cause), readability (can be used as a Test as Documentation) Reliability Reduce false-positives and false-negatives / make it easy to ensure stability Continuous automated tests tend to be fragile and flaky Performance efficiency Early feedback Reduce required resources to increase concurrency and frequency of test execution Scalability Make automated test coverage scalable with less effort e.g. parameterized testing, data driven testing, VM, model-based testing Usability Increase usability and attractiveness for the team e.g. make it easy to give feedback and read results Common reason for test automation to fail: increase in maintenance cost due to poor internal quality

Devise a test automation plan/approach 22 ● Build appropriate steps for test automation through estimating, strategizing, and planning. Drive progress by directing and controlling. ○ Choose and secure human/material/financial/time resources ■ Gather resources to ensure automated testing can be in force promptly ○ Control test automation risks ■ Create early learning opportunities through prototyping and iterative development ○ Encourage and manage collaborations with stakeholders ○ Manage test automation activities Common reason for test automation to fail: poor setup

Utilize automated tests more frequently and widely 23 ● Automated testing assets can be used by various stakeholders for a variety of applications ○ Examples of integration tests that were automated successfully ■ for developers ● Drive programming with double-loop TDD ● Monitor regression during development with CI ● Use as acceptance criteria for PR acceptances and pipelines ■ for post-process testers ● Use as smoke testing or to complement manual testing ■ for post projects ● Use as the “Test as Documentation” for maintenance ● Cost-effectiveness can be increased by increasing the number of automated test users and utilizing automated tests more widely. ○ Prerequisite: make automated tests more widespread by making them more attractive ■ e.g. improve conviction, easy-to-understand feedback

Integrate into automation infrastructure 24 ● Integrate automated testing into automation infrastructure that supports development to increase maintainability and effectiveness. ○ Ensure integrity of development by integrating into the deployment pipeline ○ Establish responsibility for automated testing ○ Utilize infrastructure functions to build and maintain automated tests ■ Manage testware ● Manage configuration of the test system and the test code ■ Quality assurance for automated tests ● Dynamic/static analysis of test code ● Tests for the test system ■ Test execution ● Effective test execution management during the development cycle ○ Trigger by change ○ Concurrent execution ■ Monitor automated tests and help fix issues ● Measure coverage, monitor false positives and false negatives, monitor flaky tests

Maintain and improve by monitoring and controlling ● Monitor the quality and performance of automated tests to expedite problem detection and remediation and ensure effectiveness Maintain automated tests and work to maintain and improve the effectiveness of automated tests ● Problem detection ○ Problem detection before development ■ Prototyping/Trial/Iterative development ■ Review/Walkthrough ● How to detect bugs/How to assure specs ○ Problem detection while building tests ■ Red/green test-driven cycle ■ Fault injection ○ Monitoring during operation ■ Continuously monitor false-positives ■ Assess test coverage ■ Assess missed defects ○ Post-evaluation ■ Evaluation by Objective Achievement Indicators ● e.g. defect detection rate/test coverage/post-evaluation of cost-effectiveness/subjective evaluation by stakeholders 25

Activities that support test automation 26 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Process Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Gain and improve skills Whole Team Establish responsibilitie s Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control

Team culture that supports test automation 27

Team culture that supports test automation 28 ● Successful test automation can only be achieved with support from the team and having systems. ● e.g. Achieve good testability ○ Team culture ■ collaboration between developers and testers ■ basic development skills to achieve/maintain testability ● Practice patterns and idioms to achieve testability ● Refactor and monitor to maintain and improve testability ○ Processes ■ Consider testing during the design process ● Review by testers ■ Continuously get testability feedback from tester to developer ○ Development infrastructure ■ Infrastructure to ensure and maintain testability ● Static analysis of rules that support testability

Team culture that supports teams and systems 29 Team culture Vision/objectives, values, literacy, behavior, environment, people, relationships, education and evolution Teams and systems Management system, processes, pipeline, infrastructure/support service, test system, roles/structures Develop teams and systems Lead the culture

Team culture that supports test automation 30 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Processes Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Gain and improve skills Whole Team Establish responsibilitie s Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control

Orient the team culture to make test automation successful 31 1. Enhance the team's test automation capabilities ○ Improve efficiency and validity, build up the power to make a breakthrough 2. Accelerate test automation with the Whole Team approach ○ Automate with everyone, not just testers. Involve developers, domain experts, and managers ○ Consider testability every time ■ e.g. Write a testable specifications ■ e.g. Prepare a monitoring interface to test during development 3. Establish the team’s responsibilities for automated testing ○ Establish and fulfill responsibilities for automated testing within overall development activities 4. Improve the value of automated tests for the team ○ Recognize the needs and seeds of test automation with the team and achieve them ○ Maintain and improve the value of test automation 5. Maintain sustainability of the automated tests as a team ○ Keep maintainability, scalability, and evolvability ○ Continuously repay the technical debt 6. Motivate the team to automate testing ○ Show the value of automated test, and encourage independent and autonomous test automation

Teams and systems that support test automation 32 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Process Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Gain and improve skill Whole Team Establish responsibilitie s Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control

【Teams and systems that support test automation】 People and teams that have test automation skills ● Retain people who have test automation skills ○ Establish a skill model and an assessment program for test automation skills ○ Attract people who have the test automation skill ■ Essential skills: development, infrastructure, test and test automation ● Establish the team ○ Retain essential roles ■ Test architect / Test analyst ● Lead toward ideal test automation in terms of QA and testing as a whole ■ SET (Software Engineer in Test) / SETI (Software Engineer in Test and Infra) ● Focus on automation and infrastructure by utilizing development skills ○ Build a collaboration system (e.g. developers and testers) ● Build a system to keep improving the team’s skills ○ Build a learning cycle such as a prototyping 33 Common reason for test automation to fail: lack of skill and knowledge (especially development skills)

【Teams and systems that support test automation】 Build an automation infrastructure ● Build an automation infrastructure for development as an integration target of automated tests ○ Automate the entire deployment pipeline ■ And integrate automated tests into it ● Enable development infrastructure to support test automation efforts ○ Support processes: Build a pipeline of consistent automated processes ○ Support testability: Provide a means for static analysis and test coverage evaluation ○ Support building tests: Support automation through model-based testing, virtualization, etc. 34

【Teams and systems that support test automation】 Develop processes for test automation ● Build a good test automation process and embed it in the team. ○ Provide learning opportunities to improve test automation skills Prepare feedback cycle to control risks ■ Prototyping, iterative development ○ Build a framework ○ Turn a successful test automation approach into a process ■ Build a process to choose an appropriate tool at the right time ■ Build a process of a “Whole Team” approach ● Consider the test at each step (e.g. W Model) ○ Make criteria for required test automation activities ■ Set up quality criteria for test code 35

【Teams and systems that support test automation】 Establish good test automation practices ● By automating operations/checks, making rules, and using criteria, good test automation practices can be embedded in the team ● Good test automation practices: ○ FIRST principle ■ Fast, Independent, Repeatable, Self-Validating, Timely ○ 12 principles of xUTP ○ Principles of unit testing from “Software Engineering at Google” ■ Strive for unchanging tests. ■ Test via public APIs. ■ Test state, not interactions. ■ Make your tests complete and concise. ■ Test behaviors, not methods. ■ Structure tests to emphasize behaviors. ■ Name tests after the behavior being tested. ■ Don’t put logic in tests. ■ Write clear failure messages. ■ Follow DAMP over DRY when sharing code for tests. 36 (Software Engineering at Google, Winters et al. O'Reilly Media)

【Teams and systems that support test automation】 Overall QA and test structure ● Build and maintain an overall QA/testing design structure to lead the responsibility that automated testing should have in the development lifecycle ○ Who checks specs? Who finds bugs? 37 Responsibility structure of QA and test activities Structure of QA and test activities

【Teams and systems that support test automation】 Build the test automation system into the product ● Establish a test automation system and support its efforts ○ Implementation support for writing test scripts ■ Use ecosystem of programming languages like Python or JS ■ Use BDD syntax like Gherkin ■ Support structural script with data-driven and parameterized tests ■ Specify the base structure of test scripts like xUnit or xSpec ○ Provide utilities that support test automation ■ Monitor, report, test doubles, setup, and teardown ○ Specify test automation architecture and framework ○ Provide an efficient way to manage testware and test data 38

Teams and systems that support test automation ● We can facilitate activities that make test automation successful by establishing teams and systems. ● We can increase the likelihood of test automation to succeed by continuing to build a foundation of a team and systems. 39

Conclusion 40 Team culture that supports test automation Teams and systems that support test automation Activities that support test automation Successful test automation Pt1 Pt2 Pt3 Team Automation Infrastructu re Processes Test automation system Appropriate objective setting Testability Validity of test Internal test quality Utilize more Integration into automation infrastructure Plan and approach Gain and improve skill Whole Team Establish responsibilitie s Expand value Habituatio n Test design Maintain sustainability Motivate Improve through observation and control