Teams and Systems for Successful Test Automation

Teams and Systems for
Successful Test Automation
JaSST’22 Tohoku Invited lecture
Hiroki Iseri
1
This presentation material is translated from ”テスト自動化の成功を支える
チームと仕組み(JaSST'22 Tohoku)" by @tsueeemura (Takuya Suemura).

View Slide

About this presentation
● Tips and considerations for successful test automation
● This slide will be available on Discord
● Target:
○ Target test level/type: All
■ General test automation, including unit testing to system
testing
○ Target domain: General
■ Findings in this presentation are not dependent on specific
domains such as web/embed/native apps
2

View Slide

About me
● Current roles
○ Tech lead of test team at Toyota Motors
■ Responsible for test architecture, strategy, and advocating test automation
○ Technical committee member of JSTQB (Japan Software Testing Qualifications Board)
○ A chairman of the Test Design Contest U30 class
● Main Works
○ “Standard guide to system test automation,” “Android app testing techniques,” “Building
maintainability in unit tests,” “How to build test architecture to lead test design.”
I have implemented and operated various test automation systems in diverse roles such
as developer, tester, and consultant.
Today, I would like to organize and share the knowledge I have gained from these
experiences.
3

View Slide

Outline
4
Team culture that supports test automation
Teams and systems that support test automation
Activities that support test automation
Successful test automation
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
Process Test automation
system
Appropriate
objective
setting
Testability
Test
validity
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Learn and
improve skills
Whole
Team
Establish
responsibility
Expand
values
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through
observation and control

View Slide

What is successful test automation?
5

View Slide

6
= Achieve objectives
with reasonable compensation

View Slide

Objectives of the test automation
7
Business level
・Address the customer’s needs, seeds and complaints faster
・Add more value to the product
Team level
・Faster development and testing feedback cycles
・Boost developer experience
・Improve quality and features
Tester level
・Improve test efficiency, effectivity, and accuracy
e.g. implement test that could not be done by manual testing

View Slide

Key benefits of achieving objectives through test
automation
● Improve test effectiveness
○ Implement tests that are difficult to perform manually
○ Help achieve essential objectives
● Improve testing efficiency
○ Save time and resources spent on testing
○ Run more tests for the same cost
● Improve test accuracy and reliability
○ Reduce manual errors
○ Address errors and instability
● Improve test maintainability
○ Easier to QA, maintain, modify, and reuse tests
○ Help maintain the integrity and traceability of tests
8

View Slide

Main considerations for test automation
● Required resources
○ People: members, skills, and organization
○ Materials: equipment, tools, licenses, environment
○ Time: education, implementation, operation/maintenance, management
● Required efforts
○ Automate tests
■ Build and maintain test automation environments
■ Build and maintain automated tests
■ Deal with additional work associated with test automation (e.g., increased
complexity, integration with other tests)
○ Improve test coverage and prepare test basis
■ Build and maintain testability to automate tests
■ Prepare and maintain a test basis for automated tests
○ Build automation infrastructure
■ Build and maintain infrastructure that supports automated tests
○ Set up and maintain management that supports test automation
9

View Slide

What is successful automation?
10

View Slide

11

View Slide

What makes test automation successful?
12
1. Make appropriate
objectives
2. Improve cost-
effectiveness

View Slide

Outline
13
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Gain and
improve skill
Whole
Team
Establish
responsibility
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through
Common reasons for failed test automation:
setting inappropriate objectives and goals

View Slide

Decide objectives that will lead the team to success
● Depending on the needs, seeds, constraints, and circumstances of the
team:
○ Achievable objectives may change
○ The risk and cost-efficiency of test automation may change
● Set objectives that are beneficial/feasible for the team at the time
14
Target: Develop a smartphone
application for a server-client system.
Needs: Detect regression of server
service.
Constraints: The UI has not yet been
finalized and is likely to change
frequently.
Technology owned by the team: The
server execution environment is
expected to be virtualized.
Effective and likely to be
successful:
Continuous detection of server API
regressions
Likely to fail:
Reduce the cost of quality assurance
of end-to-end behavior dependent on
UI

View Slide

● By broadening the scope of test automation activities...
- More essential/effective objectives can be selected
- More means/people available to achieve objectives
Everyone’s perspective, including the business
Objective: Rapid response to business requirements, increased product value
Means: Collaboration with users, continuous improvement throughout the product lifecycle
Developer + tester’s perspective
Objective: Faster development, improve developer experience, optimize whole QA process
Means: Improve testability, collaboration between testers and developers
Set objectives for a wider range of activities
15
Tester’s perspective
Objective: Improve cost-efficiency of a specific test execution process
Means: Test execution

View Slide

● More difficult objectives can be achieved by accumulating successful cases and techniques
while continuing test automation efforts. Thus, sustainability is most important!
Achieve objectives step by step
16
High technical
level
Takes time to see effect
Low technical
level
Effect can be seen earlier
- Continuous unit testing
- Automated tests supported
by application framework
- Use of standard test tools
- Test automation includes
hardware
- Need to develop testing
tools from scratch

View Slide

17
1. Set appropriate
objectives
2. Improve cost-
effectiveness

View Slide

Improve cost-efficiency
18
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration with
automation
infrastructure
Plan and
approach
Gain and
improve skills
Whole
Team
Establish
responsibility
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through

View Slide

Improve testability for test automation
19
Quality
characteristic
Characteristics for test automation Implementation
Operability Can execute automated processes smoothly Simple setup procedure
Observability Easy to observe outputs (including indirect outputs
and side-effects) from automated tests
Easy to recognize errors, appropriate logging design, less side
effects (e.g. stateless), debugging feature for observation
Controllability Automated tests can easily control direct/indirect
inputs
Easy-to-use API, Stubbing, Debugger support
Decomposability Easy to isolate automatable target. Easy to identify
the issues with the automated test
Appropriate seams (e.g. Dependency injection),
appropriate association design
Simplicity Specs, architecture and procedure of the test target
are simple
Reduce redundancy of design and code
Stability Fewer bugs and instability that block automated tests Fewer bugs and spec changes, immutable interfaces
Understandability Specs, architectures and procedures of test target are
easy to understand
Specs, code, and documentation that are easy to
understand
Common reason for test automation to fail: lack of testability
※Definitions of quality characteristics: Software engineering: a practitioner's approach, Roger S Pressman, McGraw-Hill Education / Asia

View Slide

Improve test validity
20
● Design test cases that are worth automating and maintain their value
● Keys to achieving valuable and valid test cases:
Test requirements analysis
Work with a wide range of stakeholders (users, teams, PjM, PdM, etc.) to broadly
uncover needs, seeds, and constraints for testing and identify requirements
Test level/test type/test architecture design
Develop a test strategy and define the role of the overall test
so that the automated test can produce value
Test design
Design test cases that efficiently fulfills responsibility
through appropriate approach
Maintain test design
Maintain test design so validity is kept throughout the
development lifecycle as test requirements/constraints change
Fundamentals / Prerequisites

View Slide

Improve the internal quality of automated tests
21
● Improve the internal quality of the test system and code through general
software engineering and utilizing know-how
Quality
characteristics
Necessary characteristics
Maintainability Make automated tests maintainable
e.g.Observability (easy to observe), testability (easy to validate), analyzability and fixability (easy
to identify and fix the root cause), readability (can be used as a Test as Documentation)
Reliability Reduce false-positives and false-negatives / make it easy to ensure stability
Continuous automated tests tend to be fragile and flaky
Performance
efficiency
Early feedback
Reduce required resources to increase concurrency and frequency of test execution
Scalability Make automated test coverage scalable with less effort
e.g. parameterized testing, data driven testing, VM, model-based testing
Usability Increase usability and attractiveness for the team
e.g. make it easy to give feedback and read results
Common reason for test automation to fail:
increase in maintenance cost due to poor internal quality

View Slide

Devise a test automation plan/approach
22
● Build appropriate steps for test automation through estimating,
strategizing, and planning. Drive progress by directing and controlling.
○ Choose and secure human/material/financial/time resources
■ Gather resources to ensure automated testing can be in force
promptly
○ Control test automation risks
■ Create early learning opportunities through prototyping and
iterative development
○ Encourage and manage collaborations with stakeholders
○ Manage test automation activities
Common reason for test automation to fail: poor setup

View Slide

Utilize automated tests more frequently and widely
23
● Automated testing assets can be used by various stakeholders for a variety
of applications
○ Examples of integration tests that were automated successfully
■ for developers
● Drive programming with double-loop TDD
● Monitor regression during development with CI
● Use as acceptance criteria for PR acceptances and pipelines
■ for post-process testers
● Use as smoke testing or to complement manual testing
■ for post projects
● Use as the “Test as Documentation” for maintenance
● Cost-effectiveness can be increased by increasing the number of
automated test users and utilizing automated tests more widely.
○ Prerequisite: make automated tests more widespread by making them
more attractive
■ e.g. improve conviction, easy-to-understand feedback

View Slide

Integrate into automation infrastructure
24
● Integrate automated testing into automation infrastructure that supports
development to increase maintainability and effectiveness.
○ Ensure integrity of development by integrating into the deployment pipeline
○ Establish responsibility for automated testing
○ Utilize infrastructure functions to build and maintain automated tests
■ Manage testware
● Manage configuration of the test system and the test code
■ Quality assurance for automated tests
● Dynamic/static analysis of test code
● Tests for the test system
■ Test execution
● Effective test execution management during the development cycle
○ Trigger by change
○ Concurrent execution
■ Monitor automated tests and help fix issues
● Measure coverage, monitor false positives and false negatives, monitor flaky
tests

View Slide

Maintain and improve by monitoring and controlling
● Monitor the quality and performance of automated tests to expedite problem detection and
remediation and ensure effectiveness
Maintain automated tests and work to maintain and improve the effectiveness of automated
tests
● Problem detection
○ Problem detection before development
■ Prototyping/Trial/Iterative development
■ Review/Walkthrough
● How to detect bugs/How to assure specs
○ Problem detection while building tests
■ Red/green test-driven cycle
■ Fault injection
○ Monitoring during operation
■ Continuously monitor false-positives
■ Assess test coverage
■ Assess missed defects
○ Post-evaluation
■ Evaluation by Objective Achievement Indicators
● e.g. defect detection rate/test coverage/post-evaluation of cost-effectiveness/subjective
evaluation by stakeholders
25

View Slide

26
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Gain and
improve skills
Whole
Team
Establish
responsibilitie
s
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through

View Slide

27

View Slide

28
● Successful test automation can only be achieved with support from
the team and having systems.
● e.g. Achieve good testability
○ Team culture
■ collaboration between developers and testers
■ basic development skills to achieve/maintain testability
● Practice patterns and idioms to achieve testability
● Refactor and monitor to maintain and improve testability
○ Processes
■ Consider testing during the design process
● Review by testers
■ Continuously get testability feedback from tester to developer
○ Development infrastructure
■ Infrastructure to ensure and maintain testability
● Static analysis of rules that support testability

View Slide

Team culture that supports teams and systems
29
Team culture
Vision/objectives, values, literacy, behavior, environment, people,
relationships, education and evolution
Teams and systems
Management system, processes, pipeline,
infrastructure/support service, test system, roles/structures
Develop teams and systems Lead the culture

View Slide

30
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
Processes Test automation
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Gain and
improve skills
Whole
Team
Establish
responsibilitie
s
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through

View Slide

Orient the team culture to make test automation
successful
31
1. Enhance the team's test automation capabilities
○ Improve efficiency and validity, build up the power to make a breakthrough
2. Accelerate test automation with the Whole Team approach
○ Automate with everyone, not just testers. Involve developers, domain experts, and managers
○ Consider testability every time
■ e.g. Write a testable specifications
■ e.g. Prepare a monitoring interface to test during development
3. Establish the team’s responsibilities for automated testing
○ Establish and fulfill responsibilities for automated testing within overall development activities
4. Improve the value of automated tests for the team
○ Recognize the needs and seeds of test automation with the team and achieve them
○ Maintain and improve the value of test automation
5. Maintain sustainability of the automated tests as a team
○ Keep maintainability, scalability, and evolvability
○ Continuously repay the technical debt
6. Motivate the team to automate testing
○ Show the value of automated test, and encourage independent and autonomous test automation

View Slide

32
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Gain and
improve skill
Whole
Team
Establish
responsibilitie
s
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through

View Slide

【Teams and systems that support test automation】
People and teams that have test automation skills
● Retain people who have test automation skills
○ Establish a skill model and an assessment program for test automation
skills
○ Attract people who have the test automation skill
■ Essential skills: development, infrastructure, test and test
automation
● Establish the team
○ Retain essential roles
■ Test architect / Test analyst
● Lead toward ideal test automation in terms of QA and testing as a whole
■ SET (Software Engineer in Test) / SETI (Software Engineer in Test
and Infra)
● Focus on automation and infrastructure by utilizing development skills
○ Build a collaboration system (e.g. developers and testers)
● Build a system to keep improving the team’s skills
○ Build a learning cycle such as a prototyping
33
Common reason for test automation to fail:
lack of skill and knowledge (especially development skills)

View Slide

Build an automation infrastructure
● Build an automation infrastructure for development as an
integration target of automated tests
○ Automate the entire deployment pipeline
■ And integrate automated tests into it
● Enable development infrastructure to support test automation
efforts
○ Support processes: Build a pipeline of consistent automated
processes
○ Support testability: Provide a means for static analysis and test
coverage evaluation
○ Support building tests: Support automation through model-based
testing, virtualization, etc.
34

View Slide

Develop processes for test automation
● Build a good test automation process and embed it in the team.
○ Provide learning opportunities to improve test automation
skills
Prepare feedback cycle to control risks
■ Prototyping, iterative development
○ Build a framework
○ Turn a successful test automation approach into a process
■ Build a process to choose an appropriate tool at the right time
■ Build a process of a “Whole Team” approach
● Consider the test at each step (e.g. W Model)
○ Make criteria for required test automation activities
■ Set up quality criteria for test code
35

View Slide

Establish good test automation practices
● By automating operations/checks, making rules, and using criteria, good test
automation practices can be embedded in the team
● Good test automation practices:
○ FIRST principle
■ Fast, Independent, Repeatable, Self-Validating, Timely
○ 12 principles of xUTP
○ Principles of unit testing from “Software Engineering at Google”
■ Strive for unchanging tests.
■ Test via public APIs.
■ Test state, not interactions.
■ Make your tests complete and concise.
■ Test behaviors, not methods.
■ Structure tests to emphasize behaviors.
■ Name tests after the behavior being tested.
■ Don’t put logic in tests.
■ Write clear failure messages.
■ Follow DAMP over DRY when sharing code for tests.
36
(Software Engineering at Google, Winters et al. O'Reilly Media)

View Slide

Overall QA and test structure
● Build and maintain an overall QA/testing design structure to lead the
responsibility that automated testing should have in the development
lifecycle
○ Who checks specs? Who finds bugs?
37
Responsibility structure of QA and
test activities
Structure of QA and test activities

View Slide

Build the test automation system into the product
● Establish a test automation system and support its efforts
○ Implementation support for writing test scripts
■ Use ecosystem of programming languages like Python or JS
■ Use BDD syntax like Gherkin
■ Support structural script with data-driven and parameterized
tests
■ Specify the base structure of test scripts like xUnit or xSpec
○ Provide utilities that support test automation
■ Monitor, report, test doubles, setup, and teardown
○ Specify test automation architecture and framework
○ Provide an efficient way to manage testware and test data
38

View Slide

●
We can facilitate activities that make test automation successful by
establishing teams and systems.
●
We can increase the likelihood of test automation to succeed by
continuing to build a foundation of a team and systems.
39

View Slide

Conclusion
40
Pt1
Pt2
Pt3
Team
Automation
Infrastructu
re
Processes Test automation
system
Appropriate
objective
setting
Testability
Validity of
test
Internal test
quality
Utilize
more
Integration into
automation
infrastructure
Plan and
approach
Gain and
improve skill
Whole
Team
Establish
responsibilitie
s
Expand
value
Habituatio
n
Test
design
Maintain
sustainability
Motivate
Improve through

View Slide

Teams and Systems for Successful Test Automation

Teams and Systems for Successful Test Automation

Hiroki Iseri

More Decks by Hiroki Iseri

Other Decks in Technology

Featured

Transcript