© 2024 CGI Inc. 1 Test Automation Friend or Foe? Maaret Pyhäjärvi November 2024

© 2024 CGI Inc. 2 Tropes …commonly recurring or overused literary and rhetorical devices, motifs or clichés in creative works 2

© 2024 CGI Inc. 3 Collecting evidence: Friend or Foe? Opposing evidence • Seeking opposing evidence builds a stronger case • Learning through experimentation • Scientific method Supporting evidence • Feeding our beliefs • Sense of learning

© 2024 CGI Inc. 4 Conversation a year ago 4 …with Ben Dowen, Beth Marshall, Brijesh Deb, Maaret Pyhäjärvi What is test automation? How to decide when to automate a particular test and when to do manual instead? What if I figure out that test case becomes more complicated to automate than I thought originally? What is the biggest challenge to test automation and how to overcome it? Does every tester need to be a coder in the future? How can you balance need for speed and efficiency, and maintaining high level of test coverage? If we focus on test automation, are we risking that we don’t focus on other things? https://visible-quality.blogspot.com/2024/07/test-automation-friend-or-foe.html 50% of jobs changed, only 5% replaced by automation

© 2024 CGI Inc. 5 Team up with an old enemy Friend Foe “High-maintenance friend” “Mutually beneficial boundaries or you will fall out” ”Requires pampering. Fickle to foe.” “Treat with respect regardless of dislike. Collaborate with enemy when stakes are high.”

© 2024 CGI Inc. 6 Time used on warning about test automation is time away from succeeding with it. 6 Photo by Laura Ockel on Unsplash

© 2024 CGI Inc. 7 My Journey to Test Automation Falling in testing …like falling in love. –Ash Coleman Hynie Versus checking Automation belittlement. Not worth my schedules. Except for warnings. Read to collaborate All layers of pyramid. Pair and ensemble testing. Design for feedback. Opportunity cost Time to warn is time away from supporting success. Polyglot programmer Learning by pairing and ensembling. Team success is only sustained success. .

© 2024 CGI Inc. 8 Contemporary Exploratory Testing Programmatic tests as output. Feedback with agency. 8

© 2024 CGI Inc. 9 Test automation gives us more than regression Specification Examples shared before implementation act as specification. Feedback We know when it works as we’re implementing the progress. Regression We safeguard our past intent over lifetime of system ownership. Granularity We know when and what changed without extensive analysis. Documenting Capture examples whenever we discuss them. Extending reach Time. Environments. Data. Alerting to attend Changes, both intended and unintended. Guiding to detail Drives attention to details humans do repair for.

© 2024 CGI Inc. 10 The Three Contemporary Testing Profiles (Contemporary) Exploratory Tester Start with information, document with automation. Finish with testing done. (Contemporary) Test Automation Engineer Start with automation, design for information. Finish with testing done. Programmer Start with intent, document with automation. Finish with testing done. Great collaboration skills compensate for lack in other skills. See the bigger picture for customer benefit.

© 2024 CGI Inc. 11 For test automation, you choose a language Tradeoffs • Visual / Robot Framework / Gherkin / TypeScript / Python / Java / C# / … • Enable ownership over time when people change positions • Granularity: feedback to where things got broken • Reuse and integrations Usual choice: automation in the language of the implementation.

© 2024 CGI Inc. 12 It’s not just the language / tools 03 02 01 04 Tools Language + Libraries. CI/CD pipeline. Test warehouse of results. Environments Conceptual home for where tests run, with needs different for automation. Test Data Business systems, business data. Varying degrees of logic impact and complexity. Long flow knowledge Multiple business expert to test real end to end flow. Analyze and enhance tests It’s a system you react to. Fix and adjust. Enhance with functionality.

© 2024 CGI Inc. 13 13

© 2024 CGI Inc. 14 Prepare to Shift: Left and Down 14 Shift Left Shift down Quality engineering over quality assurance Whole team approach Clarifying core examples Single-commit delivery Test-driven development Exploratory unit testing “77% of production bugs could have been found with a unit test” We’re still half-way into the automation transformation that AI supports.

© 2024 CGI Inc. 15 Future is already here, it’s just not evenly divided Automation Lifecycle investment. More than regression testing. Opportunity cost Lean waste awareness. Shift continuous. Social Developers test. Learning is essential. Quality engineering. Next gen artifacts (AI) Programming and test programming. Insights to recall. From teams of testers to testing by teams From quality assurance through quality assistance to quality engineering From significant time to test after collection of changes to significant time to document with programmatic tests that test continuously From testers to team members with testing emphasis From testers as holders of domain knowledge to everyone as holders of domain knowledge From doing testing tasks to choosing actions for purpose of quality From transforming manual tests into automated tests to decomposing testing for scales of architectures From testing end to end integrated systems to decomposing feedback mechanisms differently From change control to releases as routine.

© 2024 CGI Inc. 16 Tropes Recognize the genre Know it all leads to love Speed up your place on the story arc The stories we tell make up the results we deliver. 16

© 2024 CGI Inc. 17 Insights you can act on Founded in 1976, CGI is among the largest IT and business consulting services firms in the world. We are insights-driven and outcomes-based to help accelerate returns on your investments. Across hundreds of locations worldwide, we provide comprehensive, scalable and sustainable IT and business consulting services that are informed globally and delivered locally. cgi.com