© 2024 CGI Inc. 1 Experiences of AI of Today in Testing Maaret Pyhäjärvi

© 2024 CGI Inc. 2 Acting Humanly for Testing 01 Natural Language processing to communicate successfully in human language • Summarizing details to insights • Generating charters • Creating data, instructions and oracles • Understanding risk coverage 02 Knowledge representation to store what it knows or hears • Remembering features and recognizing feature relationships • Avoiding reporting accepted problems without change in knowledge • Remembering what was done • Knowing what to look at 03 Automated reasoning to answer questions and draw new conclusions • Deciding what conversations to start • Deciding when we can automatically revert • Reporting with repro scripts • Recognizing responsibility of fix 04 Machine learning to adapt to new circumstances and to extrapolate and detect patterns • Bug taxonomies • Priorities • Cross industry reuse of standard tests 05 Computer vision, speech recognition to perceive the world • Multilingual projects • Sources of data • Visual testing aids 06 Robotics to manipulate objects and move around • Robotic process automation as basis of testing • Operating interfaces abstracting away technology of target Any software is marketed as AI since it is doing something humans could do.

© 2024 CGI Inc. 3 An AI World Map 3 Text Text /code Sound Image Video Copyrights Training Data Machine Learning Algorithm e.g., LLM input Filter output Filter Fore- ground Back- ground Retrieval pipeline Text Text /code Sound Image Video Copyrights Parameters e.g., temperature Tasks Prompt engineering is the discipline of formulating future background requests in the foreground Copyrights From 2048 tokens (1500 words) to 1M

© 2024 CGI Inc. 4 Mapping the acronyms 4 ML Machine Learning LLM Large Language Model RAG Retrieval Augmented Generation CoT Chain of Thought

© 2024 CGI Inc. 5 The most significant challenges for use of AI in testing are data and availability of organizations ready to pay for it. - Heini Ahven 2023 5

© 2024 CGI Inc. 6 Prepare to Shift: Left and Down 6 Shift Left Shift down Quality engineering over quality assurance Whole team approach Clarifying core examples Single-commit delivery Test-driven development Exploratory unit testing “77% of production bugs could have been found with a unit test” We’re still half-way into the automation transformation that AI supports.

© 2024 CGI Inc. 7 AI in Testing Experiences 2024 Test code generation in IDE 1 AI tools in review stage and pipelines in general ’agentification’ 2 Creating a RAG- assistant for testing 3 Applying today’s AI in testing

© 2024 CGI Inc. 8 What changed? Trust relationships Corporation to corporation trust set up by contracts. Context size Input to general purpose model is of relevant size. Packaging as services Abstractions hiding setup that is not AI on side of AI creating hybrid products. Task-specific application External imagination on level of tasks with human in loop.

© 2024 CGI Inc. 9 9 Test code generation in IDE 1 AI tools in review stage and pipelines in general ’agentification’ 2

© 2024 CGI Inc. 10 10

© 2024 CGI Inc. 11 AI of Today in Developer Testing 11 Idea of change *Intent* Pull Request Commit Commit Build and Deploy write code write tests document review review codiumai pr-agent pro code review by gpt github action write tests github copilot codiumai codiumate earlyai jetbrains ai assistant run tests run tools run tools IDE nodes Agents running in

© 2024 CGI Inc. 12 “I love the extra autocompletion that I get with it, it feels like I never have to write any kind of boilerplate code anymore, and I also find it very useful to just ask stuff directly in the IDE. I used to google stuff all the time, and ended up on Stackoverflow a lot, but nowadays I rarely have to do that.” 12

© 2024 CGI Inc. 13 GitHub Copilot 13 Code Completion in IDE Chat, inline and in IDE Participants: @workspace Variables: #file #selection #editor Commands: /fix /tests /explain /doc In GitHub Pull request summaries Knowledge bases Spec-Plan-Implement flows Bing for recent ADO for pipelines and environments Lines and blocks *Enterprise Only

© 2024 CGI Inc. 14 14 Creating a RAG- assistant for testing 3

© 2024 CGI Inc. 15 15

© 2024 CGI Inc. 16 Contemporary Exploratory Testing Assistant ‘AI-Maaret’ 16 Pretrained LLM Body of authored text from one author Responses closer to your bias Cloud models such as ChatGPT Self-hosted models from huggingface

© 2024 CGI Inc. 17 Simplest form of larger context 17

© 2024 CGI Inc. 18 GenAI Pair Testing External imagination Confidentiality Ethical compensations Intent and productivity 18 Photo by Rajvir Kaur on Unsplash

© 2024 CGI Inc. 19 GenAI Pair Testing Search boundaries: argue for different stances on assumptions Recognize insufficiency and fix it – creating average text is not *your* goal Freedom to criticize as the pair takes no offense Dare to ask things you’d not dare to ask from a colleague Co-piloting allows for repair 19 Photo by Rajvir Kaur on Unsplash

© 2024 CGI Inc. 20 Predictive text generation useful for recall not reasoning 20

© 2024 CGI Inc. 21 What else? Today – Soon - Future 21

© 2024 CGI Inc. 22 22

© 2024 CGI Inc. 23 Contemporary Exploratory Testing Programmatic tests as output. Feedback with agency. 23

© 2024 CGI Inc. 24 Picked up folklore that may be useful in framing 24 Automating leaves behind the most difficult 10%. Debugging your own creation is twice as hard as creating it in the first place. 50% of jobs change, 5% of jobs replaced by automation Software industry doubles in 5 years. 1% (4 min) better daily would make 37.8x you in 365 days. There’s plenty of things we want but can’t get to because creating what we have now hogs all bandwidth.

© 2024 CGI Inc. 25 Testing is too important to be left just for testers. It’s too important to be left without testers. 25

© 2024 CGI Inc. 26 Improve first –thinking. 26 https://www.tivi.fi/uutiset/maaret-pyhajarvi-is- revolutionizing-software-testing-at-cgi-artificial- intelligence-is-like-an-external- imagination/0765bc05-c989-4f9e-b882- 33604c165d8e

© 2024 CGI Inc. 27 Quality Engineering and Testing at CGI Finland 01 AI transformation Applying ML, GenAI, RAG, CoT in human-centric contemporary exploratory testing 02 Automation transformation Decomposing to actionable feedback and capabilities in pipelines 03 Testing in the world The Creative Commons baseline and integrations with community 04 Testing in CGI Real work with our customers

© 2024 CGI Inc. 28 Insights you can act on Founded in 1976, CGI is among the largest IT and business consulting services firms in the world. We are insights-driven and outcomes-based to help accelerate returns on your investments. Across hundreds of locations worldwide, we provide comprehensive, scalable and sustainable IT and business consulting services that are informed globally and delivered locally. cgi.com