Better together: Humans, AI and the next quality benchmark

Talk at PNSQC.

Transcript

1. Better together Humans, AI and the next quality benchmark Maaret Pyhäjärvi October 2026 © 2026 CGI Inc. 1
2. “Introduce yourself for 30 people” “Tell a joke” © 2026 CGI Inc. 2
3. “Make people get testing” © 2026 CGI Inc. 3
4. How Would You Test This? Raster Reveal by James Lyndsay https://www.workroomproductions.com/raster-reveal/ © 2026 CGI Inc. 4
5. ABOUT TESTING Search for information that matters Test design Test execution Test cases Exploratory Manual Automated Managing Doing Avoid co nfidentl y incorr ect judg ement © 2026 CGI Inc. 5
6. Find (some of) what other’s have missed! How Would You Test This? Raster Reveal by James Lyndsay https://www.workroomproductions.com/raster-reveal/ Quality / Results Gap Balancing detailed reveal vs. jumping to conclusions Invest RIGHT – too much, too little, the wrong kind No one left alone with responsibilities too big for individuals © 2026 CGI Inc. 6
7. AI ENABLES TASK EXPANSION People can now do things by asking for it. © 2026 CGI Inc. 7
8. ASKING FOR IT Use playwright-cli to operate website https://www.workroomproductions.com/raster-reveal/ and create a script that systematically reveals the image that gets more accurate by mouse movement. Take regular screenshots and figure out for each if you know what is in the picture. Keep the browser open so that I can watch it unfold. © 2026 CGI Inc. 8
9. ABOUT TESTING Stopping criteria when automation is more accessible Test design Test cases 0:00 Test execution 0:46 animal-like shape ManualAis dark emerging on a light Confirmed: it's a quadtree that reveals finer detail in blocks my cursor actually crosses. Let me sweep the remaining bands top-tobottom. background, with Managing orange/gold accents. 3:12 Exploratory 13:26 That's unmistakably a black rubber duck with an orange beak and a gold Batman bat-symbol on its side/wing — a "Batman rubber duck" bath toy. Final identification Automated stands: a black rubber duck with an orange Doing beak and a gold Batman logo on its side. Avoid co nfidentl y incorr ect judg ement © 2026 CGI Inc. 9
10. 2018 RESEARCH A walker tool with 2 simple oracles beats commodity testers at results. © 2026 CGI Inc. 10
11. 20.11.2023 In this particular case, chatGPT did better than colleagues. People are so afraid to say what they think and make decisions. © 2026 CGI Inc. 11
12. Practice at: https://exploratory-testing-academy.github.io/capture-the-bugs/ Scale d assess ment = self -asse ssment © 2026 CGI Inc. 12
13. LATEST GROUP Asking AI for it: 7% People in pairs: 35% average 79% maximum © 2026 CGI Inc. 13
14. ESSENTIAL INSIG HTS ESSENTIAL ACTIO NS RESULTS AUTOMATIO N MISTAKES THAT NEED EXPLAINING © 2026 CGI Inc. https://github.com/QE-at-CGI-FI/todoapp-solution 14
15. NEED TO DO BETTER Benchmark result: results gap Exploratory tester with AI: 73 Exploratory tester: 45 AI with human: 40 Traditional testers (n=57): 13.5 Playwright agents: 12 Future talent ’25 (n=9): 11.5 AI without human: 4 100% 62% 55 % 18% 16% 16% 5% Invis ible ink: proble ms are there ev en if no one re ported them. © 2026 CGI Inc. 15
16. TEACHING DISCOVERY Finding empirical information about software is a teachable skill. © 2026 CGI Inc. 16
17. https://qe-at-cgi-fi.github.io/atm/ © 2026 CGI Inc. 17
18. NEED TO DO BETTER Bringing in Properties 3/4 reasons for failure previously unknown © 2026 CGI Inc. 18
19. Most valuable form of coverage is results: empirical, actionable, and current information. © 2026 CGI Inc. 19
20. HIERARCHY OF IMPACT TO QUALITY Three layers of subsumed power User experiences Tester searches the experience space · reports · decides Developer fixes · learns © 2026 CGI Inc. 20
21. INVISIBLE INK REVEAL We are *users* to this conference RESULTS ABSENCE OF RESULTS © 2026 CGI Inc. 21
22. PROPERTIES AS ONE WAY TO TEST THIS PNSQC Conference Website, three slice testing Effort: 2 min Duration: 30 min © 2026 CGI Inc. 22
23. TIME AN D EF FORT What do you do while AI runs a job for you? © 2026 CGI Inc. 23
24. SPLIT TING WORK WHILE MAINTAINING AGENCY The ways we split testing don’t make sense. MANAGING | DO ING MANUAL | AUTOMATED TEST CASES | EXPLORING Te s t i n g i s l e a r n i n g . L e a r n i n g i s l a y e r e d . © 2026 CGI Inc. 24
25. CO LLABO RATIVE ACTIVIT Y Extend from reporting issues into debugging, repairs, and shared team practices, including exploration at unit level. © 2026 CGI Inc. 25
26. INTENTIONAL ABOUT HUMANS Human-in-the-loop, not Human-as-the-loop © 2026 CGI Inc. 26
27. Solo – Pair - Ensemble © 2026 CGI Inc. Getting best out of everyone into the work we are doing 27
28. TESTING FOLDED INTO ENGINEERING Metaphorically speaking – testing not testers 15 $/kg Commodity testing © 2026 CGI Inc. 450 $/kg Contemporary exploratory testing 28
29. CONTEMPORARY EXPLORATORY TESTING IN AGE OF AI Things you should ask for 1 Identify properties with skill, create programmatic tests and report visually. 2 What insights this week’s conversations add to what we try to remember from past agreements? 3 Could we talk over a cup of coffee? IDENTIF YING G APS Fully automated, deeply human © 2026 CGI Inc. 29
30. AGE OF AI Exploring intent, value, context, alignment and experience © 2026 CGI Inc. 30
31. BREAKING ILLUSIONS WITH TESTING Skills that move you away from ’tester’ and ‘developer’ © 2026 CGI Inc. 31
32. Gym membership Access to tools isn’t the tools doing the testing for you. © 2026 CGI Inc. 32
33. AGE OF AI Actionable feedback that challenges well-maintained illusions has never been more important. © 2026 CGI Inc. 33
34. Insights you can act on Founded in 1976, CGI is among the largest IT and business consulting services firms in the world. We are insights-driven and outcomes-focused to help accelerate returns on your investments. Across hundreds of locations worldwide, we provide comprehensive, scalable and sustainable IT and business consulting services that are informed globally and delivered locally. cgi.com © 2026 CGI Inc. 34