Testing Assembly: Working through uncertainty - collaborative learning

Slides for co-delivered presentation with Arya at Testing Assembly.

Transcript

1. TRACK 1 – Auditorium 13:15–13:45 Working through uncertainty Collaborative learning of AI Aryadevi Neelakantabhattathiri · Searching for next opportunity Maaret Pyhäjärvi · CGI Suomi Oy © 2026 CGI Inc. 1
2. NEWTON’S LAWS APPLIED FOR SOCIAL FORCES Direction change by an external force Testing now 4 1 Testing before 3 2 © 2026 CGI Inc. Immediate insights Disruption of knowledge 2
3. FOUR SESSIONS OF COLLABORATIVE LEARNING Disruption of knowledge 2 Access to code + web UI List of 32 functional problems Automate tests that fail for those problems Generate fixes for those problems and run test automation to show change Generate basic flows test automation as documentation Audit for accessibility and security Gene rating unit and webU I tests to lock existing applica tions fo r chang e https://qe-at-cgi-fi.github.io/two-versions/version1/ © 2026 CGI Inc. 3 Te s t t a r g e t b y C h r i s t i n e P i n t o
4. INSIGHTS WITH ARYADEVI NEELAKANTABHATTATHIRI Tester task expansion through vibe coding 3 1 2 3 4 5 Spec lives in your head and AI guesses it Code looks right even when its wrong You cannot verify what you cannot see Validation had a finish line – now it never ends You are simultaneously too close, too far The date format specified was no more complete than asking for a date picker. Silence turns to assumptions. Clean, linted, running code with tests. Reading with tests for intended behavior at boundaries becomes a necessity. Illusion of past “full visibility”. Black-boxes through AI in the architecture and cognitive distance from writing code. Debugging is a focus. Moving target, particularly with AI in the architecture. Shift from phase to habit. Other techniques (personas, delay, AI assistance, adversial thinking) to objectivity when roles no longer suffice. © 2026 CGI Inc. 4
5. INCREASE IN SENSE OF UNCERTAINTY – EPISTEMIC COVER: HOW WE KNOW WHAT WE KNOW Destabilize the fixed anchor Responsible for information on what is true, not just agreed. Generate the expected. Responsible for the standard. Imaginary anchor Exploratory pull (internal) What the world could be. Uncertainty to spec. Tester Spec If correctness is defined externally, tester job is comparison. © 2026 CGI Inc. Imagination is now a job requirement. Accountability shift from document to judgement. AI in testing pull (external) Imagination is no longer a private resource. Task expansion to product and developer. 5
6. Practice at: https://exploratory-testing-academy.github.io/capture-the-bugs/ Scale d assess ment = self -asse ssment © 2026 CGI Inc. 6
7. Exploratory testing: what should happen? LIVE DEMO · CAPTURE THE BUGS / E-PRIMER The task: check text for forms of “to be” 1 · Conflicting results “To be or not to be” Discouraged Words: 2 Possible Violations: 0 2 · Different characters “We are ready” “We're ready” / “We’re ready” Observation ≠ confirmed bug © 2026 CGI Inc. ? 3 · Formatting “We are ready” versus a line break before “ready” What I don't know yet: Are contractions in scope? How are words counted? What is the expected behavior? 7
8. Three uncertainties—and why a score isn't enough FROM FINDINGS TO VALIDATION 1 2 3 Specification Test oracle Coverage What counts as a violation? How do I know the result is wrong? What have I tested? What behavior is intended? What is my independent reference? What remains unexplored? A score tells me how findings match a reference list. It does not, by itself, prove the tool is trustworthy. Also question the evaluator: consistency, wording, and limitations. © 2026 CGI Inc. 8
9. Working code ≠ correct behavior THE AI CONNECTION 1 What AI may do ASK YOURSELF Generate code while filling gaps with assumptions. What am I assuming? 2 What testers do Question assumptions and define expected behavior. What should happen? 3 What evidence does Is the evidence enough? © 2026 CGI Inc. Helps us decide what we can trust. 9
10. AI ENABLES TASK EXPANSION People can now do things by asking for it. © 2026 CGI Inc. 10
11. AI-native journey: individual experiences by role 4 30 experiences people grow into. Knowledge workers track 12; developers track those plus 18 more. Knowledge worker role 01 02 12 experiences External imagination Developer role 01 Code autocomplete 10 Team/Product context 02 Reactive agentic 11 Team/Product context advanced 03 Selecting and configuring 3rd party MCPs 12 Building own MCPs 04 Integrating 3rd party AI system components 13 Delivering AI native project 05 Skills and AI-friendly git-first templates 14 Product AI feature delivery for proof of concept Local summarizing 03 Online summarizing 04 Integrated AI features in domain tools 05 AI-enabled practices 18 experiences you start it e.g. .md files e.g. MCPs, memory, skills, orchestration e.g. Rovo for Confluence & Jira e.g. AI notetaking memory, context compression 06 Task expansion 07 Personal applications 08 2nd brain 06 Using local models 15 Added business value in domain AI proof of concept 09 Token awareness 07 Proactive agentic 16 Product AI feature delivery to run in production 08 Dark factory 17 Product feature AI feedback 09 Spec-driven development 18 Sandboxing 10 Generating artifacts 11 Pitch to project for AI usecase 12 Mentioned in CV it starts automatically unattended An experience counts when you can tell its story: “…when AI makes it possible for you to do things you used to think as someone else’s work.” Task expansion © 2026 CGI Inc. Source: AI-Native Journey Tracker, qe-at-cgi-fi.github.io/ai-journey 11
12. AGE OF AI AI ends the split between verification and exploration: you can’t check against requirements nobody wrote down. © 2026 CGI Inc. 12
13. Event program and Feedback https://testingassembly.fistb.fi/program/ Scan the code and tell us what you thought. © 2026 CGI Inc. 13