Nykypäivän tekoäly testauksessa

Tulevaisuus on jo täällä, se ei vain ole tasaisesti jakautunut. Scifi-kirjailija Gibsonin sanat toimivat erinomaisena pohjana kun tarkastellaan miten jo käynnissä olevat ketteryyteen ja automatisointiin liittyvät teemat sekä tekoälyn liittyminen muutosten joukkoon vievät testauksen työnjakoa uuteen suuntaan. Yhteisöllisen tekemisen ja sosiaalisten ohjelmistokehityksen ja -testauksen menetelmien asiantuntijana Maaret tuo tekoälyn mukaan testaukseen sosio-teknisten turvarajojen puitteissa, ja tässä puheenvuorossa käydään läpi kolme soveltavaa esimerkkiä tekoälyn hyödyntämisestä tänä päivänä yleisesti saatavilla olevilla teknologioilla.

Tapojen muuttaminen omaa vaikuttavuutta arvioiden on keskeinen taito, teki testausta testaajana tai muussa roolissa. Tänä päivänä testaus on liian tärkeää jätettäväksi vain testaajille ja sen pitäisi kiinnostaa kaikkia. Vastaavasti se on liian tärkeää jätettäväksi ilman erityisosaamista.

Soveltavat esimerkit:
- Testikoodin generointi IDE:ssä
- AI-työkalut katselmoinnissa ja testigeneroinnissa kehitysputkissa
- Oman RAG-apulaisen luominen paritestaukseen

Transcript

1. © 2024 CGI Inc. 1 Nykypäivän tekoäly testauksessa Maaret Pyhäjärvi
2. © 2024 CGI Inc. 2 Acting Humanly for Testing 01 Natural Language processing to communicate successfully in human language • Summarizing details to insights • Generating charters • Creating data, instructions and oracles • Understanding risk coverage 02 Knowledge representation to store what it knows or hears • Remembering features and recognizing feature relationships • Avoiding reporting accepted problems without change in knowledge • Remembering what was done • Knowing what to look at 03 Automated reasoning to answer questions and draw new conclusions • Deciding what conversations to start • Deciding when we can automatically revert • Reporting with repro scripts • Recognizing responsibility of fix 04 Machine learning to adapt to new circumstances and to extrapolate and detect patterns • Bug taxonomies • Priorities • Cross industry reuse of standard tests 05 Computer vision, speech recognition to perceive the world • Multilingual projects • Sources of data • Visual testing aids 06 Robotics to manipulate objects and move around • Robotic process automation as basis of testing • Operating interfaces abstracting away technology of target Any software is marketed as AI since it is doing something humans could do.
3. © 2024 CGI Inc. 3 An AI World Map Text Text /code Sound Image Video Copyrights Training Data Machine Learning Algorithm e.g., LLM input Filter output Filter Fore- ground Back- ground Retrieval pipeline Text Text /code Sound Image Video Copyrights Parameters e.g., temperature Tasks Prompt engineering is the discipline of formulating future background requests in the foreground Copyrights From 2048 tokens (1500 words) to 1M
4. © 2024 CGI Inc. 4 Suurimmat ongelmat AI:n käytölle testauksessa ovat data ja maksaja. - Heini Ahven 2023
5. © 2024 CGI Inc. 5 Prepare to Shift: Left and Down Shift Left Shift down Quality engineering over quality assurance Whole team approach Clarifying core examples Single-commit delivery Test-driven development Exploratory unit testing “77% of production bugs could have been found with a unit test” We’re still half-way into the automation transformation that AI supports.
6. © 2024 CGI Inc. 6 AI in Testing Experiences 2024 Testikoodin generointi IDE:ssä 1 AI-työkalut katselmoinnissa ja testigeneroinnissa kehitysputkissa 2 Oman RAG- apulaisen luominen paritestaukseen 3 Applying today’s AI in testing
7. © 2024 CGI Inc. 7 What changed? Trust relationships Corporation to corporation trust set up by contracts. Context size Input to general purpose model is of relevant size. Packaging as services Abstractions hiding setup that is not AI on side of AI creating hybrid products. Task-specific application External imagination on level of tasks with human in loop.
8. © 2024 CGI Inc. 8 Testikoodin generointi IDE:ssä 1 AI-työkalut katselmoinnissa ja testigeneroinnissa kehitysputkissa 2
9. © 2024 CGI Inc. 9
10. © 2024 CGI Inc. 10 AI of Today in Developer Testing Idea of change *Intent* Pull Request Commit Commit Build and Deploy write code write tests document review review codiumai pr-agent pro code review by gpt github action write tests github copilot codiumai codiumate earlyai jetbrains ai assistant run tests run tools run tools IDE nodes Agents running in
11. © 2024 CGI Inc. 11 “I love the extra autocompletion that I get with it, it feels like I never have to write any kind of boilerplate code anymore, and I also find it very useful to just ask stuff directly in the IDE. I used to google stuff all the time, and ended up on Stackoverflow a lot, but nowadays I rarely have to do that.”
12. © 2024 CGI Inc. 12 GitHub Copilot Code Completion in IDE Chat, inline and in IDE Participants: @workspace Variables: #file #selection #editor Commands: /fix /tests /explain /doc In GitHub Pull request summaries Knowledge bases Spec-Plan-Implement flows Bing for recent ADO for pipelines and environments Lines and blocks *Enterprise Only
13. © 2024 CGI Inc. 13 Oman RAG- apulaisen luominen paritestaukseen 3
14. © 2024 CGI Inc. 14
15. © 2024 CGI Inc. 15 Contemporary Exploratory Testing Assistant ‘AI-Maaret’ Pretrained LLM Body of authored text from one author Responses closer to your bias Cloud models such as ChatGPT Self-hosted models from huggingface
16. © 2024 CGI Inc. 16 Simplest form of larger context
17. © 2024 CGI Inc. 17 GenAI Pair Testing External imagination Confidentiality Ethical compensations Intent and productivity Photo by Rajvir Kaur on Unsplash
18. © 2024 CGI Inc. 18 GenAI Pair Testing Search boundaries: argue for different stances on assumptions Recognize insufficiency and fix it – creating average text is not *your* goal Freedom to criticize as the pair takes no offense Dare to ask things you’d not dare to ask from a colleague Co-piloting allows for repair Photo by Rajvir Kaur on Unsplash
19. © 2024 CGI Inc. 19 Predictive text generation useful for recall not reasoning
20. © 2024 CGI Inc. 20 What else? Today – Soon - Future
21. © 2024 CGI Inc. 21
22. © 2024 CGI Inc. 22 Contemporary Exploratory Testing Programmatic tests as output. Feedback with agency.
23. © 2024 CGI Inc. 23 Picked up folklore that may be useful in framing Automating leaves behind the most difficult 10%. Debugging your own creation is twice as hard as creating it in the first place. 50% of jobs change, 5% of jobs replaced by automation Software industry doubles in 5 years. 1% (4 min) better daily would make 37.8x you in 365 days. There’s plenty of things we want but can’t get to because creating what we have now hogs all bandwidth.
24. © 2024 CGI Inc. 24 Testing is too important to be left just for testers. It’s too important to be left without testers.
25. © 2024 CGI Inc. 25 Quality Engineering and Testing at CGI 01 AI transformation Applying ML, GenAI, RAG in human-centric contemporary exploratory testing 02 Automation transformation Decomposing to actionable feedback and capabilities in pipelines 03 Testing in the world The Creative Commons baseline and integrations with community 04 Testing in CGI Real work with our customers
26. © 2024 CGI Inc. 26 Insights you can act on Founded in 1976, CGI is among the largest IT and business consulting services firms in the world. We are insights-driven and outcomes-based to help accelerate returns on your investments. Across hundreds of locations worldwide, we provide comprehensive, scalable and sustainable IT and business consulting services that are informed globally and delivered locally. cgi.com