Why Mastering QA Skills Beyond AI is Needed

https://itlead.usim.edu.my/ai-day/

Transcript

1. Arrangility Sdn. Bhd. Why Mastering QA Skills Beyond AI is Needed CEO Tomotaka Asagi
2. © 2025 Arrangility Sdn. Bhd. AGENDA 2 1. Introduction 2. AI-Powered Software Test Design 3. Test Automation Circles 4. Case Study: Dealing with Operational Issues 5. Specialized Skills That Cannot Be Replaced by AI 6. Summary
3. © 2025 Arrangility Sdn. Bhd. Self-introduction Tomotaka Asagi CEO 3 • Passion: Automated Testing • Hobbies: Photography, Driving • Work History: Sales → Development → Testing Software testing professional #ScrumMaster #CTAL-TM #CTAL- TA #CTFL
4. © 2025 Arrangility Sdn. Bhd. Introduction 1. 4
5. © 2025 Arrangility Sdn. Bhd. Specializes in Automated Software Testing & QA Consulting 1. Introduction: My Company 5
6. © 2025 Arrangility Sdn. Bhd. 1. Introduction: My Group Company 6 Japan Quality Co., Ltd.／ DAD Lab (Overseas verification base) 16F, Da Nang Software Park, 02 Quang Trung, Thach Thang Ward, Hai Chau District, Da Nang, VIETNAM ARRANGILITY SDN. BHD. Unit LG-02 & LG-03, Ground Floor, CoPlace 5 Block 3710, Persiaran APEC, Cyber 8, 63000 Cyberjaya, Selangor Darul Ehsan, MALAYSIA Human Crest Co., Ltd.（HQ） 26F Yokohama Landmark Tower, 2-2-1, Minatomirai, Nishi-ku, Yokohama City, Kanagawa, JAPAN 220-8126 Japan・Vietnam・Malaysia
7. © 2025 Arrangility Sdn. Bhd. 1. Introduction: Meet our team 7 CEO Tomotaka Asagi tomotaka- asagi syazwanswong YukiMiyamoto1210 fadzlianafarouk COO Yuki Miyamoto Solution Sales Leader Muhd Syazwan Solution Sales Nur Fadzliana
8. © 2025 Arrangility Sdn. Bhd. Possibilities and Limitations 2. AI-Powered Software Test Design 8
9. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design - Challenges of software testing • Increase in ◦ testing man-hours ◦ the complexity of systems ◦ fast-paced development cycles • Limitations of test automation and expectations for AI “Why Mastering QA Skills Beyond AI is Needed” 9
10. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design • How AI can be used in test design ◦ Automated test case generation ◦ Automated test data generation ◦ Support for risk analysis in risk-based testing • Example: 10 1. Training Data AI 3. Test Case 2. Prompt • Requirement definitions • Functional specifications • ISTQB syllabus (test methodologies, etc.) Test Objectives: Test Scope: Test Type: Test Techniques: —---------------- Output Format: Test Code Language:
11. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design • Benefits of AI-powered test design ◦ Improved test coverage ◦ Reduced test man-hours ◦ Early defect detection 11 UI Service Unit No of tests
12. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design • Cautions for AI-powered test design ◦ AI hallucinations (generation of incorrect information) ◦ Biases in AI-generated test cases ◦ Black box of AI's reasoning 12
13. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design • Example 13 Test technique: Create a decision table from the shipping fee conditions. AI This is Pattern table
14. © 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design • Example 14 Test technique: Create a decision table from the shipping fee conditions. AI ✘ Pattern table ✓ Decision table
15. © 2025 Arrangility Sdn. Bhd. A Framework for AI Utilization 3. Test Automation Circles 15
16. 3. Overview of Test Automation Circles 16
17. © 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: Why? • Why (Core): Clarify the objective of the test ◦ AI's role Propose optimal strategies, risk analysis, and prioritization ◦ QA Engineer skills Evaluate and adjust strategies, clearly define the goal, and reach agreement with stakeholders • Examples of test goal: ◦ No bugs ◦ Fulfill the requirements ◦ Key functions are working 17
18. © 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: How? • How (Concept): Select test strategies and techniques ◦ AI's role Select appropriate test techniques, automatically generating test cases ◦ QA Engineer skills Review test cases, verify completeness and validity, identify risks and boundary conditions • Examples of test techniques: ◦ Unit Test : White box testing , coverage ◦ E2E : Use Case / scenario, Functional testing 18
19. © 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: What? • What (Architecture): Prepare tools and environment ◦ AI's role Select tools and frameworks needed for test automation, generate test scripts, and linking with CI/CD systems ◦ QA Engineer skills Maintain and improve test scripts, build and maintain test environment • Examples of tools and environments: ◦ Test automation tools ◦ CI/CD tools ◦ Test environment construction tool 19
20. © 2025 Arrangility Sdn. Bhd. Dealing with operational issues 4. Case Study 20
21. © 2025 Arrangility Sdn. Bhd. 4. Case Study - Scope • Scope of Automated Testing: Currently limited to End-to-End (E2E) testing that interacts with the UI layer. Unit tests and other types are not included • No Blame Assigned: This is not about blaming anyone 21 UI Service Unit No of tests
22. Execute Reporting Resource Team Culture Skill Set Test Automation Solution Objective HERE 4. Case Study 22
23. © 2025 Arrangility Sdn. Bhd. 4. Case Study 23 CASIO COMPUTER CO., LTD. CASIO WATCHES
24. © 2025 Arrangility Sdn. Bhd. 4. Case Study: Overview [CASIO WATCHES] Conduct an automated test for regression checks to ensure basic functions are operating normally. • Goal ◦ Early detection of defects during development, market defects ◦ Reduction of rework, development cost, and testing man-hours • Target ◦ Regression testing in development and production environment (regression testing) for Production and Development App • Test Content Functional Test - 284 caes/suite Test Execution Frequency - Run twice a day (12AM / 12PM) 24
25. 4. Case Study: Automated Testing Overall Configuration 25 1,200 Execution/day Production Env. Production Env. Development Env. Development Env. within the same network Watch A Watch A Watch B Watch B Real Devices Send result
26. © 2025 Arrangility Sdn. Bhd. 4. Case Study: Issues 1. Difficulty in managing test failures 2. Test is too unstable 26
27. © 2025 Arrangility Sdn. Bhd. 4. Case Study: Issues Faced in Automated Test Operations Automated test & Problem Solving 27
28. © 2025 Arrangility Sdn. Bhd. 4. Case Study: First Issue in Automated Test Operations • Issue: Test failures lead to lengthy investigation and resolution time ◦ Each case execution time takes 10-30 minutes → Rerun → Crashes midway... And crashes again during correction checks • Cause ◦ Automated screen-by-screen test cases • Countermeasure ◦ Redesign and split cases for automated testing. ◦ Increased from 48 → 284 cases (Final : 500 cases) • Outcome ◦ Significantly reduces investigation time for test failures ◦ Test are now stable 28 Too Long!
29. © 2025 Arrangility Sdn. Bhd. 4. Case Study: Second Issues in Automated Test Operations • Issue: Tests fail due to minor screen changes ◦ Frequent modification of test scripts, leading to instability (There were always a lot of unstable parts) • Cause ◦ Long XPATH changes during scrolling ◦ Accuracy issues with OpenCV image matching • Countermeasure ◦ Add an ID (recognized by Appium) • Outcome ◦ Test are now stable ◦ Enhanced collaboration with the development team 29
30. © 2025 Arrangility Sdn. Bhd. 4. Case Study: Key Points • Points to solve the problem ◦ Reevaluating test objective ◦ Emphasize human collaboration ◦ Improve QA engineers’ skills 30
31. © 2025 Arrangility Sdn. Bhd. The Importance of Review 5. Specialized Skills That Cannot Be Replaced by AI 31
32. © 2025 Arrangility Sdn. Bhd. 5. Specialized Skills • Necessity of reviewing test designs generated by AI ◦ Correct AI errors and biases ◦ Identify risks that AI has not considered • Types of reviews: ◦ Test plan review ◦ Test design review ◦ Test case review ◦ Code review (Product & Test) 32
33. © 2025 Arrangility Sdn. Bhd. 5. Specialized Skills • Specialized QA Engineering Skills That AI Can't Replace ◦ Knowledge of test design ◦ Domain knowledge ◦ Risk analysis ability ◦ Creative test execution 33
34. © 2025 Arrangility Sdn. Bhd. Summary 6. 34
35. © 2025 Arrangility Sdn. Bhd. 6. Summary • AI can be a powerful tool for software testing • Human expertise is essential to effectively utilize AI 35 That is why Quality assurance through human-AI collaboration is important
36. © 2025 Arrangility Sdn. Bhd. Thank you. 36