Why Mastering QA Skills Beyond AI is Needed

Slide 1

Slide 1 text

Arrangility Sdn. Bhd. Why Mastering QA Skills Beyond AI is Needed CEO Tomotaka Asagi

Slide 2

Slide 2 text

© 2025 Arrangility Sdn. Bhd. AGENDA 2 1. Introduction 2. AI-Powered Software Test Design 3. Test Automation Circles 4. Case Study: Dealing with Operational Issues 5. Specialized Skills That Cannot Be Replaced by AI 6. Summary

Slide 3

Slide 3 text

© 2025 Arrangility Sdn. Bhd. Self-introduction Tomotaka Asagi CEO 3 ● Passion: Automated Testing ● Hobbies: Photography, Driving ● Work History: Sales → Development → Testing Software testing professional #ScrumMaster #CTAL-TM #CTAL- TA #CTFL

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

© 2025 Arrangility Sdn. Bhd. 1. Introduction: My Group Company 6 Japan Quality Co., Ltd.／ DAD Lab (Overseas verification base) 16F, Da Nang Software Park, 02 Quang Trung, Thach Thang Ward, Hai Chau District, Da Nang, VIETNAM ARRANGILITY SDN. BHD. Unit LG-02 & LG-03, Ground Floor, CoPlace 5 Block 3710, Persiaran APEC, Cyber 8, 63000 Cyberjaya, Selangor Darul Ehsan, MALAYSIA Human Crest Co., Ltd.（HQ） 26F Yokohama Landmark Tower, 2-2-1, Minatomirai, Nishi-ku, Yokohama City, Kanagawa, JAPAN 220-8126 Japan・Vietnam・Malaysia

Slide 7

Slide 7 text

© 2025 Arrangility Sdn. Bhd. 1. Introduction: Meet our team 7 CEO Tomotaka Asagi tomotaka- asagi syazwanswong YukiMiyamoto1210 fadzlianafarouk COO Yuki Miyamoto Solution Sales Leader Muhd Syazwan Solution Sales Nur Fadzliana

Slide 8

Slide 8 text

Slide 9

Slide 9 text

© 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design - Challenges of software testing ● Increase in ○ testing man-hours ○ the complexity of systems ○ fast-paced development cycles ● Limitations of test automation and expectations for AI “Why Mastering QA Skills Beyond AI is Needed” 9

Slide 10

Slide 10 text

© 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design ● How AI can be used in test design ○ Automated test case generation ○ Automated test data generation ○ Support for risk analysis in risk-based testing ● Example: 10 1. Training Data AI 3. Test Case 2. Prompt ● Requirement definitions ● Functional specifications ● ISTQB syllabus (test methodologies, etc.) Test Objectives: Test Scope: Test Type: Test Techniques: —---------------- Output Format: Test Code Language:

Slide 11

Slide 11 text

© 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design ● Benefits of AI-powered test design ○ Improved test coverage ○ Reduced test man-hours ○ Early defect detection 11 UI Service Unit No of tests

Slide 12

Slide 12 text

© 2025 Arrangility Sdn. Bhd. 2. AI-Powered Software Test Design ● Cautions for AI-powered test design ○ AI hallucinations (generation of incorrect information) ○ Biases in AI-generated test cases ○ Black box of AI's reasoning 12

Slide 13

Slide 13 text

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

3. Overview of Test Automation Circles 16

Slide 17

Slide 17 text

© 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: Why? ● Why (Core): Clarify the objective of the test ○ AI's role Propose optimal strategies, risk analysis, and prioritization ○ QA Engineer skills Evaluate and adjust strategies, clearly define the goal, and reach agreement with stakeholders ● Examples of test goal: ○ No bugs ○ Fulfill the requirements ○ Key functions are working 17

Slide 18

Slide 18 text

© 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: How? ● How (Concept): Select test strategies and techniques ○ AI's role Select appropriate test techniques, automatically generating test cases ○ QA Engineer skills Review test cases, verify completeness and validity, identify risks and boundary conditions ● Examples of test techniques: ○ Unit Test : White box testing , coverage ○ E2E : Use Case / scenario, Functional testing 18

Slide 19

Slide 19 text

© 2025 Arrangility Sdn. Bhd. 3. Test Automation Circles: What? ● What (Architecture): Prepare tools and environment ○ AI's role Select tools and frameworks needed for test automation, generate test scripts, and linking with CI/CD systems ○ QA Engineer skills Maintain and improve test scripts, build and maintain test environment ● Examples of tools and environments: ○ Test automation tools ○ CI/CD tools ○ Test environment construction tool 19

Slide 20

Slide 20 text

Slide 21

Slide 21 text

© 2025 Arrangility Sdn. Bhd. 4. Case Study - Scope ● Scope of Automated Testing: Currently limited to End-to-End (E2E) testing that interacts with the UI layer. Unit tests and other types are not included ● No Blame Assigned: This is not about blaming anyone 21 UI Service Unit No of tests

Slide 22

Slide 22 text

Execute Reporting Resource Team Culture Skill Set Test Automation Solution Objective HERE 4. Case Study 22

Slide 23

Slide 23 text

Slide 24

Slide 24 text

© 2025 Arrangility Sdn. Bhd. 4. Case Study: Overview [CASIO WATCHES] Conduct an automated test for regression checks to ensure basic functions are operating normally. ● Goal ○ Early detection of defects during development, market defects ○ Reduction of rework, development cost, and testing man-hours ● Target ○ Regression testing in development and production environment (regression testing) for Production and Development App ● Test Content Functional Test - 284 caes/suite Test Execution Frequency - Run twice a day (12AM / 12PM) 24

Slide 25

Slide 25 text

4. Case Study: Automated Testing Overall Configuration 25 1,200 Execution/day Production Env. Production Env. Development Env. Development Env. within the same network Watch A Watch A Watch B Watch B Real Devices Send result

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2025 Arrangility Sdn. Bhd. 4. Case Study: First Issue in Automated Test Operations ● Issue: Test failures lead to lengthy investigation and resolution time ○ Each case execution time takes 10-30 minutes → Rerun → Crashes midway... And crashes again during correction checks ● Cause ○ Automated screen-by-screen test cases ● Countermeasure ○ Redesign and split cases for automated testing. ○ Increased from 48 → 284 cases (Final : 500 cases) ● Outcome ○ Significantly reduces investigation time for test failures ○ Test are now stable 28 Too Long!

Slide 29

Slide 29 text

© 2025 Arrangility Sdn. Bhd. 4. Case Study: Second Issues in Automated Test Operations ● Issue: Tests fail due to minor screen changes ○ Frequent modification of test scripts, leading to instability (There were always a lot of unstable parts) ● Cause ○ Long XPATH changes during scrolling ○ Accuracy issues with OpenCV image matching ● Countermeasure ○ Add an ID (recognized by Appium) ● Outcome ○ Test are now stable ○ Enhanced collaboration with the development team 29

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2025 Arrangility Sdn. Bhd. 5. Specialized Skills ● Necessity of reviewing test designs generated by AI ○ Correct AI errors and biases ○ Identify risks that AI has not considered ● Types of reviews: ○ Test plan review ○ Test design review ○ Test case review ○ Code review (Product & Test) 32

Slide 33

Slide 33 text

© 2025 Arrangility Sdn. Bhd. 5. Specialized Skills ● Specialized QA Engineering Skills That AI Can't Replace ○ Knowledge of test design ○ Domain knowledge ○ Risk analysis ability ○ Creative test execution 33

Slide 34

Slide 34 text

Slide 35

Slide 35 text

© 2025 Arrangility Sdn. Bhd. 6. Summary ● AI can be a powerful tool for software testing ● Human expertise is essential to effectively utilize AI 35 That is why Quality assurance through human-AI collaboration is important