Explore the world of test automation through insightful real-world case studies

Slide 1

Slide 1 text

Arrangility Sdn. Bhd. Explore the world of test automation through insightful real-world case studies CEO Tomotaka Asagi

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

© 2025 Arrangility Sdn. Bhd. 1. Introduction: My Group Company 6 Japan Quality Co., Ltd.／ DAD Lab (Overseas verification base) 16F, Da Nang Software Park, 02 Quang Trung, Thach Thang Ward, Hai Chau District, Da Nang, VIETNAM ARRANGILITY SDN. BHD. Unit LG-02 & LG-03, Ground Floor, CoPlace 5 Block 3710, Persiaran APEC, Cyber 8, 63000 Cyberjaya Selangor Darul Ehsan, MALAYSIA Human Crest Co., Ltd.（HQ） 26F Yokohama Landmark Tower, 2-2-1, Minatomirai, Nishi-ku, Yokohama City, Kanagawa, JAPAN 220-8126 Japan・Vietnam・Malaysia

Slide 7

Slide 7 text

Slide 8

Slide 8 text

© 2025 Arrangility Sdn. Bhd. 8 ● Product/Service Used Automated regression testing is efficient but incurs significant costs due to frequent UI changes, updates to the test execution environment, network stability issues, etc. "Lynx" is a regression test automation service. Provides a one-stop service from construction to operation of test automation systems. Continuously runs tests to discover and notify you of problems. Checks response time and display distortion. Multi-browser compatibility Early detection of problems and Significant reduction in operating costs Easy installation with no hassle. Maintenance handled by us

Slide 9

Slide 9 text

Slide 10

Slide 10 text

© 2025 Arrangility Sdn. Bhd. ● Scope ● Scope of Automated Testing: Currently limited to End-to-End (E2E) testing that interacts with the UI layer. Unit tests and other types are not included ● No Blame Assigned: This is not about blaming anyone 10 UI Service Unit

Slide 11

Slide 11 text

© 2025 Arrangility Sdn. Bhd. ● Question that client normally asked? ● Where to begin: ○ I want to do it, but where should I start? ● How to start: ○ What technology do you use? ○ What kind of tests are you running? ● What is the challenges to continue: ○ Too many failures ○ Why? Crash? 11

Slide 12

Slide 12 text

● Test Automation Circles 12 Goal

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© 2025 Arrangility Sdn. Bhd. 2. Common tools & technology ● Test Automation Framework ○ Selenium（Selenium Grid) ○ Appium ○ Cypress ○ Playwrite ● Testing Framework ○ JUnit ○ Cucumber ● Test Execution Environment ○ Real Machine ○ Workspaces ○ Docker ● CI Environment ○ Jenkins ○ GitLab ● Scripting Language ○ Java ● Test Case Management ○ Excel ○ (Test Framework） ● Others ○ OpenCV（Image Comparison） 14

Slide 15

Slide 15 text

© 2025 Arrangility Sdn. Bhd. 2. General Configuration Diagram 15 WorkSpaces Node Hub EC2 Availability Zone ap-northeast-1a Availability Zone ap-northeast-1c LynxDashboard LynxRun GitLab Excel Management Test Case Test Target DockerCompose

Slide 16

Slide 16 text

Slide 17

Slide 17 text

3. Case 1 : Decide your goal 17 Execute Reporting Resource Team Culture Skill Set Test Automation Solution GOAL HERE

Slide 18

Slide 18 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: Overview ● Business-to-Business (B2B) Web Services Company ● Test target System ○ System A (FE: AngularJS , BE: PHP) ○ System B (FE: AngularJS , BE: Java) ● Staging Environment 18

Slide 19

Slide 19 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: Issues ● Issues ○ Concerns regarding product quality, such as the lack of regression testing (involves over 1,000 cases...) ○ System issues are sometimes only detected through user reports ● Request ○ Want to verify if any components have ceased functioning following the update ○ Want to identify any existing issues ○ Want to run daily automated tests 19 Goal → Risk Reduction

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Slide 23

Slide 23 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: How I Managed the Situation 23 Screen Name Function Name Class Function Risk Manual costs (cost to do manually) Automation cost (cost to automate) Whole Multi-user operations Function Low Low Medium Whole Automatic version upgrade Function Low Low Medium HOME Screen User management Data Medium Low Low HOME Screen Login information Display Low Low Low Login Screen Login/Logout Login High Low Low HOME Screen Notice Display Low Low Low HOME Screen Work memo Display Low Low Low HOME Screen Help Display Low Low Low Whole Smartphone compatibility Function High High Medium Product Management Product information list Display High Low Low Product Management Product search (product list) Search Low Low Low Product Management CSV import Import High High Medium

Slide 24

Slide 24 text

© 2025 Arrangility Sdn. Bhd. Screen Name Function Name Class Function Risk Manual costs (cost to do manually) Automation cost (cost to automate) Test Case Test Case No. Mall Setting Inventory ratio setting Setting High High Low inventoryRatioSetting_testCase inventoryRatioSetting_mallLink_testCcase 2 Product Management CSV import Import High High Medium CSVImport_registration_testCase CSVImport_edit_testCase 2 Linkage by product and mall Display High High Medium productAndMallUnitLink_product_testCase productAndMallUnitLink_stock_testCase productAndMallUnitLink_mall_testCase productAndMallUnitLink_stock_mallLink_testCase productAndMallUnitLink_mall_mallLink_testCase 5 Log Mall update log Display High High Medium mallUpdateLog_log_testCase mallUpdateLog_csvExport_testCase mallUpdateLog_search_testCase 3 Mall Setting Mall update time setting Setting High High Medium mallUpdateTimeSetting_testCase 1 System Setting Low stock email notification Setting High High High? lowStockEmailNotification_testCase lowStockEmailNotification_canSent_mallLink_testCase lowStockEmailNotification_cannotSent_mallLink_testCase 3 Log sales log Display High High High? salesLog_log_testCase salesLog_csvExport_testCase salesLog_search_testCase 3 Mall Setting Order cancellation setting Data High High High? orderCancellationSetting_testCase orderCancellationSetting_start_mallLink_testCase orderCancellationSetting_stop_mallLink_testCase 3 Mall Setting FBA multichannel setting Setting High High High? fbaMultichannelSetting_testCase 1 Mall Setting API setting Setting High High High? apiSetting_doneSetting_testCase 1 3. Case 1: How I Managed the Situation 24

Slide 25

Slide 25 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: Result ● Test case number ○ System A: 20 cases ○ System B: 77 cases ● Time until the first test case is executed regularly: 3 weeks ● Test execution frequency: Once a day ● When a test fails: Send summary and screenshot to Slack ● Completion of all test cases: 2 months 25

Slide 26

Slide 26 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: Changes ● Scrum was implemented ● Automation of unit tests has been initiated ● Now able to refactor ● Future cautions ○ Avoid becoming an anti-pattern 26 https://testing.googleblog.com/2020/11/fixing-test-hourglass.html

Slide 27

Slide 27 text

Slide 28

Slide 28 text

© 2025 Arrangility Sdn. Bhd. 3. Case 1: Summary ● Do’s ○ Test independency ○ Test idempotency ■ Preparation（Condition necessary for testing, e.g. test data, etc.） ■ Cleaning up（Deleting unnecessary data） ● Don’ts ○ Automate manual test cases ○ Automate regression test cases → Do not aim automate as goal 28

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Execute Reporting Resource Team Culture Skill Set Test Automation Solution GOAL HERE 4. Case 2 30

Slide 31

Slide 31 text

Slide 32

Slide 32 text

© 2025 Arrangility Sdn. Bhd. 4. Case 2: Overview 「CASIO WATCHES」 conduct automated test focuses on regression checks, to confirm that basic functions are operating normally. ● Goal ○ Early detection of defects during development, reduction of rework, reduction of development cost, and reduction of testing man-hours ○ Overall check after production release, early detection of market defects, ensuring market quality ● Target ○ Regression testing in development environment (regression testing) ○ Regression testing in production environment (regression testing) 32

Slide 33

Slide 33 text

© 2025 Arrangility Sdn. Bhd. 4. Case 2: Overview ● Target App ○ Production app ○ Development app ○ （just before release） ● Test Content ○ Functional test ○ 284 cases/suite ● Text Execution Frequency ○ Run twice a day（12 AM, 12 PM) 33

Slide 34

Slide 34 text

4. Case 2: Automated Testing Overall Configuration 34 1,200 Execution/day Production Env. Production Env. Development Env. Development Env.

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

© 2025 Arrangility Sdn. Bhd. 4. Case 2: First Issue in Automated Test Operations ● Issue: Test failures lead to lengthy investigation and resolution time ○ Each case execution time takes 10-30 minutes → Rerun → Crashes midway... And crashes again during correction checks ● Cause ○ Automated screen-by-screen test cases ● Countermeasure ○ Redesign and split cases for automated testing. ○ Increased from 48 → 284 cases (Final : 500 cases) ● Effect ○ Significantly reduces investigation time for test failures ○ Test are now stable 37 Too Long!

Slide 38

Slide 38 text

© 2025 Arrangility Sdn. Bhd. 4. Case 2: Second Issues in Automated Test Operations ● Issue: Tests fail due to minor screen changes ○ Frequent modification of test scripts, leading to instability (There were always a lot of unstable parts) ● Cause ○ Long XPATH changes during scrolling ○ Accuracy issues with OpenCV image matching ● Countermeasure ○ Add an ID (recognized by Appium) ● Effect ○ Test are now stable ○ Enhanced collaboration with the development team 38

Slide 39

Slide 39 text

© 2025 Arrangility Sdn. Bhd. 4. Case 2: Key Points for Continuation 1. Assign a Responsible Person 2. Address Failed Tests Promptly: Establish a policy within 24 hours 3. Ensure Test Stability: Remove any unstable tests 4. Continuously Refactor Tests: Reinforce the goal and value of your tests 5. Seek Team Collaboration: Engage developers and other team members 39