Efficient automation regression testing using historical testing data @ TECHPULSE 2023

- Speaker: Steven Lee
- Event: http://techpulse.line.me/

泰國作為 LINE TODAY 第二大市場，坐擁許多各式各樣的在地化功能。隨著功能越來越多，許多功能都需要透過回歸測試來確保功能完整。在此議程中，將會分享如何透過各式工具與手法，與跨國 QA 團隊合作，提高回歸測試的穩定性，若您重視測試品質與穩定性，此議程絕對是您不可錯過的一場！

Transcript

1. View Slide

2. Agenda
   › Introduction
   › Infrastructure
   › MySQL Database
   › Verda Function
   › GitHub Webhook
   › Grafana Dashboard
   › Result
   › Conclusion
   2
   

   View Slide

3. Introduction
   Problem
   › The runtime of the job when running tests in parallel is equal to the longest time spent
   among the parallel groups
   › Manually allocating the test cases to each group requires man power
   › Cannot dynamically control how many parallel groups to run
   3
   3 parallel groups with 77 files which contains 291 test cases
   

   View Slide

4. Introduction
   Goal
   › Reduce the time difference of each parallel run by allocating the testing time
   of each group more evenly
   › Automatically distribute the tests without requiring manpower
   › Dynamically control how many parallel groups to run
   4
   Group1
   20 minutes
   Group2
   20 minutes
   GroupN
   20 minutes
   Parallel running docker Jenkins job
   

   View Slide

5. 5
   Request VERDA
   function API
   Calculate how to group
   from data in DB
   Insert new created spec file
   data to MySQL database
   Run test with
   Cypress
   Insert
   test data
   Create dashboard
   with DB
   Deployment
   completes
   QA Engineer
   merges PR
   Infrastructure
   

   View Slide

6. 6
   Request VERDA
   function API
   Calculate how to group
   from data in DB
   Insert new created spec file
   data to MySQL database
   Run test with
   Cypress
   Insert test data
   Create dashboard
   with DB
   MySQL Database
   Deployment
   completes
   QA Engineer
   merges PR
   Deployment
   completes
   QA Engineer
   merges PR
   

   View Slide

7. 7
   Field Type
   Duration int
   Start time timestamp
   Test file varchar(1024)
   Data that saves to MySQL
   JSON received after cypress run
   MySQL Database
   

   View Slide

8. Generate data for calculation and analyzation
   › The test file isn't unique
   › Data is inserted to MySQL database every time cypress test is conducted
   › Clear data which has start time 3 months ago
   8
   Data in MySQL
   MySQL Database
   

   View Slide

9. 9
   Deployment
   completes
   Request VERDA
   function API
   Calculate how to group
   from data in DB
   Insert new created spec file
   data to MySQL database
   Run test with
   Cypress
   Insert test data
   Create dashboard
   with DB
   QA Engineer
   merges PR
   Verda Function
   Deployment
   completes
   QA Engineer
   merges PR
   

   View Slide

10. 10
    Example of test files
    Example of VERDA function API response
    Verda Function
    › Create an API to calculate how to allocate each test file to each group and show the expected
    time for each group to run
    › API Query Parameters
    › folder => input the folder path you want to test ex: folder = e2e/2-advanced-examples
    › group => input how many parallel jobs you want to create ex: group = 3
    › API Responses
    › Response: Distributed test file in each group
    › Times: Expected time for each group to finish running (milliseconds)
    

    View Slide

11. 11
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 800 2022/02/15
    /e2e/1-getting-started/to-do.cy.js 1200 2022/02/14
    /e2e/1-getting-started/to-do.cy.js 1000 2022/02/13
    /e2e/1-getting-started/to-do.cy.js 1500 2022/02/01
    Example of MySQL data
    Test file Duration
    /e2e/1-getting-started/to-do.cy.js (800+1200+1000)/3=1000
    Verda Function
    › Select data with
    test file matching
    the folder query
    and start time within
    5 days
    › Group by test file
    using the average of
    these data’s duration
    

    View Slide

12. 12
    Test file Duration
    /e2e/1-getting-started/to-do.cy.js 100000
    /e2e/2-advanced-example/assertions.cy.js 80000
    /e2e/2-advanced-example/cookies.cy.js 40000
    nth test file 1000
    › Create N groups according to group query
    › List the test file by duration descending
    › Repeatably throw the test file to the group with the lowest accumulated run time
    Verda Function
    Group1
    Group2
    GroupN
    

    View Slide

13. 13
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    GitHub Webhook
    Deployment
    completes
    QA Engineer
    merges PR
    

    View Slide

14. Prevent MySQL database to have missing test files
    › Create PR to add new feature automation test
    › Merge code into develop branch of GitHub repo
    › Trigger Jenkins job with GitHub web hook (Jenkins Git SCM polling plugin)
    14
    Create new
    pull request
    Automation
    GitHub repo
    merged
    Update test files
    to MySQL db
    trigger
    Jenkins
    GitHub Webhook
    

    View Slide

15. 15
    Test file structure tree
    MySQL data
    › Scans all test files inside your Git Repository and insert new data only if there is
    no corresponding test file in MySQL database within 5 days
    › The inserted row has test file (missing folder), duration (5000), start time
    (Current time)
    GitHub Webhook
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 8000 2022/02/15
    /e2e/2-advanced-examples/actions.cy.js 12000 2022/02/14
    /e2e/2-advanced-examples/aliasing.cy.js 10000 2022/02/13
    /e2e/1-getting-started/newTest.cy.js 5000 Current time
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 8000 2022/02/15
    /e2e/2-advanced-examples/actions.cy.js 12000 2022/02/14
    /e2e/2-advanced-examples/aliasing.cy.js 10000 2022/02/13
    

    View Slide

16. 16
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    Grafana Dashboard
    Deployment
    completes
    QA Engineer
    merges PR
    

    View Slide

17. 17
    Grafana Dashboard
    

    View Slide

18. 18
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    Infrastructure
    Deployment
    completes
    QA Engineer
    merges PR
    

    View Slide

19. 19
    › With historical data running test in parallel – less than 10 mins
    › Distributed 77 files evenly in the parallel groups
    Result
    

    View Slide

20. 20
    Pros
    › Can easily distribute the total time of running the test files evenly
    › Can have records of how your tests are running
    › Won't need to manually distribute what cases need to be run when new
    automation cases are created
    › Can directly choose what test you want to run from the folder parameter
    Tips
    › Dependency between tests files should be isolated
    Conclusion
    

    View Slide

21. Thank you
    21
    

    View Slide