Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Efficient automation regression testing using historical testing data @ TECHPULSE 2023

Efficient automation regression testing using historical testing data @ TECHPULSE 2023

- Speaker: Steven Lee
- Event: http://techpulse.line.me/

泰國作為 LINE TODAY 第二大市場,坐擁許多各式各樣的在地化功能。隨著功能越來越多,許多功能都需要透過回歸測試來確保功能完整。在此議程中,將會分享如何透過各式工具與手法,與跨國 QA 團隊合作,提高回歸測試的穩定性,若您重視測試品質與穩定性,此議程絕對是您不可錯過的一場!

LINE Developers Taiwan

February 21, 2023
Tweet

More Decks by LINE Developers Taiwan

Other Decks in Technology

Transcript

  1. Agenda
    › Introduction
    › Infrastructure
    › MySQL Database
    › Verda Function
    › GitHub Webhook
    › Grafana Dashboard
    › Result
    › Conclusion
    2

    View full-size slide

  2. Introduction
    Problem
    › The runtime of the job when running tests in parallel is equal to the longest time spent
    among the parallel groups
    › Manually allocating the test cases to each group requires man power
    › Cannot dynamically control how many parallel groups to run
    3
    3 parallel groups with 77 files which contains 291 test cases

    View full-size slide

  3. Introduction
    Goal
    › Reduce the time difference of each parallel run by allocating the testing time
    of each group more evenly
    › Automatically distribute the tests without requiring manpower
    › Dynamically control how many parallel groups to run
    4
    Group1
    20 minutes
    Group2
    20 minutes
    GroupN
    20 minutes
    Parallel running docker Jenkins job

    View full-size slide

  4. 5
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert
    test data
    Create dashboard
    with DB
    Deployment
    completes
    QA Engineer
    merges PR
    Infrastructure

    View full-size slide

  5. 6
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    MySQL Database
    Deployment
    completes
    QA Engineer
    merges PR
    Deployment
    completes
    QA Engineer
    merges PR

    View full-size slide

  6. 7
    Field Type
    Duration int
    Start time timestamp
    Test file varchar(1024)
    Data that saves to MySQL
    JSON received after cypress run
    MySQL Database

    View full-size slide

  7. Generate data for calculation and analyzation
    › The test file isn't unique
    › Data is inserted to MySQL database every time cypress test is conducted
    › Clear data which has start time 3 months ago
    8
    Data in MySQL
    MySQL Database

    View full-size slide

  8. 9
    Deployment
    completes
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    QA Engineer
    merges PR
    Verda Function
    Deployment
    completes
    QA Engineer
    merges PR

    View full-size slide

  9. 10
    Example of test files
    Example of VERDA function API response
    Verda Function
    › Create an API to calculate how to allocate each test file to each group and show the expected
    time for each group to run
    › API Query Parameters
    › folder => input the folder path you want to test ex: folder = e2e/2-advanced-examples
    › group => input how many parallel jobs you want to create ex: group = 3
    › API Responses
    › Response: Distributed test file in each group
    › Times: Expected time for each group to finish running (milliseconds)

    View full-size slide

  10. 11
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 800 2022/02/15
    /e2e/1-getting-started/to-do.cy.js 1200 2022/02/14
    /e2e/1-getting-started/to-do.cy.js 1000 2022/02/13
    /e2e/1-getting-started/to-do.cy.js 1500 2022/02/01
    Example of MySQL data
    Test file Duration
    /e2e/1-getting-started/to-do.cy.js (800+1200+1000)/3=1000
    Verda Function
    › Select data with
    test file matching
    the folder query
    and start time within
    5 days
    › Group by test file
    using the average of
    these data’s duration

    View full-size slide

  11. 12
    Test file Duration
    /e2e/1-getting-started/to-do.cy.js 100000
    /e2e/2-advanced-example/assertions.cy.js 80000
    /e2e/2-advanced-example/cookies.cy.js 40000
    nth test file 1000
    › Create N groups according to group query
    › List the test file by duration descending
    › Repeatably throw the test file to the group with the lowest accumulated run time
    Verda Function
    Group1
    Group2
    GroupN

    View full-size slide

  12. 13
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    GitHub Webhook
    Deployment
    completes
    QA Engineer
    merges PR

    View full-size slide

  13. Prevent MySQL database to have missing test files
    › Create PR to add new feature automation test
    › Merge code into develop branch of GitHub repo
    › Trigger Jenkins job with GitHub web hook (Jenkins Git SCM polling plugin)
    14
    Create new
    pull request
    Automation
    GitHub repo
    merged
    Update test files
    to MySQL db
    trigger
    Jenkins
    GitHub Webhook

    View full-size slide

  14. 15
    Test file structure tree
    MySQL data
    › Scans all test files inside your Git Repository and insert new data only if there is
    no corresponding test file in MySQL database within 5 days
    › The inserted row has test file (missing folder), duration (5000), start time
    (Current time)
    GitHub Webhook
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 8000 2022/02/15
    /e2e/2-advanced-examples/actions.cy.js 12000 2022/02/14
    /e2e/2-advanced-examples/aliasing.cy.js 10000 2022/02/13
    /e2e/1-getting-started/newTest.cy.js 5000 Current time
    Test file Duration Start time
    /e2e/1-getting-started/to-do.cy.js 8000 2022/02/15
    /e2e/2-advanced-examples/actions.cy.js 12000 2022/02/14
    /e2e/2-advanced-examples/aliasing.cy.js 10000 2022/02/13

    View full-size slide

  15. 16
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    Grafana Dashboard
    Deployment
    completes
    QA Engineer
    merges PR

    View full-size slide

  16. 17
    Grafana Dashboard

    View full-size slide

  17. 18
    Request VERDA
    function API
    Calculate how to group
    from data in DB
    Insert new created spec file
    data to MySQL database
    Run test with
    Cypress
    Insert test data
    Create dashboard
    with DB
    Infrastructure
    Deployment
    completes
    QA Engineer
    merges PR

    View full-size slide

  18. 19
    › With historical data running test in parallel – less than 10 mins
    › Distributed 77 files evenly in the parallel groups
    Result

    View full-size slide

  19. 20
    Pros
    › Can easily distribute the total time of running the test files evenly
    › Can have records of how your tests are running
    › Won't need to manually distribute what cases need to be run when new
    automation cases are created
    › Can directly choose what test you want to run from the folder parameter
    Tips
    › Dependency between tests files should be isolated
    Conclusion

    View full-size slide