Quantifying the Health of End-to-End Test Automation

The slides for Tokyo Test Fest, held on November 15, 2024.

Transcript

1. Quantifying the Health of End-to-End Test Automation Nozomi Ito
2. • CEO of MagicPod • MagicPod: a no-code, cloud-based test automation tool • Organizer of the Japan Selenium User Community • Author of a Japanese Selenium book About me Nozomi Ito
3. Agenda 1. Health Score of End-to-End Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score
4. Agenda 1. Health Score of End-to-End Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score
5. • By reduced cost? • By number of bugs found? • By DORA Metrics? ⚬ Deployment Frequency, Lead Time, Change Failure Rate, MTTR • By business growth? • By profit margin? How do we quantify the success of End-to-End test automation?
6. Continue Results justify the cost Cancel For MagicPod Sucess of the customer’s test automation They continue to use the product without cancellation Results do not justify the cost
7. Interviewed over 100 MagicPod customers (including those who cancelled) Measuring Success with the ‘Health Score’ Defined ‘health score’ metrics based on key indicators correlated with automation success Calibrated the logic over 6 months by monitoring scores and customer cancellations Implemented Customer Success actions based on the health score ❶ ❷ ❸ ❹
8. • Total Score: 100 points • Green(Healthy) for 80 and above, Orange(Normal) for 50-79 points, Red(Unhealthy) for below 50 points • Calculated daily, weekly, monthly Health Score Details 35 pts Tests are run at least once a day (Num. of tests run in a week) / (Num. of tests x Num. of weekdays) 35 pts Tests are successful Num. of passed tests / Num. of executed tests 20 pts Tests are maintenable Num. of shared steps, percentage of overly long tests, etc. 10 pts Others Approximate breakdown
9. Does the Health Score measure test automation sucess? If the score improves from Red to Green, the cancellation rate is reduced to If the score improves from Red to Orange, the cancellation rate is reduced to 50% or less 20% or less
10. Does the Health Score measure test automation sucess? If the score improves from Red to Green, the cancellation rate is reduced to If the score improves from Red to Orange, the cancellation rate is reduced to 50% or less 20% or less The score measures cancellation rate (≈success of test automation) well
11. Users can check their Health Score
12. Agenda 1. Health score of End-toEnd Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score
13. • Health score quantifies ‘whether daily reliable feedbacks from tests are available and sustainable’ • Not ‘whether cost is cut’ nor ‘whether bugs are found’ 35 pts Tests are run at least once per day → Is the test feedback delivered in faster cycles? 35 pts Tests are successful → Is the test feedback reliable? 20 pts Tests are maintenable → Is the reliability of the test feedback sustainable? 10 pts Others Overview of Health Score Calculation Logic
14. Details of Health Score Calculation Logic Item Score 1. Sufficient number of test cases 6 pts 2. Sufficient members in the project 3 pts 3. Utilization of shared steps 8 pts 4. Avoid long tests 5 pts 5. Use stable locators 8 pts 6. Tests are run at least once a day 35 pts 7. Low test failure rate 35 pts
15. 1. Sufficient number of test cases (6 pts) To maximize the benefits of automated testing, sufficient number of test cases is essential • 20+ test cases (user stories) earn full points
16. 2. Sufficient members in the project (3 pts) • 4+ members earn full points Involvement of other members is important for long- term success
17. • Creating shared steps for 10-20% of total tests earn full points Shared steps (common functions) promote standardization and reduce maintenance efforts 3. Utilization of shared steps (8 pts)
18. Long tests make error isolation difficult and should be avoided when possible 4. Avoid long tests (5 pts) • Tests with over 200-300 steps are considered long • Keeping long tests within 10% of total tests earn full points
19. 5. Use stable locators (8 pts) • Using stable locators for at least 80% of total tests earn full points Complex locators are vulnerable to page structure changes
20. What locators are considered stable? Stable Unstable All other locators • Attributes or text that uniquely identify elements • ‘contains’, ‘starts-with’ are allowed • The same rules apply for CSS locators ・id and accessibility information ・#id CSS selector ・//xxx[yyy=‘zzz’] XPath For elements used in tests, we recommend assigning a unique id, accessibility information, attribute, or text within the application
21. Tests that aren't run daily become difficult to maintain 6. Tests are run at least once a day (35 pts) • Running all tests at least 4.2 days (average number of weekdays) per week earn full points • (Num. of tests run in a week) / (Num. of tests x Num. of weekdays)
22. 7. Low test failure rate (35 pts) High failure rate indicates lack of maintenance • Batch runs with a success rate of 90% or higher earn full points
23. Summary Item Score 1. Sufficient number of test cases 6 pts 2. Sufficient members in the project 3 pts 3. Utilization of shared steps 8 pts 4. Avoid long tests 5 pts 5. Use stable locators 8 pts 6. Tests are run at least once a day 35 pts 7. Low test failure rate 35 pts
24. Agenda 1. Health Score of End-toEnd Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score
25. Why is the Health Score low? 1.Project structure 2.Lack of involvement from developers 3.Over-reliance on End-to-End test automation Main reasons we found through user interviews
26. Development Test Type Deploy frequency Health score In-house In-house Fully In-house High Outsourced Outsourced Fully Outsourced Medium In-house Outsourced Third-party Testing Medium Outsourced In-house Automated UAT Low 1.Project structure Generally, fully “in-house” projects tend to achieve a high health score
27. Solution Development Test Type Health score Solution In-house In-house Fully In-house Run tests every day Outsourced Outsourced Fully Outsourced Ask the development company to run tests every day In-house Outsourced Third-party Testing Ask the test staffs to involve deeply as members of the team Outsourced In-house Automated UAT Ask the development company to automate and run tests
28. 2.Lack of involvement from developers • CI integration needs developer’s support • Developers can improve the testability of the product • The benefits of test automation cannot be maximized if it does not contribute to developer productivity Successful End-to-End test automation cannot be realized without the help of developers
29. Solution • Involve developers in End-to-End test automation • Run tests every day on the CI server • Improve developer productivity through faster feedbak • Improve testability of the product
30. 3.Over-reliance on End-to-End test automation Some people try to automate everything in the End-to-End test layer The project fails due to flakiness and slowness Automate all existing manual tests!!
31. Solution to maintain appropriate test pyramid Unit tests Integration tests End-to-End tests Involve developers in the planning of test automation
32. What is important for the success of End-to-End test automation? • Run tests continually to get faster feedback • Keep tests stable and reliable • Involve developers in End-to-End testing