Slide 1

Slide 1 text

Quantifying the Health of End-to-End Test Automation Nozomi Ito

Slide 2

Slide 2 text

• CEO of MagicPod • MagicPod: a no-code, cloud-based test automation tool • Organizer of the Japan Selenium User Community • Author of a Japanese Selenium book About me Nozomi Ito

Slide 3

Slide 3 text

Agenda 1. Health Score of End-to-End Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score

Slide 4

Slide 4 text

Agenda 1. Health Score of End-to-End Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score

Slide 5

Slide 5 text

• By reduced cost? • By number of bugs found? • By DORA Metrics? ⚬ Deployment Frequency, Lead Time, Change Failure Rate, MTTR • By business growth? • By profit margin? How do we quantify the success of End-to-End test automation?

Slide 6

Slide 6 text

Continue Results justify the cost Cancel For MagicPod Sucess of the customer’s test automation They continue to use the product without cancellation Results do not justify the cost

Slide 7

Slide 7 text

Interviewed over 100 MagicPod customers (including those who cancelled) Measuring Success with the ‘Health Score’ Defined ‘health score’ metrics based on key indicators correlated with automation success Calibrated the logic over 6 months by monitoring scores and customer cancellations Implemented Customer Success actions based on the health score ❶ ❷ ❸ ❹

Slide 8

Slide 8 text

• Total Score: 100 points • Green(Healthy) for 80 and above, Orange(Normal) for 50-79 points, Red(Unhealthy) for below 50 points • Calculated daily, weekly, monthly Health Score Details 35 pts Tests are run at least once a day (Num. of tests run in a week) / (Num. of tests x Num. of weekdays) 35 pts Tests are successful Num. of passed tests / Num. of executed tests 20 pts Tests are maintenable Num. of shared steps, percentage of overly long tests, etc. 10 pts Others Approximate breakdown

Slide 9

Slide 9 text

Does the Health Score measure test automation sucess? If the score improves from Red to Green, the cancellation rate is reduced to If the score improves from Red to Orange, the cancellation rate is reduced to 50% or less 20% or less

Slide 10

Slide 10 text

Does the Health Score measure test automation sucess? If the score improves from Red to Green, the cancellation rate is reduced to If the score improves from Red to Orange, the cancellation rate is reduced to 50% or less 20% or less The score measures cancellation rate (≈success of test automation) well

Slide 11

Slide 11 text

Users can check their Health Score

Slide 12

Slide 12 text

Agenda 1. Health score of End-toEnd Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score

Slide 13

Slide 13 text

• Health score quantifies ‘whether daily reliable feedbacks from tests are available and sustainable’ • Not ‘whether cost is cut’ nor ‘whether bugs are found’ 35 pts Tests are run at least once per day → Is the test feedback delivered in faster cycles? 35 pts Tests are successful → Is the test feedback reliable? 20 pts Tests are maintenable → Is the reliability of the test feedback sustainable? 10 pts Others Overview of Health Score Calculation Logic

Slide 14

Slide 14 text

Details of Health Score Calculation Logic Item Score 1. Sufficient number of test cases 6 pts 2. Sufficient members in the project 3 pts 3. Utilization of shared steps 8 pts 4. Avoid long tests 5 pts 5. Use stable locators 8 pts 6. Tests are run at least once a day 35 pts 7. Low test failure rate 35 pts

Slide 15

Slide 15 text

1. Sufficient number of test cases (6 pts) To maximize the benefits of automated testing, sufficient number of test cases is essential • 20+ test cases (user stories) earn full points

Slide 16

Slide 16 text

2. Sufficient members in the project (3 pts) • 4+ members earn full points Involvement of other members is important for long- term success

Slide 17

Slide 17 text

• Creating shared steps for 10-20% of total tests earn full points Shared steps (common functions) promote standardization and reduce maintenance efforts 3. Utilization of shared steps (8 pts)

Slide 18

Slide 18 text

Long tests make error isolation difficult and should be avoided when possible 4. Avoid long tests (5 pts) • Tests with over 200-300 steps are considered long • Keeping long tests within 10% of total tests earn full points

Slide 19

Slide 19 text

5. Use stable locators (8 pts) • Using stable locators for at least 80% of total tests earn full points Complex locators are vulnerable to page structure changes

Slide 20

Slide 20 text

What locators are considered stable? Stable Unstable All other locators • Attributes or text that uniquely identify elements • ‘contains’, ‘starts-with’ are allowed • The same rules apply for CSS locators ・id and accessibility information ・#id CSS selector ・//xxx[yyy=‘zzz’] XPath For elements used in tests, we recommend assigning a unique id, accessibility information, attribute, or text within the application

Slide 21

Slide 21 text

Tests that aren't run daily become difficult to maintain 6. Tests are run at least once a day (35 pts) • Running all tests at least 4.2 days (average number of weekdays) per week earn full points • (Num. of tests run in a week) / (Num. of tests x Num. of weekdays)

Slide 22

Slide 22 text

7. Low test failure rate (35 pts) High failure rate indicates lack of maintenance • Batch runs with a success rate of 90% or higher earn full points

Slide 23

Slide 23 text

Summary Item Score 1. Sufficient number of test cases 6 pts 2. Sufficient members in the project 3 pts 3. Utilization of shared steps 8 pts 4. Avoid long tests 5 pts 5. Use stable locators 8 pts 6. Tests are run at least once a day 35 pts 7. Low test failure rate 35 pts

Slide 24

Slide 24 text

Agenda 1. Health Score of End-toEnd Test Automation 2. Breakdown of the Health Score 3. Improving the Health Score

Slide 25

Slide 25 text

Why is the Health Score low? 1.Project structure 2.Lack of involvement from developers 3.Over-reliance on End-to-End test automation Main reasons we found through user interviews

Slide 26

Slide 26 text

Development Test Type Deploy frequency Health score In-house In-house Fully In-house High Outsourced Outsourced Fully Outsourced Medium In-house Outsourced Third-party Testing Medium Outsourced In-house Automated UAT Low 1.Project structure Generally, fully “in-house” projects tend to achieve a high health score

Slide 27

Slide 27 text

Solution Development Test Type Health score Solution In-house In-house Fully In-house Run tests every day Outsourced Outsourced Fully Outsourced Ask the development company to run tests every day In-house Outsourced Third-party Testing Ask the test staffs to involve deeply as members of the team Outsourced In-house Automated UAT Ask the development company to automate and run tests

Slide 28

Slide 28 text

2.Lack of involvement from developers • CI integration needs developer’s support • Developers can improve the testability of the product • The benefits of test automation cannot be maximized if it does not contribute to developer productivity Successful End-to-End test automation cannot be realized without the help of developers

Slide 29

Slide 29 text

Solution • Involve developers in End-to-End test automation • Run tests every day on the CI server • Improve developer productivity through faster feedbak • Improve testability of the product

Slide 30

Slide 30 text

3.Over-reliance on End-to-End test automation Some people try to automate everything in the End-to-End test layer The project fails due to flakiness and slowness Automate all existing manual tests!!

Slide 31

Slide 31 text

Solution to maintain appropriate test pyramid Unit tests Integration tests End-to-End tests Involve developers in the planning of test automation

Slide 32

Slide 32 text

What is important for the success of End-to-End test automation? • Run tests continually to get faster feedback • Keep tests stable and reliable • Involve developers in End-to-End testing

Slide 33

Slide 33 text

No content