No content

About me edward.chen@linecorp.com Automation Engineer

Agenda - Flaky test is an old story - What situation does LINE TODAY encounter - LINE TODAY test analysis dashboard - Conclusion

Flaky test is an old story

Flaky test is an old story Do you ever face flaky testing squarely? Fail Pass Pass Pass Fail

The impact of flaky testing on cost - Execution cost – retry test, manual test - Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required) - Creation cost - this includes both the time and effort

What situation does LINE TODAY encounter?

How we did it Test retries We are able to retry failed tests to help reduce test flakiness Test doubles Use test doubles to enhance the quality of the automation, we use objects to reduce the complex testing Independency Test cases are independency so that defects can be spotted easily in isolation

LINE TODAY automated test status Deployment / day 3 - 6 Failure rate 1% - 3% Test cases 700

We wasted the time and cost If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day Failure cases 21 Rerun execution time (mins) 40 - 60 30 secs per case * 3 times * 21 cases

LINE TODAY Test Analysis Dashboard

Test Analysis Infrastructure Runner Reporter Custom parsing module Database QA Repo Dashboard

No content

No content

How to calculate it A Internal [ Sum (count (probability of bad state)) ] [ A Internal ] * 100%

How to calculate it Pass Case study 1 2021-09-06 2021-09-10 Pass Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 A Interval [ 0 ] [ 5 (day) ] * 100% 3 2 1 0 % Pass Pass

How to calculate it Pass Fail Case study 2 2021-09-06 2021-09-10 Pass Fail Fail Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 3 2 1 A Interval [ Sum (count (0.5, 0.5+0.25)) ] [ 5 (day) ] * 100% 25 % 0.5 0.5 0.25

How to calculate it Case study 3 2021-09-06 2021-09-10 Fail Fail Pass 2021-09-07 2021-09-08 2021-09-09 Fail Fail Fail Fail Fail Fail Fail Fail Fail Fail 3 2 1 A Interval [ Sum (count (0.5+0.25+0.125)) * fourth] * 100% [ 5 (day) ] 70 % 0.5 0.25 0.125

Efficiency Failure cases 7 Rerun execution time (mins) 10 – 20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 1 30 secs per case * 3 times * 1 cases

Benefits - Decrease tracing time for finding out flaky cases - We know which case is a real bug and then fix it directly - Summary of findings by LINE TODAY - Reduce debug time and development costs - Getting real-time failure cases

Conclusion - Smart test analysis dashboard - Focus on Exploratory testing , User-journey testing - Focus on new feature automation implementation

Thank you