How Do We Assess Our Tests to Measure and Monitor Their Reliability?

Sz-Lin Chen (Edward)
LINE Taiwan / LINE TODAY / Senior Automation Engineer

https://linedevday.linecorp.com/2021/ja/sessions/106
https://linedevday.linecorp.com/2021/en/sessions/106
https://linedevday.linecorp.com/2021/ko/sessions/106

Transcript

2. About me edward.chen@linecorp.com Automation Engineer
3. Agenda - Flaky test is an old story - What situation does LINE TODAY encounter - LINE TODAY test analysis dashboard - Conclusion
4. Flaky test is an old story
5. Flaky test is an old story Do you ever face flaky testing squarely? Fail Pass Pass Pass Fail
6. The impact of flaky testing on cost - Execution cost – retry test, manual test - Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required) - Creation cost - this includes both the time and effort
7. What situation does LINE TODAY encounter?
8. How we did it Test retries We are able to retry failed tests to help reduce test flakiness Test doubles Use test doubles to enhance the quality of the automation, we use objects to reduce the complex testing Independency Test cases are independency so that defects can be spotted easily in isolation
9. LINE TODAY automated test status Deployment / day 3 - 6 Failure rate 1% - 3% Test cases 700
10. We wasted the time and cost If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day Failure cases 21 Rerun execution time (mins) 40 - 60 30 secs per case * 3 times * 21 cases
11. LINE TODAY Test Analysis Dashboard
12. Test Analysis Infrastructure Runner Reporter Custom parsing module Database QA Repo Dashboard
15. How to calculate it A Internal [ Sum (count (probability of bad state)) ] [ A Internal ] * 100%
16. How to calculate it Pass Case study 1 2021-09-06 2021-09-10 Pass Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 A Interval [ 0 ] [ 5 (day) ] * 100% 3 2 1 0 % Pass Pass
17. How to calculate it Pass Fail Case study 2 2021-09-06 2021-09-10 Pass Fail Fail Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 3 2 1 A Interval [ Sum (count (0.5, 0.5+0.25)) ] [ 5 (day) ] * 100% 25 % 0.5 0.5 0.25
18. How to calculate it Case study 3 2021-09-06 2021-09-10 Fail Fail Pass 2021-09-07 2021-09-08 2021-09-09 Fail Fail Fail Fail Fail Fail Fail Fail Fail Fail 3 2 1 A Interval [ Sum (count (0.5+0.25+0.125)) * fourth] * 100% [ 5 (day) ] 70 % 0.5 0.25 0.125
19. Efficiency Failure cases 7 Rerun execution time (mins) 10 – 20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 1 30 secs per case * 3 times * 1 cases
20. Benefits - Decrease tracing time for finding out flaky cases - We know which case is a real bug and then fix it directly - Summary of findings by LINE TODAY - Reduce debug time and development costs - Getting real-time failure cases
21. Conclusion - Smart test analysis dashboard - Focus on Exploratory testing , User-journey testing - Focus on new feature automation implementation
22. Thank you