LINE TECHPULSE 2022 - How Do We Assess Our Tests To Measure And Monitor Their Reliability

How Do We Assess Our Tests To Measure And Monitor Their Reliability by Edward Chen / LINE TODAY @ LINE TECHPULSE 2022 https://techpulse.line.me/

Transcript

1. Edward Chen / LINE TODAY How Do We Assess Our Tests To Measure And Monitor Their Reliability 
2. About me
3. Agenda › Flaky test is an old story › What situation does LINE TODAY encounter › LINE TODAY test analysis dashboard › Conclusion
4. Flaky test is an old story
5. Flaky test is an old story Do you ever face flaky testing squarely?
6. The impact of flaky testing on cost › Creation cost - this includes both the time and effort › Execution cost – retry test, manual test › Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required)
7. What situation does LINE TODAY encounter?
8. How we did it
9. LINE TODAY automated test status Deployment / day 3 - 6 700 1% - 3% Test cases Failure rate
10. We wasted the time and cost If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day 21 Rerun execution time (mins) 40 - 60 Failure cases 30 secs per case * 3 times * 21 cases
11. LINE TODAY Test Analysis Dashboard
12. Test Analysis Infrastructure
15. How to calculate it A Internal [ Sum (count (probability of bad state)) ] [ A Internal ] * 100%
16. How to calculate it Case study 1 0 %
17. How to calculate it Case study 2
18. How to calculate it Case study 3
19. Efficiency 10 – 20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 7 Rerun execution time (mins) 1 30 secs per case * 3 times * 7 cases Rerun execution time (mins)
20. Benefits › Getting real-time failure cases › Decrease tracing time for finding out flaky cases › Reduce debug time and development costs › Summary of findings by LINE TODAY
21. Conclusion › Smart test analysis dashboard › Focus on Exploratory testing , User-journey testing › Focus on new feature automation implementation
22. Thank you