Slide 1
Slide 1 text
Edward Chen / LINE TODAY
How Do We Assess Our Tests To
Measure And Monitor Their Reliability
Slide 2
Slide 2 text
About me
Slide 3
Slide 3 text
Agenda
› Flaky test is an old story
› What situation does LINE TODAY encounter
› LINE TODAY test analysis dashboard
› Conclusion
Slide 4
Slide 4 text
Flaky test is an old story
Slide 5
Slide 5 text
Flaky test is an old story
Do you ever face flaky testing squarely?
Slide 6
Slide 6 text
The impact of flaky testing on cost
› Creation cost - this includes both the time and effort
› Execution cost – retry test, manual test
› Fixing cost - debugging and fixing, dependencies (test suite or team
infrastructure required)
Slide 7
Slide 7 text
What situation does
LINE TODAY encounter?
Slide 8
Slide 8 text
How we did it
Slide 9
Slide 9 text
LINE TODAY automated test status
Deployment / day
3 - 6 700 1% - 3%
Test cases Failure rate
Slide 10
Slide 10 text
We wasted the time and cost
If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day
21
Rerun
execution time (mins)
40 - 60
Failure cases
30 secs per case
* 3 times * 21 cases
Slide 11
Slide 11 text
LINE TODAY
Test Analysis Dashboard
Slide 12
Slide 12 text
Test Analysis Infrastructure
Slide 13
Slide 13 text
No content
Slide 14
Slide 14 text
No content
Slide 15
Slide 15 text
How to calculate it
A Internal [ Sum (count (probability of bad state)) ]
[ A Internal ] * 100%
Slide 16
Slide 16 text
How to calculate it
Case study 1 0 %
Slide 17
Slide 17 text
How to calculate it
Case study 2
Slide 18
Slide 18 text
How to calculate it
Case study 3
Slide 19
Slide 19 text
Efficiency
10 – 20
30 secs per case * 3 times
* 7 cases
Rerun
execution time (mins)
7
Rerun
execution time (mins)
1
30 secs per case * 3 times
* 7 cases
Rerun
execution time (mins)
Slide 20
Slide 20 text
Benefits
› Getting real-time failure cases
› Decrease tracing time for finding out flaky cases
› Reduce debug time and development costs
› Summary of findings by LINE TODAY
Slide 21
Slide 21 text
Conclusion
› Smart test analysis dashboard
› Focus on Exploratory testing , User-journey testing
› Focus on new feature automation implementation
Slide 22
Slide 22 text
Thank you