The impact of flaky testing on cost - Execution cost – retry test, manual test - Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required) - Creation cost - this includes both the time and effort
How we did it Test retries We are able to retry failed tests to help reduce test flakiness Test doubles Use test doubles to enhance the quality of the automation, we use objects to reduce the complex testing Independency Test cases are independency so that defects can be spotted easily in isolation
We wasted the time and cost If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day Failure cases 21 Rerun execution time (mins) 40 - 60 30 secs per case * 3 times * 21 cases
Efficiency Failure cases 7 Rerun execution time (mins) 10 – 20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 1 30 secs per case * 3 times * 1 cases
Benefits - Decrease tracing time for finding out flaky cases - We know which case is a real bug and then fix it directly - Summary of findings by LINE TODAY - Reduce debug time and development costs - Getting real-time failure cases