Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

About me [email protected] Automation Engineer

Slide 3

Slide 3 text

Agenda - Flaky test is an old story - What situation does LINE TODAY encounter - LINE TODAY test analysis dashboard - Conclusion

Slide 4

Slide 4 text

Flaky test is an old story

Slide 5

Slide 5 text

Flaky test is an old story Do you ever face flaky testing squarely? Fail Pass Pass Pass Fail

Slide 6

Slide 6 text

The impact of flaky testing on cost - Execution cost – retry test, manual test - Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required) - Creation cost - this includes both the time and effort

Slide 7

Slide 7 text

What situation does LINE TODAY encounter?

Slide 8

Slide 8 text

How we did it Test retries We are able to retry failed tests to help reduce test flakiness Test doubles Use test doubles to enhance the quality of the automation, we use objects to reduce the complex testing Independency Test cases are independency so that defects can be spotted easily in isolation

Slide 9

Slide 9 text

LINE TODAY automated test status Deployment / day 3 - 6 Failure rate 1% - 3% Test cases 700

Slide 10

Slide 10 text

We wasted the time and cost If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day Failure cases 21 Rerun execution time (mins) 40 - 60 30 secs per case * 3 times * 21 cases

Slide 11

Slide 11 text

LINE TODAY Test Analysis Dashboard

Slide 12

Slide 12 text

Test Analysis Infrastructure Runner Reporter Custom parsing module Database QA Repo Dashboard

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

How to calculate it A Internal [ Sum (count (probability of bad state)) ] [ A Internal ] * 100%

Slide 16

Slide 16 text

How to calculate it Pass Case study 1 2021-09-06 2021-09-10 Pass Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 A Interval [ 0 ] [ 5 (day) ] * 100% 3 2 1 0 % Pass Pass

Slide 17

Slide 17 text

How to calculate it Pass Fail Case study 2 2021-09-06 2021-09-10 Pass Fail Fail Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 3 2 1 A Interval [ Sum (count (0.5, 0.5+0.25)) ] [ 5 (day) ] * 100% 25 % 0.5 0.5 0.25

Slide 18

Slide 18 text

How to calculate it Case study 3 2021-09-06 2021-09-10 Fail Fail Pass 2021-09-07 2021-09-08 2021-09-09 Fail Fail Fail Fail Fail Fail Fail Fail Fail Fail 3 2 1 A Interval [ Sum (count (0.5+0.25+0.125)) * fourth] * 100% [ 5 (day) ] 70 % 0.5 0.25 0.125

Slide 19

Slide 19 text

Efficiency Failure cases 7 Rerun execution time (mins) 10 – 20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 1 30 secs per case * 3 times * 1 cases

Slide 20

Slide 20 text

Benefits - Decrease tracing time for finding out flaky cases - We know which case is a real bug and then fix it directly - Summary of findings by LINE TODAY - Reduce debug time and development costs - Getting real-time failure cases

Slide 21

Slide 21 text

Conclusion - Smart test analysis dashboard - Focus on Exploratory testing , User-journey testing - Focus on new feature automation implementation

Slide 22

Slide 22 text

Thank you