Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Do We Assess Our Tests to Measure and Monitor Their Reliability?

LINE DEVDAY 2021
PRO
November 11, 2021
73

How Do We Assess Our Tests to Measure and Monitor Their Reliability?

LINE DEVDAY 2021
PRO

November 11, 2021
Tweet

Transcript

  1. None
  2. About me edward.chen@linecorp.com Automation Engineer

  3. Agenda - Flaky test is an old story - What

    situation does LINE TODAY encounter - LINE TODAY test analysis dashboard - Conclusion
  4. Flaky test is an old story

  5. Flaky test is an old story Do you ever face

    flaky testing squarely? Fail Pass Pass Pass Fail
  6. The impact of flaky testing on cost - Execution cost

    – retry test, manual test - Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required) - Creation cost - this includes both the time and effort
  7. What situation does LINE TODAY encounter?

  8. How we did it Test retries We are able to

    retry failed tests to help reduce test flakiness Test doubles Use test doubles to enhance the quality of the automation, we use objects to reduce the complex testing Independency Test cases are independency so that defects can be spotted easily in isolation
  9. LINE TODAY automated test status Deployment / day 3 -

    6 Failure rate 1% - 3% Test cases 700
  10. We wasted the time and cost If we got 3%

    failure rate, we will waste at least 40 mins to make sure which case is flakiness a day Failure cases 21 Rerun execution time (mins) 40 - 60 30 secs per case * 3 times * 21 cases
  11. LINE TODAY Test Analysis Dashboard

  12. Test Analysis Infrastructure Runner Reporter Custom parsing module Database QA

    Repo Dashboard
  13. None
  14. None
  15. How to calculate it A Internal [ Sum (count (probability

    of bad state)) ] [ A Internal ] * 100%
  16. How to calculate it Pass Case study 1 2021-09-06 2021-09-10

    Pass Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 A Interval [ 0 ] [ 5 (day) ] * 100% 3 2 1 0 % Pass Pass
  17. How to calculate it Pass Fail Case study 2 2021-09-06

    2021-09-10 Pass Fail Fail Pass Pass Pass 2021-09-07 2021-09-08 2021-09-09 3 2 1 A Interval [ Sum (count (0.5, 0.5+0.25)) ] [ 5 (day) ] * 100% 25 % 0.5 0.5 0.25
  18. How to calculate it Case study 3 2021-09-06 2021-09-10 Fail

    Fail Pass 2021-09-07 2021-09-08 2021-09-09 Fail Fail Fail Fail Fail Fail Fail Fail Fail Fail 3 2 1 A Interval [ Sum (count (0.5+0.25+0.125)) * fourth] * 100% [ 5 (day) ] 70 % 0.5 0.25 0.125
  19. Efficiency Failure cases 7 Rerun execution time (mins) 10 –

    20 30 secs per case * 3 times * 7 cases Rerun execution time (mins) 1 30 secs per case * 3 times * 1 cases
  20. Benefits - Decrease tracing time for finding out flaky cases

    - We know which case is a real bug and then fix it directly - Summary of findings by LINE TODAY - Reduce debug time and development costs - Getting real-time failure cases
  21. Conclusion - Smart test analysis dashboard - Focus on Exploratory

    testing , User-journey testing - Focus on new feature automation implementation
  22. Thank you