Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE TECHPULSE 2022 - How Do We Assess Our Tests To Measure And Monitor Their Reliability

LINE TECHPULSE 2022 - How Do We Assess Our Tests To Measure And Monitor Their Reliability

How Do We Assess Our Tests To Measure And Monitor Their Reliability by Edward Chen / LINE TODAY @ LINE TECHPULSE 2022 https://techpulse.line.me/

2102a6b8760bd6f57f672805723dd83a?s=128

LINE Developers Taiwan
PRO

January 21, 2022
Tweet

More Decks by LINE Developers Taiwan

Other Decks in Programming

Transcript

  1. Edward Chen / LINE TODAY How Do We Assess Our

    Tests To Measure And Monitor Their Reliability 
  2. About me

  3. Agenda › Flaky test is an old story › What

    situation does LINE TODAY encounter › LINE TODAY test analysis dashboard › Conclusion
  4. Flaky test is an old story

  5. Flaky test is an old story Do you ever face

    flaky testing squarely?
  6. The impact of flaky testing on cost › Creation cost

    - this includes both the time and effort › Execution cost – retry test, manual test › Fixing cost - debugging and fixing, dependencies (test suite or team infrastructure required)
  7. What situation does LINE TODAY encounter?

  8. How we did it

  9. LINE TODAY automated test status Deployment / day 3 -

    6 700 1% - 3% Test cases Failure rate
  10. We wasted the time and cost If we got 3%

    failure rate, we will waste at least 40 mins to make sure which case is flakiness a day 21 Rerun execution time (mins) 40 - 60 Failure cases 30 secs per case * 3 times * 21 cases
  11. LINE TODAY Test Analysis Dashboard

  12. Test Analysis Infrastructure

  13. None
  14. None
  15. How to calculate it A Internal [ Sum (count (probability

    of bad state)) ] [ A Internal ] * 100%
  16. How to calculate it Case study 1 0 %

  17. How to calculate it Case study 2

  18. How to calculate it Case study 3

  19. Efficiency 10 – 20 30 secs per case * 3

    times * 7 cases Rerun execution time (mins) 7 Rerun execution time (mins) 1 30 secs per case * 3 times * 7 cases Rerun execution time (mins)
  20. Benefits › Getting real-time failure cases › Decrease tracing time

    for finding out flaky cases › Reduce debug time and development costs › Summary of findings by LINE TODAY
  21. Conclusion › Smart test analysis dashboard › Focus on Exploratory

    testing , User-journey testing › Focus on new feature automation implementation
  22. Thank you