$30 off During Our Annual Pro Sale. View Details »

How Do We Assess Our Tests to Measure and Monitor Their Reliability?

LINE DEVDAY 2021
November 11, 2021
110

How Do We Assess Our Tests to Measure and Monitor Their Reliability?

LINE DEVDAY 2021

November 11, 2021
Tweet

Transcript

  1. View Slide

  2. About me
    [email protected]
    Automation Engineer

    View Slide

  3. Agenda
    - Flaky test is an old story
    - What situation does LINE TODAY encounter
    - LINE TODAY test analysis dashboard
    - Conclusion

    View Slide

  4. Flaky test is an old story

    View Slide

  5. Flaky test is an old story
    Do you ever face flaky testing squarely?
    Fail
    Pass
    Pass
    Pass
    Fail

    View Slide

  6. The impact of flaky testing on cost
    - Execution cost – retry test, manual test
    - Fixing cost - debugging and fixing, dependencies (test suite or team
    infrastructure required)
    - Creation cost - this includes both the time and effort

    View Slide

  7. What situation does
    LINE TODAY encounter?

    View Slide

  8. How we did it
    Test retries
    We are able to retry
    failed tests to help
    reduce test flakiness
    Test doubles
    Use test doubles to
    enhance the quality of
    the automation, we use
    objects to reduce the
    complex testing
    Independency
    Test cases are
    independency so that
    defects can be spotted
    easily in isolation

    View Slide

  9. LINE TODAY automated test status
    Deployment / day
    3 - 6
    Failure rate
    1% - 3%
    Test cases
    700

    View Slide

  10. We wasted the time and cost
    If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day
    Failure cases
    21
    Rerun
    execution time (mins)
    40 - 60
    30 secs per case * 3 times
    * 21 cases

    View Slide

  11. LINE TODAY
    Test Analysis Dashboard

    View Slide

  12. Test Analysis Infrastructure
    Runner Reporter Custom parsing module
    Database
    QA Repo
    Dashboard

    View Slide

  13. View Slide

  14. View Slide

  15. How to calculate it
    A Internal [ Sum (count (probability of bad state)) ]
    [ A Internal ]
    * 100%

    View Slide

  16. How to calculate it
    Pass
    Case study 1
    2021-09-06 2021-09-10
    Pass Pass Pass Pass
    2021-09-07 2021-09-08 2021-09-09
    A Interval [ 0 ]
    [ 5 (day) ]
    * 100%
    3
    2
    1
    0 %
    Pass Pass

    View Slide

  17. How to calculate it
    Pass
    Fail
    Case study 2
    2021-09-06 2021-09-10
    Pass
    Fail
    Fail
    Pass
    Pass Pass
    2021-09-07 2021-09-08 2021-09-09
    3
    2
    1
    A Interval [ Sum (count (0.5, 0.5+0.25)) ]
    [ 5 (day) ]
    * 100%
    25 %
    0.5 0.5
    0.25

    View Slide

  18. How to calculate it
    Case study 3
    2021-09-06 2021-09-10
    Fail
    Fail
    Pass
    2021-09-07 2021-09-08 2021-09-09
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    3
    2
    1
    A Interval [ Sum (count (0.5+0.25+0.125)) * fourth]
    * 100%
    [ 5 (day) ]
    70 %
    0.5
    0.25
    0.125

    View Slide

  19. Efficiency
    Failure cases
    7
    Rerun
    execution time (mins)
    10 – 20
    30 secs per case * 3 times
    * 7 cases
    Rerun
    execution time (mins)
    1
    30 secs per case * 3 times
    * 1 cases

    View Slide

  20. Benefits
    - Decrease tracing time for finding out flaky cases
    - We know which case is a real bug and then fix it directly
    - Summary of findings by LINE TODAY
    - Reduce debug time and development costs
    - Getting real-time failure cases

    View Slide

  21. Conclusion
    - Smart test analysis dashboard
    - Focus on Exploratory testing , User-journey testing
    - Focus on new feature automation implementation

    View Slide

  22. Thank you

    View Slide