How Do We Assess Our Tests to Measure and Monitor Their Reliability?

Sz-Lin Chen (Edward)
LINE Taiwan / LINE TODAY / Senior Automation Engineer

https://linedevday.linecorp.com/2021/ja/sessions/106
https://linedevday.linecorp.com/2021/en/sessions/106
https://linedevday.linecorp.com/2021/ko/sessions/106

Transcript

1. View Slide

2. About me
   edward.chen@linecorp.com
   Automation Engineer
   

   View Slide

3. Agenda
   - Flaky test is an old story
   - What situation does LINE TODAY encounter
   - LINE TODAY test analysis dashboard
   - Conclusion
   

   View Slide

4. Flaky test is an old story
   

   View Slide

5. Flaky test is an old story
   Do you ever face flaky testing squarely?
   Fail
   Pass
   Pass
   Pass
   Fail
   

   View Slide

6. The impact of flaky testing on cost
   - Execution cost – retry test, manual test
   - Fixing cost - debugging and fixing, dependencies (test suite or team
   infrastructure required)
   - Creation cost - this includes both the time and effort
   

   View Slide

7. What situation does
   LINE TODAY encounter?
   

   View Slide

8. How we did it
   Test retries
   We are able to retry
   failed tests to help
   reduce test flakiness
   Test doubles
   Use test doubles to
   enhance the quality of
   the automation, we use
   objects to reduce the
   complex testing
   Independency
   Test cases are
   independency so that
   defects can be spotted
   easily in isolation
   

   View Slide

9. LINE TODAY automated test status
   Deployment / day
   3 - 6
   Failure rate
   1% - 3%
   Test cases
   700
   

   View Slide

10. We wasted the time and cost
    If we got 3% failure rate, we will waste at least 40 mins to make sure which case is flakiness a day
    Failure cases
    21
    Rerun
    execution time (mins)
    40 - 60
    30 secs per case * 3 times
    * 21 cases
    

    View Slide

11. LINE TODAY
    Test Analysis Dashboard
    

    View Slide

12. Test Analysis Infrastructure
    Runner Reporter Custom parsing module
    Database
    QA Repo
    Dashboard
    

    View Slide

13. View Slide

14. View Slide

15. How to calculate it
    A Internal [ Sum (count (probability of bad state)) ]
    [ A Internal ]
    * 100%
    

    View Slide

16. How to calculate it
    Pass
    Case study 1
    2021-09-06 2021-09-10
    Pass Pass Pass Pass
    2021-09-07 2021-09-08 2021-09-09
    A Interval [ 0 ]
    [ 5 (day) ]
    * 100%
    3
    2
    1
    0 %
    Pass Pass
    

    View Slide

17. How to calculate it
    Pass
    Fail
    Case study 2
    2021-09-06 2021-09-10
    Pass
    Fail
    Fail
    Pass
    Pass Pass
    2021-09-07 2021-09-08 2021-09-09
    3
    2
    1
    A Interval [ Sum (count (0.5, 0.5+0.25)) ]
    [ 5 (day) ]
    * 100%
    25 %
    0.5 0.5
    0.25
    

    View Slide

18. How to calculate it
    Case study 3
    2021-09-06 2021-09-10
    Fail
    Fail
    Pass
    2021-09-07 2021-09-08 2021-09-09
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    Fail
    3
    2
    1
    A Interval [ Sum (count (0.5+0.25+0.125)) * fourth]
    * 100%
    [ 5 (day) ]
    70 %
    0.5
    0.25
    0.125
    

    View Slide

19. Efficiency
    Failure cases
    7
    Rerun
    execution time (mins)
    10 – 20
    30 secs per case * 3 times
    * 7 cases
    Rerun
    execution time (mins)
    1
    30 secs per case * 3 times
    * 1 cases
    

    View Slide

20. Benefits
    - Decrease tracing time for finding out flaky cases
    - We know which case is a real bug and then fix it directly
    - Summary of findings by LINE TODAY
    - Reduce debug time and development costs
    - Getting real-time failure cases
    

    View Slide

21. Conclusion
    - Smart test analysis dashboard
    - Focus on Exploratory testing , User-journey testing
    - Focus on new feature automation implementation
    

    View Slide

22. Thank you
    

    View Slide