FrUITeR: A Framework for Evaluating UI Test Reuse

Presentation slides of the paper "FrUITeR: A Framework for Evaluating UI Test Reuse" at the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2020.
Presentation link: https://youtu.be/zVWpT5aLyQo

Transcript

1. FrUITeR:
   A Framework for Evaluating UI Test Reuse
   Yixue Zhao1, Justin Chen2, Adriana Sejfia1, Marcelo Schmitt Laser1,
   Jie Zhang3, Federica Sarro3, Mark Harman3, Nenad Medvidović1
   ESEC/FSE 2020, Virtual Event
   1 2 3
   

   View Slide

2. What’s UI Test Reuse?
   2
   ▪ New and exciting!! J
   ▪ Test generation technique
   ▪ “Usage-based” UI test
   ▪ Reuse existing tests
   ▪ UI similarities
   a1 b1
   a1-1
   a1-2
   a1-3
   b2
   b1-1
   b2-1
   a1 b1
   a1-1
   a1-2
   a1-3
   b2 b3
   b1-1
   b2-1
   b3-3
   b3-2
   b3-1
   Source app Target app
   

   View Slide

3. Why evaluation framework?
   3
   Ideal
   Happy Yixue in her proposal
   UI Test Reuse!
   

   View Slide

4. Why evaluation framework?
   4
   Ideal
   Happy Yixue in her proposal
   UI Test Reuse!
   Reality
   Happier people submitting papers
   I’m done hehe!
   

   View Slide

5. ▪ Limitations of existing work
   ▪ Which techniques are better
   ▪ Compare my new technique
   5
   FrUITeR was born
   A Framework for Evaluating UI Test Reuse
   

   View Slide

6. So hard!
   ▪ Identify 5 key challenges
   ▪ Establish 5 requirements
   ▪ Design framework
   ▪ Build benchmark
   ▪ Migrate existing work
   6
   

   View Slide

7. Challenges
   ▪ Evaluation metrics are different and limited
   ▪ Significant manual effort
   ▪ No guidelines for manual inspection
   ▪ One-off solutions evaluated as a whole
   ▪ Different benchmarks
   7
   

   View Slide

8. Metrics
   Evaluation metrics are different and limited
   1.
   50% != 50% Accuracy != Good
   

   View Slide

9. 9
   a1 b1
   a1-1
   a1-2
   a1-3
   b2 b3
   b1-1
   b2-1
   b3-3
   b3-2
   b3-1
   Wish Etsy
   

   View Slide

10. Manual
    Significant manual effort
    2.
    

    View Slide

11. 11
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times
    

    View Slide

12. 12
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times
    10 apps (100 pairs)
    10 tests w/ 10 events
    100 × 100 times!
    

    View Slide

13. Guideline
    No guidelines for manual inspection
    3.
    Biased &
    Unreproducible
    

    View Slide

14. 14
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Etsy
    a1-1 à b3-1???
    

    View Slide

15. Address Challenge #1 Metrics
    15
    1. Evaluation metrics are different and limited
    

    View Slide

16. Address Challenge #1 Metrics
    16
    1. Evaluation metrics are different and limited
    7 Fidelity Metrics
    

    View Slide

17. Address Challenge #1 Metrics
    17
    1. Evaluation metrics are different and limited
    Effort = Levenshtein Distance (transEvents, gtEvents)
    2 Utility Metrics
    

    View Slide

18. Address Challenges #2 & #3
    18
    2. Significant manual effort
    3. No guidelines for manual inspection
    

    View Slide

19. View Slide

20. 20
    Uniform
    Representation
    

    View Slide

21. 21
    Uniform
    Representation
    

    View Slide

22. View Slide

23. 23
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b2-1
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in
    Canonical
    Map
    

    View Slide

24. 24
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in
    Address
    Challenge
    #2
    ONLY manual effort
    

    View Slide

25. 25
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times
    10 apps (100 pairs)
    10 tests w/ 10 events
    100 × 100 times!
    

    View Slide

26. 26
    Address
    Challenge
    #3
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b
    a1 b1
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in
    

    View Slide

27. 27
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b
    a1 b1
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email username
    b3-2 à password
    b3-3 à sign in
    Address
    Challenge
    #3
    

    View Slide

28. More challenges in reality…
    28
    Contact
    authors
    Study
    implementation
    Modify/verify
    implementation
    Establish 239
    benchmark tests
    Construct 20 ground-
    truth Canonical Maps
    

    View Slide

29. FrUITeR’s
    Empirical
    Results
    29
    ▪ 1,000 source-target app pairs (2
    app categories × 100 app pairs in
    each category × 5 techniques)
    ▪ 11,917 results entries
    ▪ 7 fidelity metrics
    ▪ 2 utility metrics
    

    View Slide

30. FrUITeR’s
    Empirical
    Results
    30
    

    View Slide

31. FrUITeR’s
    Empirical
    Results
    31
    https://felicitia.github.io/FrUITeR/
    

    View Slide

32. FrUITeR’s
    Selected
    Implications
    32
    ▪ Perfect isn’t perfect
    (e.g., fidelity vs. utility)
    ▪ Source app selection
    (e.g., app company, code clone)
    ▪ Testing technique selection
    (e.g., trade-offs, manual vs. auto)
    

    View Slide

33. Thanks!
    Any questions?
    yixue.zhao@usc.edu
    @yixue_zhao
    https://softarch.usc.edu/~yixue/
    

    View Slide