$30 off During Our Annual Pro Sale. View Details »

FrUITeR: A Framework for Evaluating UI Test Reuse

Yixue Zhao
November 02, 2020

FrUITeR: A Framework for Evaluating UI Test Reuse

Presentation slides of the paper "FrUITeR: A Framework for Evaluating UI Test Reuse" at the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2020.
Presentation link: https://youtu.be/zVWpT5aLyQo

Yixue Zhao

November 02, 2020
Tweet

More Decks by Yixue Zhao

Other Decks in Research

Transcript

  1. FrUITeR:
    A Framework for Evaluating UI Test Reuse
    Yixue Zhao1, Justin Chen2, Adriana Sejfia1, Marcelo Schmitt Laser1,
    Jie Zhang3, Federica Sarro3, Mark Harman3, Nenad Medvidović1
    ESEC/FSE 2020, Virtual Event
    1 2 3

    View Slide

  2. What’s UI Test Reuse?
    2
    ▪ New and exciting!! J
    ▪ Test generation technique
    ▪ “Usage-based” UI test
    ▪ Reuse existing tests
    ▪ UI similarities
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b2-1
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Source app Target app

    View Slide

  3. Why evaluation framework?
    3
    Ideal
    Happy Yixue in her proposal
    UI Test Reuse!

    View Slide

  4. Why evaluation framework?
    4
    Ideal
    Happy Yixue in her proposal
    UI Test Reuse!
    Reality
    Happier people submitting papers
    I’m done hehe!

    View Slide

  5. ▪ Limitations of existing work
    ▪ Which techniques are better
    ▪ Compare my new technique
    5
    FrUITeR was born
    A Framework for Evaluating UI Test Reuse

    View Slide

  6. So hard!
    ▪ Identify 5 key challenges
    ▪ Establish 5 requirements
    ▪ Design framework
    ▪ Build benchmark
    ▪ Migrate existing work
    6

    View Slide

  7. Challenges
    ▪ Evaluation metrics are different and limited
    ▪ Significant manual effort
    ▪ No guidelines for manual inspection
    ▪ One-off solutions evaluated as a whole
    ▪ Different benchmarks
    7

    View Slide

  8. Metrics
    Evaluation metrics are different and limited
    1.
    50% != 50% Accuracy != Good

    View Slide

  9. 9
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Etsy

    View Slide

  10. Manual
    Significant manual effort
    2.

    View Slide

  11. 11
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times

    View Slide

  12. 12
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times
    10 apps (100 pairs)
    10 tests w/ 10 events
    100 × 100 times!

    View Slide

  13. Guideline
    No guidelines for manual inspection
    3.
    Biased &
    Unreproducible

    View Slide

  14. 14
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Etsy
    a1-1 à b3-1???

    View Slide

  15. Address Challenge #1 Metrics
    15
    1. Evaluation metrics are different and limited

    View Slide

  16. Address Challenge #1 Metrics
    16
    1. Evaluation metrics are different and limited
    7 Fidelity Metrics

    View Slide

  17. Address Challenge #1 Metrics
    17
    1. Evaluation metrics are different and limited
    Effort = Levenshtein Distance (transEvents, gtEvents)
    2 Utility Metrics

    View Slide

  18. Address Challenges #2 & #3
    18
    2. Significant manual effort
    3. No guidelines for manual inspection

    View Slide

  19. 19

    View Slide

  20. 20
    Uniform
    Representation

    View Slide

  21. 21
    Uniform
    Representation

    View Slide

  22. 22

    View Slide

  23. 23
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b2-1
    a1 b1
    a1-1
    a1-2
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in
    Canonical
    Map

    View Slide

  24. 24
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in
    Address
    Challenge
    #2
    ONLY manual effort

    View Slide

  25. 25
    Wish Etsy
    Wish → Etsy
    1 test w/ 3 events
    3 times
    10 apps (100 pairs)
    10 tests w/ 10 events
    100 × 100 times!

    View Slide

  26. 26
    Address
    Challenge
    #3
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b
    a1 b1
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email
    b3-2 à password
    b3-3 à sign in

    View Slide

  27. 27
    a1 b1
    a1-1
    a1-2
    a1-3
    b2
    b1-1
    b
    a1 b1
    a1-3
    b2 b3
    b1-1
    b2-1
    b3-3
    b3-2
    b3-1
    Wish Canonical Map
    a1-1 à email
    a1-2 à password
    a1-3 à sign in
    Etsy Canonical Map
    b3-1 à email username
    b3-2 à password
    b3-3 à sign in
    Address
    Challenge
    #3

    View Slide

  28. More challenges in reality…
    28
    Contact
    authors
    Study
    implementation
    Modify/verify
    implementation
    Establish 239
    benchmark tests
    Construct 20 ground-
    truth Canonical Maps

    View Slide

  29. FrUITeR’s
    Empirical
    Results
    29
    ▪ 1,000 source-target app pairs (2
    app categories × 100 app pairs in
    each category × 5 techniques)
    ▪ 11,917 results entries
    ▪ 7 fidelity metrics
    ▪ 2 utility metrics

    View Slide

  30. FrUITeR’s
    Empirical
    Results
    30

    View Slide

  31. FrUITeR’s
    Empirical
    Results
    31
    https://felicitia.github.io/FrUITeR/

    View Slide

  32. FrUITeR’s
    Selected
    Implications
    32
    ▪ Perfect isn’t perfect
    (e.g., fidelity vs. utility)
    ▪ Source app selection
    (e.g., app company, code clone)
    ▪ Testing technique selection
    (e.g., trade-offs, manual vs. auto)

    View Slide

  33. Thanks!
    Any questions?
    [email protected]
    @yixue_zhao
    https://softarch.usc.edu/~yixue/

    View Slide