Avgust: Automating Usage-Based Test Generation from Videos of App Executions

Presentation slides of our paper "Avgust: Automating Usage-Based Test Generation from Videos of App Executions" at ESEC/FSE 2022.
Presentation: https://youtu.be/LB-TrLhQcvI

Transcript

1. AVGUST: Automating Usage-Based Test Generation from Videos of App Executions ESEC/FSE 2022, Singapore Yixue Zhao Saghar Talebipour, Kesina Baral, Hyojae Park, Leon Yee, Safwat Ali Khan, Yuriy Brun, Nenad Medvidović, Kevin Moran
2. ▪ 6.64 billion smartphone users worldwide (80%+) ▪ 230 billion downloads in 2021 worldwide ▪ Avg American spends over 5h/day on mobile devices ▪ Avg American checks phone 96 times/day, or once every 10min 2 Source: Statista, ZIPPIA
3. 3 App Developer Oh man, I need to write a sign-in test again… Behold! Do NOT fear! Have you tried AVGUST?
4. 4 App Developer AVGUST, I want a sign-in test for my app! Oh, and also search, add cart, …
5. 5 App Developer Tests! AVGUST There you go! AVGUST, I want a sign-in test for my app! Oh, and also search, add cart, …
6. 6 App Developer Yay thank you AVGUST!!!! What about deposit money for my banking app?
7. 7 App Developer Yay thank you AVGUST!!!! What about deposit money for my banking app? AVGUST Just give me similar videos and you shall receive! J
8. 8 UI Testing
9. 9 Existing Work ▪ Random Testing, e.g., Monkey ▪ Model-based Testing (MBT), e.g., Stoat ▪ ……
10. 10 Existing Work ▪ Random Testing, e.g., Monkey ▪ Model-based Testing (MBT), e.g., Stoat ▪ …… Goal Maximize code coverage
11. 11 Usage-based Test ▪ Test usage scenarios of an app (e.g., sign in, add item to the shopping cart) ▪ Highly preferred by developers ▪ Mimics realistic user behaviors
12. 12 Usage-based Test, How? ▪ Test Transfer □ GTM ISSTA 2018 □ ATM ASE 2019 □ CraftDroid ASE 2019 □ FrUITeR ESEC/FSE 2020 (our work J) □ MAPIT ASE 2021 (our work J) □ …
13. 13 Usage-based Test, How? ▪ Test Transfer □ GTM ISSTA 2018 □ ATM ASE 2019 □ CraftDroid ASE 2019 □ FrUITeR ESEC/FSE 2020 (our work J) □ MAPIT ASE 2021 (our work J) □ … Limitation Rely on existing tests □ Unavailable □ Low quality □ Too different
14. 14 AVGUST ▪ Developer-in-the-loop tool ▪ AVGUST = App-video-based generation of usage tests ▪ Only relies on videos □ Easy to get (e.g., public, crowdsourcing) □ Vison-only (pixel-based) □ Cross platform (any apps!)
15. 15 AVGUST Overview Videos Tests! Developer Assistance Models (per usage)
16. 16 Video Analysis Event Frames § Video processing § Action identification § Keyboard detection
17. 17 Model Generation (app-specific) Event Frames (app-independent) Model
18. 18 Model Generation (app-specific) Event Frames (app-independent) Model Image Classification!
19. 19 Model Generation
20. 20 Model Generation
21. 21 Model Generation
22. 22 Model Generation
23. ▪ Widget Classifier 23 Model Generation ▪ Screen Classifier “home” “about” “account” “cart” “search” …… “menu” “password” “add cart” “bookmark” “buy” …… 37 74
24. 24 Screen Features Screen Image Visual Textual Abstract GUI Screen OCR Text
25. 25 Widget Features Visual Features Textual Features OCR + BERT Widget Image Location: Top-Left Type: ImageButton Context “Home” Screen ResNet
26. 26 Model Generation
27. 27 Model Generation …… App1 AppN
28. 28 Test Generation Videos Tests! Developer Assistance IR Models
29. 29 Current State of Target App Top-K actions Test(s)! J Model (e.g., sign in) Image Features Textual Features IR Classifiers Developer
30. 30 AVGUST Evaluation ▪ 374 videos, 18 usages, 18 apps ▪ 51 generated tests for unseen apps □ 69% successful! □ 80% state precision (avg.) □ 70% state recall (avg.) Saves efforts!
31. 31 AVGUST Evaluation Screen IR Classifier’s Accuracy
32. 32 AVGUST Contributions ▪ First usage-based test generation based on videos ▪ Effective image classification (videos à formal models) ▪ Ready-to-use trained models
33. 33 AVGUST Contributions ▪ First usage-based test generation based on videos ▪ Effective image classification (videos à formal models) ▪ Ready-to-use trained models Crowd Workers Community Database Usage-based Tests, and more!
34. Thank You Saghar Kesina Hyojae Leon Safwat Yuriy Neno Kevin
35. Questions? Let’s connect! J ▪ Email: yzhao@isi.edu ▪ Twitter: @yixue_zhao ▪ LinkedIn: www.linkedin.com/in/yixue-zhao/ 35