Android Application Testing with Firebase Robo Test

It's very common for Android app developers to write unit and instrumentation tests around their app. However, some bugs may still occur when the end users are using the app. Some bugs are quite hard to catch because it only happens under specific conditions that does not happen if the test is run on mock or fake data. In this talk I showed how you can use Firebase Robo Test to automate the UI testing of an Android app which allows you to test your app from end to end without using any mocking mechanism.

Transcript

1. ©2019 Wantedly, Inc. Tokyo Android Meetup Application Testing With Firebase Robo Test July 23, 2019 - Malvin Sutanto Photo by Steve Jurvetson on Flickr
2. ©2019 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android, Kotlin, and sometimes iOS. Have been developing Android App since 2013. Twitter/ Medium: @malvinsutanto
3. ©2019 Wantedly, Inc. Android Application Testing
4. ©2019 Wantedly, Inc. Unit tests /test folder and runs on JVM Testing functions Instrumentation tests /androidTest folder runs on emulator or physical device Testing UI elements and screens independently Android Application Testing
5. ©2019 Wantedly, Inc. Works perfectly But, your users may still encounter bugs. Page Title Page Subtitle
6. ©2019 Wantedly, Inc. Firebase Robo Test
7. ©2019 Wantedly, Inc. UI testing tool Integrated with Firebase Test Lab Analyze app’s UI structure Simulate real user interactions with the app Firebase Robo Test What is it?
8. ©2019 Wantedly, Inc. 1. Tests the app as a whole Flow from screen to screen 2. Tests against real API No mocking mechanisms 3. Tests obfuscated code apk or aab Firebase Robo Test Benefits
9. ©2019 Wantedly, Inc. 4. Supports Firebase Test Matrix Tests against multiple devices at once 5. Provides performance reports Only if it is run against physical device Firebase Robo Test Benefits - cont’d
10. ©2019 Wantedly, Inc. Setup Robo Test
11. ©2019 Wantedly, Inc. Firebase Test Lab console Setup Robo Test Gcloud command line tool
12. ©2019 Wantedly, Inc. Firebase Test Lab console Web Interface Accessible from the console’s sidebar: Test Lab | Run a test | Run a Robo test Setup Robo test parameters and uploads your app through the web interface. You can choose multiple device configurations, and specify additional options e.g., test credentials or deep links URL Setup Robo Test
13. ©2019 Wantedly, Inc. Gcloud command line tool Command Line Interface Can be run from your CI/CD infrastructure. Needs to have gcloud command line tools preinstalled. Setup guide: https://cloud.google.com/sdk/ gcloud/ Setup Robo Test $ gcloud beta firebase test android run \ !--type robo \ !--timeout 3m \ !--device model=walleye,version=28,locale=en_US, orientation=portrait \ !--app /path/to/apk_file.apk
14. ©2019 Wantedly, Inc. Setup with Yaml file Setup Robo Test $ gcloud beta firebase test android run \ /path/to/robo.yml:args Gcloud command args: type: robo timeout: 3m device: - model: walleye version: 28 locale: en_US orientation: portrait - model: walleye version: 27 locale: ja_JP orientation: portrait app: app/build/outputs/apk/debug/app-debug.apk
15. ©2019 Wantedly, Inc. Controlling Robo Test Robo Script
16. ©2019 Wantedly, Inc. JSON file to guide Robo test Contains an array of actions Actions are executed sequentially. Continue with regular robo crawl after all actions has been performed. Robo Script What is it?
17. ©2019 Wantedly, Inc. Used for Google Play Console’s pre-launch reports Created using Android Studio Robo Script What is it? - cont’d
18. ©2019 Wantedly, Inc. To Record a Robo script Tools | Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test This will trigger a build and deploy Interaction with the app will be recorded Robo Script Record from Android Studio
19. ©2019 Wantedly, Inc. Use physical device! Recording can be very slow, especially when it is run on the emulator. Page Title Page Subtitle
20. ©2019 Wantedly, Inc. Example Robo Script Example
21. ©2019 Wantedly, Inc. 1. Limited interactions Does not support gestures, or back button 2. Unable to specify multiple scripts Each test will require a separate script Robo Script Limitations
22. ©2019 Wantedly, Inc. Specifying robo script Robo Script Gcloud command line $ gcloud beta firebase test android run \ !!... !--robo-script /path/to/roboscript.json Yaml file args: !!... robo-script: /path/to/roboscript.json
23. ©2019 Wantedly, Inc. Controlling Robo Test Robo Directives
24. ©2019 Wantedly, Inc. Set of “commands” Robo test will use this as a guide 3 Types • Text • Click • Ignore Only Text can have a value Robo Directives What is it?
25. ©2019 Wantedly, Inc. Robo Directives Gcloud command line $ gcloud beta firebase test android run \ !!... !--robo-directives text:textview_resource_id=value,click:view_resource_name= Yaml file args: !!... robo-directives: "ignore:facebook_login_button" : "" "ignore:menu_share" : "" Specifying robo directives
26. ©2019 Wantedly, Inc. You can mix script and directives! Robo test will continue crawling with robo directives as a guide after robo script finishes. Page Title Page Subtitle
27. ©2019 Wantedly, Inc. Robo Test Results
28. ©2019 Wantedly, Inc.
29. ©2019 Wantedly, Inc. Robo Test Results Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase Test Lab!!... Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]] Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud. Firebase Test Lab will execute your robo test on 2 device(s). Creating individual test executions…done. Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]]. 13:34:34 Test matrix status: Finished:2 Robo testing complete. More details are available at [https:!//console.firebase.google.com/project/[REDACTED]]. ┌─────────┬───────────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────────┼──────────────┤ │ Passed │ walleye-27-ja_JP-portrait │ !-- │ │ Passed │ walleye-28-en_US-portrait │ !-- │ └─────────┴───────────────────────────┴──────────────┘ Gcloud command test result
30. ©2019 Wantedly, Inc. Accessible from Test Lab Console Robo Test Results
31. ©2019 Wantedly, Inc. 1. Test issues 2. Robo stats 3. Device logs 4. Screenshots and videos 5. Performance stats 6. GCS Bucket Robo Test Results Execution reports
32. ©2019 Wantedly, Inc. Robo Test Results Test Issues
33. ©2019 Wantedly, Inc. Robo Test Results Robo Stats
34. ©2019 Wantedly, Inc. Robo Test Results Screenshots and videos
35. ©2019 Wantedly, Inc. Robo Test Results Performance stats
36. ©2019 Wantedly, Inc. Robo Test Results Performance over time
37. ©2019 Wantedly, Inc. Robo Test Results Performance over time Demo
38. ©2019 Wantedly, Inc. Robo Test Results Raw results stored in GCS
39. ©2019 Wantedly, Inc. Limitations of Robo Test
40. ©2019 Wantedly, Inc. Limitations of Robo Test Robo script execution status
41. ©2019 Wantedly, Inc. Analyze actions.json file Limitations of Robo Test Robo script execution status !!... , { "startTimeSeconds": 16.865, "endTimeSeconds": 16.865, "roboscriptFinished": { "reason": "ELEMENT_NOT_FOUND" } } !!...
42. ©2019 Wantedly, Inc. Flaky network connection No mocking mechanism Limitations of Robo Test Network
43. ©2019 Wantedly, Inc. 3rd party authentication service Only supports Google login Prevent app from going “outside” Avoid cameras, gallery, share sheet, etc. Only Android UI Framework Unable to interact with WebView or React Native Limitations of Robo Test Use cases
44. ©2019 Wantedly, Inc. Summary
45. ©2019 Wantedly, Inc. Good for testing “flow” of use cases Especially with robo script Can be time consuming to setup and run Extra layer of testing Sanity check Summary
46. ©2019 Wantedly, Inc. Thank You Blog post: https:!//link.medium.com/I7Sr4X4XwY Page Title Page Subtitle