Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Android Application Testing with Firebase Robo Test

Android Application Testing with Firebase Robo Test

It's very common for Android app developers to write unit and instrumentation tests around their app. However, some bugs may still occur when the end users are using the app. Some bugs are quite hard to catch because it only happens under specific conditions that does not happen if the test is run on mock or fake data. In this talk I showed how you can use Firebase Robo Test to automate the UI testing of an Android app which allows you to test your app from end to end without using any mocking mechanism.

Malvin Sutanto

July 23, 2019

More Decks by Malvin Sutanto

Other Decks in Programming


  1. ©2019 Wantedly, Inc. Tokyo Android Meetup Application Testing With Firebase

    Robo Test July 23, 2019 - Malvin Sutanto Photo by Steve Jurvetson on Flickr
  2. ©2019 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android,

    Kotlin, and sometimes iOS. Have been developing Android App since 2013. Twitter/ Medium: @malvinsutanto
  3. ©2019 Wantedly, Inc. Unit tests /test folder and runs on

    JVM Testing functions Instrumentation tests /androidTest folder runs on emulator or physical device Testing UI elements and screens independently Android Application Testing
  4. ©2019 Wantedly, Inc. Works perfectly But, your users may still

    encounter bugs. Page Title Page Subtitle
  5. ©2019 Wantedly, Inc. UI testing tool Integrated with Firebase Test

    Lab Analyze app’s UI structure Simulate real user interactions with the app Firebase Robo Test What is it?
  6. ©2019 Wantedly, Inc. 1. Tests the app as a whole

    Flow from screen to screen 2. Tests against real API No mocking mechanisms 3. Tests obfuscated code apk or aab Firebase Robo Test Benefits
  7. ©2019 Wantedly, Inc. 4. Supports Firebase Test Matrix Tests against

    multiple devices at once 5. Provides performance reports Only if it is run against physical device Firebase Robo Test Benefits - cont’d
  8. ©2019 Wantedly, Inc. Firebase Test Lab console Web Interface Accessible

    from the console’s sidebar: Test Lab | Run a test | Run a Robo test Setup Robo test parameters and uploads your app through the web interface. You can choose multiple device configurations, and specify additional options e.g., test credentials or deep links URL Setup Robo Test
  9. ©2019 Wantedly, Inc. Gcloud command line tool Command Line Interface

    Can be run from your CI/CD infrastructure. Needs to have gcloud command line tools preinstalled. Setup guide: https://cloud.google.com/sdk/ gcloud/ Setup Robo Test $ gcloud beta firebase test android run \ !--type robo \ !--timeout 3m \ !--device model=walleye,version=28,locale=en_US, orientation=portrait \ !--app /path/to/apk_file.apk
  10. ©2019 Wantedly, Inc. Setup with Yaml file Setup Robo Test

    $ gcloud beta firebase test android run \ /path/to/robo.yml:args Gcloud command args: type: robo timeout: 3m device: - model: walleye version: 28 locale: en_US orientation: portrait - model: walleye version: 27 locale: ja_JP orientation: portrait app: app/build/outputs/apk/debug/app-debug.apk
  11. ©2019 Wantedly, Inc. JSON file to guide Robo test Contains

    an array of actions Actions are executed sequentially. Continue with regular robo crawl after all actions has been performed. Robo Script What is it?
  12. ©2019 Wantedly, Inc. Used for Google Play Console’s pre-launch reports

    Created using Android Studio Robo Script What is it? - cont’d
  13. ©2019 Wantedly, Inc. To Record a Robo script Tools |

    Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test This will trigger a build and deploy Interaction with the app will be recorded Robo Script Record from Android Studio
  14. ©2019 Wantedly, Inc. Use physical device! Recording can be very

    slow, especially when it is run on the emulator. Page Title Page Subtitle
  15. ©2019 Wantedly, Inc. 1. Limited interactions Does not support gestures,

    or back button 2. Unable to specify multiple scripts Each test will require a separate script Robo Script Limitations
  16. ©2019 Wantedly, Inc. Specifying robo script Robo Script Gcloud command

    line $ gcloud beta firebase test android run \ !!... !--robo-script /path/to/roboscript.json Yaml file args: !!... robo-script: /path/to/roboscript.json
  17. ©2019 Wantedly, Inc. Set of “commands” Robo test will use

    this as a guide 3 Types • Text • Click • Ignore Only Text can have a value Robo Directives What is it?
  18. ©2019 Wantedly, Inc. Robo Directives Gcloud command line $ gcloud

    beta firebase test android run \ !!... !--robo-directives text:textview_resource_id=value,click:view_resource_name= Yaml file args: !!... robo-directives: "ignore:facebook_login_button" : "" "ignore:menu_share" : "" Specifying robo directives
  19. ©2019 Wantedly, Inc. You can mix script and directives! Robo

    test will continue crawling with robo directives as a guide after robo script finishes. Page Title Page Subtitle
  20. ©2019 Wantedly, Inc. Robo Test Results Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase

    Test Lab!!... Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]] Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud. Firebase Test Lab will execute your robo test on 2 device(s). Creating individual test executions…done. Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]]. 13:34:34 Test matrix status: Finished:2 Robo testing complete. More details are available at [https:!//console.firebase.google.com/project/[REDACTED]]. ┌─────────┬───────────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────────┼──────────────┤ │ Passed │ walleye-27-ja_JP-portrait │ !-- │ │ Passed │ walleye-28-en_US-portrait │ !-- │ └─────────┴───────────────────────────┴──────────────┘ Gcloud command test result
  21. ©2019 Wantedly, Inc. 1. Test issues 2. Robo stats 3.

    Device logs 4. Screenshots and videos 5. Performance stats 6. GCS Bucket Robo Test Results Execution reports
  22. ©2019 Wantedly, Inc. Analyze actions.json file Limitations of Robo Test

    Robo script execution status !!... , { "startTimeSeconds": 16.865, "endTimeSeconds": 16.865, "roboscriptFinished": { "reason": "ELEMENT_NOT_FOUND" } } !!...
  23. ©2019 Wantedly, Inc. 3rd party authentication service Only supports Google

    login Prevent app from going “outside” Avoid cameras, gallery, share sheet, etc. Only Android UI Framework Unable to interact with WebView or React Native Limitations of Robo Test Use cases
  24. ©2019 Wantedly, Inc. Good for testing “flow” of use cases

    Especially with robo script Can be time consuming to setup and run Extra layer of testing Sanity check Summary