Android Application Testing with Firebase Robo Test

Android Application Testing with Firebase Robo Test

It's very common for Android app developers to write unit and instrumentation tests around their app. However, some bugs may still occur when the end users are using the app. Some bugs are quite hard to catch because it only happens under specific conditions that does not happen if the test is run on mock or fake data. In this talk I showed how you can use Firebase Robo Test to automate the UI testing of an Android app which allows you to test your app from end to end without using any mocking mechanism.

6338c8fa4e2e6325094fe30b1e9f9443?s=128

Malvin Sutanto

July 23, 2019
Tweet

Transcript

  1. ©2019 Wantedly, Inc. Tokyo Android Meetup Application Testing With Firebase

    Robo Test July 23, 2019 - Malvin Sutanto Photo by Steve Jurvetson on Flickr
  2. ©2019 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android,

    Kotlin, and sometimes iOS. Have been developing Android App since 2013. Twitter/ Medium: @malvinsutanto
  3. ©2019 Wantedly, Inc. Android Application Testing

  4. ©2019 Wantedly, Inc. Unit tests /test folder and runs on

    JVM Testing functions Instrumentation tests /androidTest folder runs on emulator or physical device Testing UI elements and screens independently Android Application Testing
  5. ©2019 Wantedly, Inc. Works perfectly But, your users may still

    encounter bugs. Page Title Page Subtitle
  6. ©2019 Wantedly, Inc. Firebase Robo Test

  7. ©2019 Wantedly, Inc. UI testing tool Integrated with Firebase Test

    Lab Analyze app’s UI structure Simulate real user interactions with the app Firebase Robo Test What is it?
  8. ©2019 Wantedly, Inc. 1. Tests the app as a whole

    Flow from screen to screen 2. Tests against real API No mocking mechanisms 3. Tests obfuscated code apk or aab Firebase Robo Test Benefits
  9. ©2019 Wantedly, Inc. 4. Supports Firebase Test Matrix Tests against

    multiple devices at once 5. Provides performance reports Only if it is run against physical device Firebase Robo Test Benefits - cont’d
  10. ©2019 Wantedly, Inc. Setup Robo Test

  11. ©2019 Wantedly, Inc. Firebase Test Lab console Setup Robo Test

    Gcloud command line tool
  12. ©2019 Wantedly, Inc. Firebase Test Lab console Web Interface Accessible

    from the console’s sidebar: Test Lab | Run a test | Run a Robo test Setup Robo test parameters and uploads your app through the web interface. You can choose multiple device configurations, and specify additional options e.g., test credentials or deep links URL Setup Robo Test
  13. ©2019 Wantedly, Inc. Gcloud command line tool Command Line Interface

    Can be run from your CI/CD infrastructure. Needs to have gcloud command line tools preinstalled. Setup guide: https://cloud.google.com/sdk/ gcloud/ Setup Robo Test $ gcloud beta firebase test android run \ !--type robo \ !--timeout 3m \ !--device model=walleye,version=28,locale=en_US, orientation=portrait \ !--app /path/to/apk_file.apk
  14. ©2019 Wantedly, Inc. Setup with Yaml file Setup Robo Test

    $ gcloud beta firebase test android run \ /path/to/robo.yml:args Gcloud command args: type: robo timeout: 3m device: - model: walleye version: 28 locale: en_US orientation: portrait - model: walleye version: 27 locale: ja_JP orientation: portrait app: app/build/outputs/apk/debug/app-debug.apk
  15. ©2019 Wantedly, Inc. Controlling Robo Test Robo Script

  16. ©2019 Wantedly, Inc. JSON file to guide Robo test Contains

    an array of actions Actions are executed sequentially. Continue with regular robo crawl after all actions has been performed. Robo Script What is it?
  17. ©2019 Wantedly, Inc. Used for Google Play Console’s pre-launch reports

    Created using Android Studio Robo Script What is it? - cont’d
  18. ©2019 Wantedly, Inc. To Record a Robo script Tools |

    Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test This will trigger a build and deploy Interaction with the app will be recorded Robo Script Record from Android Studio
  19. ©2019 Wantedly, Inc. Use physical device! Recording can be very

    slow, especially when it is run on the emulator. Page Title Page Subtitle
  20. ©2019 Wantedly, Inc. Example Robo Script Example

  21. ©2019 Wantedly, Inc. 1. Limited interactions Does not support gestures,

    or back button 2. Unable to specify multiple scripts Each test will require a separate script Robo Script Limitations
  22. ©2019 Wantedly, Inc. Specifying robo script Robo Script Gcloud command

    line $ gcloud beta firebase test android run \ !!... !--robo-script /path/to/roboscript.json Yaml file args: !!... robo-script: /path/to/roboscript.json
  23. ©2019 Wantedly, Inc. Controlling Robo Test Robo Directives

  24. ©2019 Wantedly, Inc. Set of “commands” Robo test will use

    this as a guide 3 Types • Text • Click • Ignore Only Text can have a value Robo Directives What is it?
  25. ©2019 Wantedly, Inc. Robo Directives Gcloud command line $ gcloud

    beta firebase test android run \ !!... !--robo-directives text:textview_resource_id=value,click:view_resource_name= Yaml file args: !!... robo-directives: "ignore:facebook_login_button" : "" "ignore:menu_share" : "" Specifying robo directives
  26. ©2019 Wantedly, Inc. You can mix script and directives! Robo

    test will continue crawling with robo directives as a guide after robo script finishes. Page Title Page Subtitle
  27. ©2019 Wantedly, Inc. Robo Test Results

  28. ©2019 Wantedly, Inc.

  29. ©2019 Wantedly, Inc. Robo Test Results Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase

    Test Lab!!... Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]] Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud. Firebase Test Lab will execute your robo test on 2 device(s). Creating individual test executions…done. Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]]. 13:34:34 Test matrix status: Finished:2 Robo testing complete. More details are available at [https:!//console.firebase.google.com/project/[REDACTED]]. ┌─────────┬───────────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────────┼──────────────┤ │ Passed │ walleye-27-ja_JP-portrait │ !-- │ │ Passed │ walleye-28-en_US-portrait │ !-- │ └─────────┴───────────────────────────┴──────────────┘ Gcloud command test result
  30. ©2019 Wantedly, Inc. Accessible from Test Lab Console Robo Test

    Results
  31. ©2019 Wantedly, Inc. 1. Test issues 2. Robo stats 3.

    Device logs 4. Screenshots and videos 5. Performance stats 6. GCS Bucket Robo Test Results Execution reports
  32. ©2019 Wantedly, Inc. Robo Test Results Test Issues

  33. ©2019 Wantedly, Inc. Robo Test Results Robo Stats

  34. ©2019 Wantedly, Inc. Robo Test Results Screenshots and videos

  35. ©2019 Wantedly, Inc. Robo Test Results Performance stats

  36. ©2019 Wantedly, Inc. Robo Test Results Performance over time

  37. ©2019 Wantedly, Inc. Robo Test Results Performance over time Demo

  38. ©2019 Wantedly, Inc. Robo Test Results Raw results stored in

    GCS
  39. ©2019 Wantedly, Inc. Limitations of Robo Test

  40. ©2019 Wantedly, Inc. Limitations of Robo Test Robo script execution

    status
  41. ©2019 Wantedly, Inc. Analyze actions.json file Limitations of Robo Test

    Robo script execution status !!... , { "startTimeSeconds": 16.865, "endTimeSeconds": 16.865, "roboscriptFinished": { "reason": "ELEMENT_NOT_FOUND" } } !!...
  42. ©2019 Wantedly, Inc. Flaky network connection No mocking mechanism Limitations

    of Robo Test Network
  43. ©2019 Wantedly, Inc. 3rd party authentication service Only supports Google

    login Prevent app from going “outside” Avoid cameras, gallery, share sheet, etc. Only Android UI Framework Unable to interact with WebView or React Native Limitations of Robo Test Use cases
  44. ©2019 Wantedly, Inc. Summary

  45. ©2019 Wantedly, Inc. Good for testing “flow” of use cases

    Especially with robo script Can be time consuming to setup and run Extra layer of testing Sanity check Summary
  46. ©2019 Wantedly, Inc. Thank You Blog post: https:!//link.medium.com/I7Sr4X4XwY Page Title

    Page Subtitle