Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Android Application Testing with Firebase Robo Test

Android Application Testing with Firebase Robo Test

It's very common for Android app developers to write unit and instrumentation tests around their app. However, some bugs may still occur when the end users are using the app. Some bugs are quite hard to catch because it only happens under specific conditions that does not happen if the test is run on mock or fake data. In this talk I showed how you can use Firebase Robo Test to automate the UI testing of an Android app which allows you to test your app from end to end without using any mocking mechanism.

Malvin Sutanto

July 23, 2019
Tweet

More Decks by Malvin Sutanto

Other Decks in Programming

Transcript

  1. ©2019 Wantedly, Inc.
    Tokyo Android Meetup
    Application Testing
    With Firebase Robo Test
    July 23, 2019 - Malvin Sutanto
    Photo by Steve Jurvetson on Flickr

    View Slide

  2. ©2019 Wantedly, Inc.
    Introduction
    Malvin Sutanto
    Software Engineer @Wantedly
    Android, Kotlin, and sometimes iOS.
    Have been developing Android App since
    2013.
    Twitter/ Medium: @malvinsutanto

    View Slide

  3. ©2019 Wantedly, Inc.
    Android Application Testing

    View Slide

  4. ©2019 Wantedly, Inc.
    Unit tests
    /test folder and runs on JVM
    Testing functions
    Instrumentation tests
    /androidTest folder runs on emulator or physical device
    Testing UI elements and screens independently
    Android Application Testing

    View Slide

  5. ©2019 Wantedly, Inc.
    Works perfectly
    But, your users may still encounter bugs.
    Page Title Page Subtitle

    View Slide

  6. ©2019 Wantedly, Inc.
    Firebase Robo Test

    View Slide

  7. ©2019 Wantedly, Inc.
    UI testing tool
    Integrated with Firebase Test Lab
    Analyze app’s UI structure
    Simulate real user interactions with the app
    Firebase Robo Test What is it?

    View Slide

  8. ©2019 Wantedly, Inc.
    1. Tests the app as a whole
    Flow from screen to screen
    2. Tests against real API
    No mocking mechanisms
    3. Tests obfuscated code
    apk or aab
    Firebase Robo Test Benefits

    View Slide

  9. ©2019 Wantedly, Inc.
    4. Supports Firebase Test Matrix
    Tests against multiple devices at once
    5. Provides performance reports
    Only if it is run against physical device
    Firebase Robo Test Benefits - cont’d

    View Slide

  10. ©2019 Wantedly, Inc.
    Setup Robo Test

    View Slide

  11. ©2019 Wantedly, Inc.
    Firebase Test Lab console
    Setup Robo Test
    Gcloud command line tool

    View Slide

  12. ©2019 Wantedly, Inc.
    Firebase Test Lab console
    Web Interface
    Accessible from the console’s sidebar: Test
    Lab | Run a test | Run a Robo test
    Setup Robo test parameters and uploads your
    app through the web interface.
    You can choose multiple device
    configurations, and specify additional options
    e.g., test credentials or deep links URL
    Setup Robo Test

    View Slide

  13. ©2019 Wantedly, Inc.
    Gcloud command line tool
    Command Line Interface
    Can be run from your CI/CD infrastructure.
    Needs to have gcloud command line tools
    preinstalled.
    Setup guide: https://cloud.google.com/sdk/
    gcloud/
    Setup Robo Test
    $ gcloud beta firebase test android run \
    !--type robo \
    !--timeout 3m \
    !--device model=walleye,version=28,locale=en_US,
    orientation=portrait \
    !--app /path/to/apk_file.apk

    View Slide

  14. ©2019 Wantedly, Inc.
    Setup with Yaml file
    Setup Robo Test
    $ gcloud beta firebase test android run \
    /path/to/robo.yml:args
    Gcloud command
    args:
    type: robo
    timeout: 3m
    device:
    - model: walleye
    version: 28
    locale: en_US
    orientation: portrait
    - model: walleye
    version: 27
    locale: ja_JP
    orientation: portrait
    app: app/build/outputs/apk/debug/app-debug.apk

    View Slide

  15. ©2019 Wantedly, Inc.
    Controlling Robo Test
    Robo Script

    View Slide

  16. ©2019 Wantedly, Inc.
    JSON file to guide Robo test
    Contains an array of actions
    Actions are executed sequentially.
    Continue with regular robo crawl after all actions has been performed.
    Robo Script What is it?

    View Slide

  17. ©2019 Wantedly, Inc.
    Used for Google Play Console’s pre-launch
    reports
    Created using Android Studio
    Robo Script What is it? - cont’d

    View Slide

  18. ©2019 Wantedly, Inc.
    To Record a Robo script
    Tools | Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test
    This will trigger a build and deploy
    Interaction with the app will be recorded
    Robo Script Record from Android Studio

    View Slide

  19. ©2019 Wantedly, Inc.
    Use physical device!
    Recording can be very slow, especially when it is run on the emulator.
    Page Title Page Subtitle

    View Slide

  20. ©2019 Wantedly, Inc.
    Example
    Robo Script
    Example

    View Slide

  21. ©2019 Wantedly, Inc.
    1. Limited interactions
    Does not support gestures, or back button
    2. Unable to specify multiple scripts
    Each test will require a separate script
    Robo Script Limitations

    View Slide

  22. ©2019 Wantedly, Inc.
    Specifying robo script
    Robo Script
    Gcloud command line
    $ gcloud beta firebase test android run \
    !!...
    !--robo-script /path/to/roboscript.json
    Yaml file
    args:
    !!...
    robo-script: /path/to/roboscript.json

    View Slide

  23. ©2019 Wantedly, Inc.
    Controlling Robo Test
    Robo Directives

    View Slide

  24. ©2019 Wantedly, Inc.
    Set of “commands”
    Robo test will use this as a guide
    3 Types
    • Text
    • Click
    • Ignore
    Only Text can have a value
    Robo Directives What is it?

    View Slide

  25. ©2019 Wantedly, Inc.
    Robo Directives
    Gcloud command line
    $ gcloud beta firebase test android run \
    !!...
    !--robo-directives text:textview_resource_id=value,click:view_resource_name=
    Yaml file
    args:
    !!...
    robo-directives:
    "ignore:facebook_login_button" : ""
    "ignore:menu_share" : ""
    Specifying robo directives

    View Slide

  26. ©2019 Wantedly, Inc.
    You can mix script and directives!
    Robo test will continue crawling with robo directives as a guide after robo script finishes.
    Page Title Page Subtitle

    View Slide

  27. ©2019 Wantedly, Inc.
    Robo Test Results

    View Slide

  28. ©2019 Wantedly, Inc.

    View Slide

  29. ©2019 Wantedly, Inc.
    Robo Test Results
    Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase Test Lab!!...
    Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]]
    Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud.
    Firebase Test Lab will execute your robo test on 2 device(s).
    Creating individual test executions…done.
    Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]].
    13:34:34 Test matrix status: Finished:2
    Robo testing complete.
    More details are available at [https:!//console.firebase.google.com/project/[REDACTED]].
    ┌─────────┬───────────────────────────┬──────────────┐
    │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │
    ├─────────┼───────────────────────────┼──────────────┤
    │ Passed │ walleye-27-ja_JP-portrait │ !-- │
    │ Passed │ walleye-28-en_US-portrait │ !-- │
    └─────────┴───────────────────────────┴──────────────┘
    Gcloud command test result

    View Slide

  30. ©2019 Wantedly, Inc.
    Accessible from Test Lab Console
    Robo Test Results

    View Slide

  31. ©2019 Wantedly, Inc.
    1. Test issues
    2. Robo stats
    3. Device logs
    4. Screenshots and videos
    5. Performance stats
    6. GCS Bucket
    Robo Test Results Execution reports

    View Slide

  32. ©2019 Wantedly, Inc.
    Robo Test Results Test Issues

    View Slide

  33. ©2019 Wantedly, Inc.
    Robo Test Results Robo Stats

    View Slide

  34. ©2019 Wantedly, Inc.
    Robo Test Results Screenshots and videos

    View Slide

  35. ©2019 Wantedly, Inc.
    Robo Test Results Performance stats

    View Slide

  36. ©2019 Wantedly, Inc.
    Robo Test Results Performance over time

    View Slide

  37. ©2019 Wantedly, Inc.
    Robo Test Results Performance over time
    Demo

    View Slide

  38. ©2019 Wantedly, Inc.
    Robo Test Results Raw results stored in GCS

    View Slide

  39. ©2019 Wantedly, Inc.
    Limitations of Robo Test

    View Slide

  40. ©2019 Wantedly, Inc.
    Limitations of Robo Test Robo script execution status

    View Slide

  41. ©2019 Wantedly, Inc.
    Analyze actions.json file
    Limitations of Robo Test Robo script execution status
    !!...
    , {
    "startTimeSeconds": 16.865,
    "endTimeSeconds": 16.865,
    "roboscriptFinished": {
    "reason": "ELEMENT_NOT_FOUND"
    }
    }
    !!...

    View Slide

  42. ©2019 Wantedly, Inc.
    Flaky network connection
    No mocking mechanism
    Limitations of Robo Test Network

    View Slide

  43. ©2019 Wantedly, Inc.
    3rd party authentication service
    Only supports Google login
    Prevent app from going “outside”
    Avoid cameras, gallery, share sheet, etc.
    Only Android UI Framework
    Unable to interact with WebView or React Native
    Limitations of Robo Test Use cases

    View Slide

  44. ©2019 Wantedly, Inc.
    Summary

    View Slide

  45. ©2019 Wantedly, Inc.
    Good for testing “flow” of use cases
    Especially with robo script
    Can be time consuming to setup and run
    Extra layer of testing
    Sanity check
    Summary

    View Slide

  46. ©2019 Wantedly, Inc.
    Thank You
    Blog post: https:!//link.medium.com/I7Sr4X4XwY
    Page Title Page Subtitle

    View Slide