Android Application Testing with Firebase Robo Test

It's very common for Android app developers to write unit and instrumentation tests around their app. However, some bugs may still occur when the end users are using the app. Some bugs are quite hard to catch because it only happens under specific conditions that does not happen if the test is run on mock or fake data. In this talk I showed how you can use Firebase Robo Test to automate the UI testing of an Android app which allows you to test your app from end to end without using any mocking mechanism.

Transcript

1. ©2019 Wantedly, Inc.
   Tokyo Android Meetup
   Application Testing
   With Firebase Robo Test
   July 23, 2019 - Malvin Sutanto
   Photo by Steve Jurvetson on Flickr
   

   View Slide

2. ©2019 Wantedly, Inc.
   Introduction
   Malvin Sutanto
   Software Engineer @Wantedly
   Android, Kotlin, and sometimes iOS.
   Have been developing Android App since
   2013.
   Twitter/ Medium: @malvinsutanto
   

   View Slide

3. ©2019 Wantedly, Inc.
   Android Application Testing
   

   View Slide

4. ©2019 Wantedly, Inc.
   Unit tests
   /test folder and runs on JVM
   Testing functions
   Instrumentation tests
   /androidTest folder runs on emulator or physical device
   Testing UI elements and screens independently
   Android Application Testing
   

   View Slide

5. ©2019 Wantedly, Inc.
   Works perfectly
   But, your users may still encounter bugs.
   Page Title Page Subtitle
   

   View Slide

6. ©2019 Wantedly, Inc.
   Firebase Robo Test
   

   View Slide

7. ©2019 Wantedly, Inc.
   UI testing tool
   Integrated with Firebase Test Lab
   Analyze app’s UI structure
   Simulate real user interactions with the app
   Firebase Robo Test What is it?
   

   View Slide

8. ©2019 Wantedly, Inc.
   1. Tests the app as a whole
   Flow from screen to screen
   2. Tests against real API
   No mocking mechanisms
   3. Tests obfuscated code
   apk or aab
   Firebase Robo Test Benefits
   

   View Slide

9. ©2019 Wantedly, Inc.
   4. Supports Firebase Test Matrix
   Tests against multiple devices at once
   5. Provides performance reports
   Only if it is run against physical device
   Firebase Robo Test Benefits - cont’d
   

   View Slide

10. ©2019 Wantedly, Inc.
    Setup Robo Test
    

    View Slide

11. ©2019 Wantedly, Inc.
    Firebase Test Lab console
    Setup Robo Test
    Gcloud command line tool
    

    View Slide

12. ©2019 Wantedly, Inc.
    Firebase Test Lab console
    Web Interface
    Accessible from the console’s sidebar: Test
    Lab | Run a test | Run a Robo test
    Setup Robo test parameters and uploads your
    app through the web interface.
    You can choose multiple device
    configurations, and specify additional options
    e.g., test credentials or deep links URL
    Setup Robo Test
    

    View Slide

13. ©2019 Wantedly, Inc.
    Gcloud command line tool
    Command Line Interface
    Can be run from your CI/CD infrastructure.
    Needs to have gcloud command line tools
    preinstalled.
    Setup guide: https://cloud.google.com/sdk/
    gcloud/
    Setup Robo Test
    $ gcloud beta firebase test android run \
    !--type robo \
    !--timeout 3m \
    !--device model=walleye,version=28,locale=en_US,
    orientation=portrait \
    !--app /path/to/apk_file.apk
    

    View Slide

14. ©2019 Wantedly, Inc.
    Setup with Yaml file
    Setup Robo Test
    $ gcloud beta firebase test android run \
    /path/to/robo.yml:args
    Gcloud command
    args:
    type: robo
    timeout: 3m
    device:
    - model: walleye
    version: 28
    locale: en_US
    orientation: portrait
    - model: walleye
    version: 27
    locale: ja_JP
    orientation: portrait
    app: app/build/outputs/apk/debug/app-debug.apk
    

    View Slide

15. ©2019 Wantedly, Inc.
    Controlling Robo Test
    Robo Script
    

    View Slide

16. ©2019 Wantedly, Inc.
    JSON file to guide Robo test
    Contains an array of actions
    Actions are executed sequentially.
    Continue with regular robo crawl after all actions has been performed.
    Robo Script What is it?
    

    View Slide

17. ©2019 Wantedly, Inc.
    Used for Google Play Console’s pre-launch
    reports
    Created using Android Studio
    Robo Script What is it? - cont’d
    

    View Slide

18. ©2019 Wantedly, Inc.
    To Record a Robo script
    Tools | Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test
    This will trigger a build and deploy
    Interaction with the app will be recorded
    Robo Script Record from Android Studio
    

    View Slide

19. ©2019 Wantedly, Inc.
    Use physical device!
    Recording can be very slow, especially when it is run on the emulator.
    Page Title Page Subtitle
    

    View Slide

20. ©2019 Wantedly, Inc.
    Example
    Robo Script
    Example
    

    View Slide

21. ©2019 Wantedly, Inc.
    1. Limited interactions
    Does not support gestures, or back button
    2. Unable to specify multiple scripts
    Each test will require a separate script
    Robo Script Limitations
    

    View Slide

22. ©2019 Wantedly, Inc.
    Specifying robo script
    Robo Script
    Gcloud command line
    $ gcloud beta firebase test android run \
    !!...
    !--robo-script /path/to/roboscript.json
    Yaml file
    args:
    !!...
    robo-script: /path/to/roboscript.json
    

    View Slide

23. ©2019 Wantedly, Inc.
    Controlling Robo Test
    Robo Directives
    

    View Slide

24. ©2019 Wantedly, Inc.
    Set of “commands”
    Robo test will use this as a guide
    3 Types
    • Text
    • Click
    • Ignore
    Only Text can have a value
    Robo Directives What is it?
    

    View Slide

25. ©2019 Wantedly, Inc.
    Robo Directives
    Gcloud command line
    $ gcloud beta firebase test android run \
    !!...
    !--robo-directives text:textview_resource_id=value,click:view_resource_name=
    Yaml file
    args:
    !!...
    robo-directives:
    "ignore:facebook_login_button" : ""
    "ignore:menu_share" : ""
    Specifying robo directives
    

    View Slide

26. ©2019 Wantedly, Inc.
    You can mix script and directives!
    Robo test will continue crawling with robo directives as a guide after robo script finishes.
    Page Title Page Subtitle
    

    View Slide

27. ©2019 Wantedly, Inc.
    Robo Test Results
    

    View Slide

28. ©2019 Wantedly, Inc.
    

    View Slide

29. ©2019 Wantedly, Inc.
    Robo Test Results
    Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase Test Lab!!...
    Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]]
    Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud.
    Firebase Test Lab will execute your robo test on 2 device(s).
    Creating individual test executions…done.
    Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]].
    13:34:34 Test matrix status: Finished:2
    Robo testing complete.
    More details are available at [https:!//console.firebase.google.com/project/[REDACTED]].
    ┌─────────┬───────────────────────────┬──────────────┐
    │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │
    ├─────────┼───────────────────────────┼──────────────┤
    │ Passed │ walleye-27-ja_JP-portrait │ !-- │
    │ Passed │ walleye-28-en_US-portrait │ !-- │
    └─────────┴───────────────────────────┴──────────────┘
    Gcloud command test result
    

    View Slide

30. ©2019 Wantedly, Inc.
    Accessible from Test Lab Console
    Robo Test Results
    

    View Slide

31. ©2019 Wantedly, Inc.
    1. Test issues
    2. Robo stats
    3. Device logs
    4. Screenshots and videos
    5. Performance stats
    6. GCS Bucket
    Robo Test Results Execution reports
    

    View Slide

32. ©2019 Wantedly, Inc.
    Robo Test Results Test Issues
    

    View Slide

33. ©2019 Wantedly, Inc.
    Robo Test Results Robo Stats
    

    View Slide

34. ©2019 Wantedly, Inc.
    Robo Test Results Screenshots and videos
    

    View Slide

35. ©2019 Wantedly, Inc.
    Robo Test Results Performance stats
    

    View Slide

36. ©2019 Wantedly, Inc.
    Robo Test Results Performance over time
    

    View Slide

37. ©2019 Wantedly, Inc.
    Robo Test Results Performance over time
    Demo
    

    View Slide

38. ©2019 Wantedly, Inc.
    Robo Test Results Raw results stored in GCS
    

    View Slide

39. ©2019 Wantedly, Inc.
    Limitations of Robo Test
    

    View Slide

40. ©2019 Wantedly, Inc.
    Limitations of Robo Test Robo script execution status
    

    View Slide

41. ©2019 Wantedly, Inc.
    Analyze actions.json file
    Limitations of Robo Test Robo script execution status
    !!...
    , {
    "startTimeSeconds": 16.865,
    "endTimeSeconds": 16.865,
    "roboscriptFinished": {
    "reason": "ELEMENT_NOT_FOUND"
    }
    }
    !!...
    

    View Slide

42. ©2019 Wantedly, Inc.
    Flaky network connection
    No mocking mechanism
    Limitations of Robo Test Network
    

    View Slide

43. ©2019 Wantedly, Inc.
    3rd party authentication service
    Only supports Google login
    Prevent app from going “outside”
    Avoid cameras, gallery, share sheet, etc.
    Only Android UI Framework
    Unable to interact with WebView or React Native
    Limitations of Robo Test Use cases
    

    View Slide

44. ©2019 Wantedly, Inc.
    Summary
    

    View Slide

45. ©2019 Wantedly, Inc.
    Good for testing “flow” of use cases
    Especially with robo script
    Can be time consuming to setup and run
    Extra layer of testing
    Sanity check
    Summary
    

    View Slide

46. ©2019 Wantedly, Inc.
    Thank You
    Blog post: https:!//link.medium.com/I7Sr4X4XwY
    Page Title Page Subtitle
    

    View Slide