Tweet
Tweet

Slide 1

Slide 1 text

©2019 Wantedly, Inc. Tokyo Android Meetup Application Testing With Firebase Robo Test July 23, 2019 - Malvin Sutanto Photo by Steve Jurvetson on Flickr

Slide 2

Slide 2 text

©2019 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android, Kotlin, and sometimes iOS. Have been developing Android App since 2013. Twitter/ Medium: @malvinsutanto

Slide 3

Slide 3 text

©2019 Wantedly, Inc. Android Application Testing

Slide 4

Slide 4 text

©2019 Wantedly, Inc. Unit tests /test folder and runs on JVM Testing functions Instrumentation tests /androidTest folder runs on emulator or physical device Testing UI elements and screens independently Android Application Testing

Slide 5

Slide 5 text

©2019 Wantedly, Inc. Works perfectly But, your users may still encounter bugs. Page Title Page Subtitle

Slide 6

Slide 6 text

©2019 Wantedly, Inc. Firebase Robo Test

Slide 7

Slide 7 text

©2019 Wantedly, Inc. UI testing tool Integrated with Firebase Test Lab Analyze app’s UI structure Simulate real user interactions with the app Firebase Robo Test What is it?

Slide 8

Slide 8 text

©2019 Wantedly, Inc. 1. Tests the app as a whole Flow from screen to screen 2. Tests against real API No mocking mechanisms 3. Tests obfuscated code apk or aab Firebase Robo Test Benefits

Slide 9

Slide 9 text

©2019 Wantedly, Inc. 4. Supports Firebase Test Matrix Tests against multiple devices at once 5. Provides performance reports Only if it is run against physical device Firebase Robo Test Benefits - cont’d

Slide 10

Slide 10 text

©2019 Wantedly, Inc. Setup Robo Test

Slide 11

Slide 11 text

©2019 Wantedly, Inc. Firebase Test Lab console Setup Robo Test Gcloud command line tool

Slide 12

Slide 12 text

©2019 Wantedly, Inc. Firebase Test Lab console Web Interface Accessible from the console’s sidebar: Test Lab | Run a test | Run a Robo test Setup Robo test parameters and uploads your app through the web interface. You can choose multiple device configurations, and specify additional options e.g., test credentials or deep links URL Setup Robo Test

Slide 13

Slide 13 text

©2019 Wantedly, Inc. Gcloud command line tool Command Line Interface Can be run from your CI/CD infrastructure. Needs to have gcloud command line tools preinstalled. Setup guide: https://cloud.google.com/sdk/ gcloud/ Setup Robo Test $ gcloud beta firebase test android run \ !--type robo \ !--timeout 3m \ !--device model=walleye,version=28,locale=en_US, orientation=portrait \ !--app /path/to/apk_file.apk

Slide 14

Slide 14 text

©2019 Wantedly, Inc. Setup with Yaml file Setup Robo Test $ gcloud beta firebase test android run \ /path/to/robo.yml:args Gcloud command args: type: robo timeout: 3m device: - model: walleye version: 28 locale: en_US orientation: portrait - model: walleye version: 27 locale: ja_JP orientation: portrait app: app/build/outputs/apk/debug/app-debug.apk

Slide 15

Slide 15 text

©2019 Wantedly, Inc. Controlling Robo Test Robo Script

Slide 16

Slide 16 text

©2019 Wantedly, Inc. JSON file to guide Robo test Contains an array of actions Actions are executed sequentially. Continue with regular robo crawl after all actions has been performed. Robo Script What is it?

Slide 17

Slide 17 text

©2019 Wantedly, Inc. Used for Google Play Console’s pre-launch reports Created using Android Studio Robo Script What is it? - cont’d

Slide 18

Slide 18 text

©2019 Wantedly, Inc. To Record a Robo script Tools | Firebase | Test Lab | Record Robo Script and Use it to guide Robo Test This will trigger a build and deploy Interaction with the app will be recorded Robo Script Record from Android Studio

Slide 19

Slide 19 text

©2019 Wantedly, Inc. Use physical device! Recording can be very slow, especially when it is run on the emulator. Page Title Page Subtitle

Slide 20

Slide 20 text

©2019 Wantedly, Inc. Example Robo Script Example

Slide 21

Slide 21 text

©2019 Wantedly, Inc. 1. Limited interactions Does not support gestures, or back button 2. Unable to specify multiple scripts Each test will require a separate script Robo Script Limitations

Slide 22

Slide 22 text

©2019 Wantedly, Inc. Specifying robo script Robo Script Gcloud command line $ gcloud beta firebase test android run \ !!... !--robo-script /path/to/roboscript.json Yaml file args: !!... robo-script: /path/to/roboscript.json

Slide 23

Slide 23 text

©2019 Wantedly, Inc. Controlling Robo Test Robo Directives

Slide 24

Slide 24 text

©2019 Wantedly, Inc. Set of “commands” Robo test will use this as a guide 3 Types • Text • Click • Ignore Only Text can have a value Robo Directives What is it?

Slide 25

Slide 25 text

©2019 Wantedly, Inc. Robo Directives Gcloud command line $ gcloud beta firebase test android run \ !!... !--robo-directives text:textview_resource_id=value,click:view_resource_name= Yaml file args: !!... robo-directives: "ignore:facebook_login_button" : "" "ignore:menu_share" : "" Specifying robo directives

Slide 26

Slide 26 text

©2019 Wantedly, Inc. You can mix script and directives! Robo test will continue crawling with robo directives as a guide after robo script finishes. Page Title Page Subtitle

Slide 27

Slide 27 text

©2019 Wantedly, Inc. Robo Test Results

Slide 28

Slide 28 text

©2019 Wantedly, Inc.

Slide 29

Slide 29 text

©2019 Wantedly, Inc. Robo Test Results Uploading [app/build/outputs/apk/debug/app-debug.apk] to Firebase Test Lab!!... Raw results will be stored in your GCS bucket at [https:!//console.developers.google.com/[REDACTED]] Test [matrix-dpg1wuq7kikza] has been created in the Google Cloud. Firebase Test Lab will execute your robo test on 2 device(s). Creating individual test executions…done. Test results will be streamed to [https:!//console.firebase.google.com/project/[REDACTED]]. 13:34:34 Test matrix status: Finished:2 Robo testing complete. More details are available at [https:!//console.firebase.google.com/project/[REDACTED]]. ┌─────────┬───────────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────────┼──────────────┤ │ Passed │ walleye-27-ja_JP-portrait │ !-- │ │ Passed │ walleye-28-en_US-portrait │ !-- │ └─────────┴───────────────────────────┴──────────────┘ Gcloud command test result

Slide 30

Slide 30 text

©2019 Wantedly, Inc. Accessible from Test Lab Console Robo Test Results

Slide 31

Slide 31 text

©2019 Wantedly, Inc. 1. Test issues 2. Robo stats 3. Device logs 4. Screenshots and videos 5. Performance stats 6. GCS Bucket Robo Test Results Execution reports

Slide 32

Slide 32 text

©2019 Wantedly, Inc. Robo Test Results Test Issues

Slide 33

Slide 33 text

©2019 Wantedly, Inc. Robo Test Results Robo Stats

Slide 34

Slide 34 text

©2019 Wantedly, Inc. Robo Test Results Screenshots and videos

Slide 35

Slide 35 text

©2019 Wantedly, Inc. Robo Test Results Performance stats

Slide 36

Slide 36 text

©2019 Wantedly, Inc. Robo Test Results Performance over time

Slide 37

Slide 37 text

©2019 Wantedly, Inc. Robo Test Results Performance over time Demo

Slide 38

Slide 38 text

©2019 Wantedly, Inc. Robo Test Results Raw results stored in GCS

Slide 39

Slide 39 text

©2019 Wantedly, Inc. Limitations of Robo Test

Slide 40

Slide 40 text

©2019 Wantedly, Inc. Limitations of Robo Test Robo script execution status

Slide 41

Slide 41 text

©2019 Wantedly, Inc. Analyze actions.json file Limitations of Robo Test Robo script execution status !!... , { "startTimeSeconds": 16.865, "endTimeSeconds": 16.865, "roboscriptFinished": { "reason": "ELEMENT_NOT_FOUND" } } !!...

Slide 42

Slide 42 text

©2019 Wantedly, Inc. Flaky network connection No mocking mechanism Limitations of Robo Test Network

Slide 43

Slide 43 text

©2019 Wantedly, Inc. 3rd party authentication service Only supports Google login Prevent app from going “outside” Avoid cameras, gallery, share sheet, etc. Only Android UI Framework Unable to interact with WebView or React Native Limitations of Robo Test Use cases

Slide 44

Slide 44 text

©2019 Wantedly, Inc. Summary

Slide 45

Slide 45 text

©2019 Wantedly, Inc. Good for testing “flow” of use cases Especially with robo script Can be time consuming to setup and run Extra layer of testing Sanity check Summary

Slide 46

Slide 46 text

©2019 Wantedly, Inc. Thank You Blog post: https:!//link.medium.com/I7Sr4X4XwY Page Title Page Subtitle