Ahmed Gad and Michael Bailey: Using Google Cloud Test Lab to improve the quality of Android apps: A case study from Amex

Android testing is hard! With new devices coming out every year, it’s very challenging to ensure the compatibility of your applications across all these form factors. In this talk, we’ll tell you how Cloud Test Lab helps developers address these issues. With a scalable infrastructure that allows you to run automated test on hundreds of devices with one click, a smart Robo that allows you to test your app launches without having to write complex scripts and clients that integrate into your workflow from the web, Android Studio and on your CI environment.

We’ll also be hearing from the lead engineer on American Express Android team on how using Cloud Test Lab significantly reduced their infrastructure investment and allowed them to run tens of thousands of tests every day. He’ll be sharing some of the best practices on using Cloud Test Lab in order to get the most benefit in your project.

Transcript

1. Using Google Cloud Test Lab to improve the quality of Android apps: A case study from Amex Ahmed Gad Product Manager, Google @amounirgad Michael Bailey Senior Staff Engineer, American Express @yogurtearl
2. 84% of users will not open an app again after 2 early usage crashes 1 Source: “What Consumers Really Need and Want”, Mobile App Survey Report, Dynatrace, 2014 App quality drives reviews, usage & revenue 1 “What Consumers Really Need and Want”, Mobile App Survey Report, Dynatrace, 2014
3. Many quality issues are controllable Classification of 1-star reviews (Sampling of reviews in Play Store, Sep 2014) 56% of the 1-star reviews were due to quality issues 32% - Crashes & bugs 14% - Slow network 10% - Laggy client 44% - Other
4. Mobile testing and quality control is hard Limited resources Copyrighted 2015. Business Insider, Inc. 116904:0515DS Cumbersome debugging Largely manual
5. Cloud Test Lab can help!
6. Copyrighted 2015. Business Insider, Inc. 116904: 0515DS Cloud Test Lab: Scalable infrastructure
7. Cloud Test Lab: Full automation support Robo-testing Intelligent app navigation Installation failures, crashes “something for nothing” Instrumentation tests Espresso / Robotium Matrix of combinations Massively parallel Results at fingertips Detailed results, logs, debug help Screenshots, videos History
8. Cloud Test Lab: Integrates with your workflow
9. Cloud Test Lab for Continuous Integration • CI is just one of many uses for CTL • Before adding any tool to your workflow ask: ◦ What challenges am I facing? ◦ Does this tool address a challenge that I have? ◦ Is there a simpler way to address that challenge?
10. Challenge • Is this code change a good code change? ◦ Does the code change accomplish the desired outcome? ◦ Does the code change introduce any negative side effects?
11. Solution Code change ( Pull Request ) Build Test Jenkins logo by Charles Lowell and Frontside CC BY-SA 3.0 https://wiki.jenkins-ci.org/display/JENKINS/Logo Git Logo by Jason Long CC BY 3.0 https://git-scm.com/downloads/logos
12. Solution Jenkins logo by Charles Lowell and Frontside CC BY-SA 3.0 https://wiki.jenkins-ci.org/display/JENKINS/Logo Git Logo by Jason Long CC BY 3.0 https://git-scm.com/downloads/logos
13. Using Cloud Test Lab with Continuous Integration Systems https://developers.google.com/cloud-test-lab/continuous
14. Pretested PRs
15. Pretested PRs
16. job-dsl-plugin https://github.com/jenkinsci/job-dsl-plugin def project = 'yogurtearl/ctl-talk' def branchApi = new URL("https://api.github.com/repos/${project}/branches") def branches = new groovy.json.JsonSlurper().parse(branchApi.newReader()) branches.each { def branchName = it.name def jobName = "${project}-${branchName}".replaceAll('/','-') job(jobName) { scm { git("git://github.com/${project}.git", branchName) } steps { maven("test -Dproject.name=${project}/${branchName}") } } } From: https://wiki.jenkins-ci.org/display/JENKINS/Job+DSL+Plugin
17. Pretested PRs
18. Pretested PRs
19. Pretested PRs
20. Pretested PRs
21. Pretested PRs
22. Hipchat Notifications
23. Android Tests • Desktop JVM “unit tests” ( /tests ) • Instrumentation Tests ( /androidTests )
24. Physical Devices
25. Physical device problems • ADB is flaky • USB is flaky • Keeping devices charged • System level dialogs
26. Why CTL? • Physical Devices ◦ CTL handles the pain of managing physical devices. ◦ Real-world full fidelity • Virtual Devices ◦ “x86 Android Virtual Devices as a Service” maintained by someone else ◦ ADB is not directly exposed ◦ Includes Google Play services
27. Run Tests on CTL Install the Google Cloud SDK -> https://cloud.google.com/sdk/ $ ./gradlew assembleDebug assembleDebugAndroidTest $ gcloud beta test android run --app=build/outputs/apk/debug.apk --test=build/outputs/apk/androidTest.apk --device-ids=Nexus5,Nexus4 --os-version-ids=19,21 --test-targets='class com.americanexpress.ctl.EspressoTest'
28. $ gcloud beta test android run … Uploading [app/build/outputs/apk/debug.apk] to the Cloud Test Lab... Uploading [app/build/outputs/apk/androidTest.apk] to the Cloud Test Lab... Raw results will be stored in your GCS bucket at [https://google.com/storage/browser/project_bucket/2016-03-15_DCSF/] Test [matrix-droidcon] has been created in the Google Cloud. Cloud Test Lab will execute your instrumentation test on 4 device(s). Creating individual test executions...done. Test results will be streamed to [https://google.com/project/project_id/testlab/mobile/histories/bh.cafebabe/executions/37]. 15:48:55 Test matrix status: Finished:4 Instrumentation testing complete. More details are available at [https://google.com/project/project_id/testlab/mobile/histories/bh.cafebabe/executions/37]. ┌─────────┬───────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────┼──────────────┤ │ Passed │ Nexus4-19-en-portrait │ -- │ │ Passed │ Nexus4-21-en-portrait │ -- │ │ Passed │ Nexus5-19-en-portrait │ -- │ │ Passed │ Nexus5-21-en-portrait │ -- │ └─────────┴───────────────────────┴──────────────┘
29. Google Cloud Storage Web UI [https://google.com/storage/browser/project_bucket/2016-03-15_DCSF/]
30. gsutil https://cloud.google.com/storage/docs/gsutil [https://google.com/storage/browser/project_bucket/2016-03-15_DCSF/] $ gsutil ls 'gs://project_bucket/2016-03-15_DCSF/’ $ gsutil cp 'gs://project_bucket/2016-03-15_DCSF/**/test_result_*.xml’ build/results
31. Jenkins xUnit Plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin
32. $ gcloud beta test android run … Uploading [app/build/outputs/apk/debug.apk] to the Cloud Test Lab... Uploading [app/build/outputs/apk/androidTest.apk] to the Cloud Test Lab... Raw results will be stored in your GCS bucket at [https://google.com/storage/browser/project_bucket/2016-03-15_DCSF/] Test [matrix-droidcon] has been created in the Google Cloud. Cloud Test Lab will execute your instrumentation test on 4 device(s). Creating individual test executions...done. Test results will be streamed to [https://google.com/project/project_id/testlab/mobile/histories/bh.cafebabe/executions/37]. 15:48:55 Test matrix status: Finished:4 Instrumentation testing complete. More details are available at [https://google.com/project/project_id/testlab/mobile/histories/bh.cafebabe/executions/37]. ┌─────────┬───────────────────────┬──────────────┐ │ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │ ├─────────┼───────────────────────┼──────────────┤ │ Passed │ Nexus4-19-en-portrait │ -- │ │ Passed │ Nexus4-21-en-portrait │ -- │ │ Passed │ Nexus5-19-en-portrait │ -- │ │ Passed │ Nexus5-21-en-portrait │ -- │ └─────────┴───────────────────────┴──────────────┘
33. Cloud Console
34. Facebook screenshot tests
35. Screenshot tests
36. Start testing today! • Test early and often • Test during development and release • Test every submission on Google Play Sign up now: https://developers.google.com/cloud-test-lab/ More tests → Better apps → $$$
37. Thanks! Time for questions Ahmed Gad @amounirgad Michael Bailey @yogurtearl