Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Android Testing Infrastructure
Search
Mohit S
August 02, 2022
Programming
1
500
Building Android Testing Infrastructure
Mohit S
August 02, 2022
Tweet
Share
More Decks by Mohit S
See All by Mohit S
Guide to Improving Compose Performance
heyitsmohit
0
240
Building Shared UIs across Platforms with Compose
heyitsmohit
1
640
Building Multiplatform Apps with Compose
heyitsmohit
2
520
Building StateFlows with Jetpack Compose
heyitsmohit
6
1.9k
Migrating to Kotlin State & Shared Flows
heyitsmohit
1
800
Using Square Workflow for Android & iOS
heyitsmohit
1
430
Building Android Infrastructure Teams at Scale
heyitsmohit
3
330
Strategies for Migrating to Jetpack Compose
heyitsmohit
2
570
Challenges of Building Kotlin Multiplatform Libraries
heyitsmohit
1
440
Other Decks in Programming
See All in Programming
MCPとデザインシステムに立脚したデザインと実装の融合
yukukotani
4
1.5k
ユーザーも開発者も悩ませない TV アプリ開発 ~Compose の内部実装から学ぶフォーカス制御~
taked137
0
190
AI時代のUIはどこへ行く?
yusukebe
18
9.1k
OSS開発者という働き方
andpad
5
1.7k
個人開発で徳島大学生60%以上の心を掴んだアプリ、そして手放した話
akidon0000
1
150
ProxyによるWindow間RPC機構の構築
syumai
3
1.2k
How Android Uses Data Structures Behind The Scenes
l2hyunwoo
0
480
Android端末で実現するオンデバイスLLM 2025
masayukisuda
1
170
ファインディ株式会社におけるMCP活用とサービス開発
starfish719
0
2.1k
The Past, Present, and Future of Enterprise Java with ASF in the Middle
ivargrimstad
0
180
機能追加とリーダー業務の類似性
rinchoku
2
1.3k
AIと私たちの学習の変化を考える - Claude Codeの学習モードを例に
azukiazusa1
11
4.4k
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
43
7.6k
Building Adaptive Systems
keathley
43
2.7k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1.1k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.7k
Statistics for Hackers
jakevdp
799
220k
Faster Mobile Websites
deanohume
309
31k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6k
The Cost Of JavaScript in 2023
addyosmani
53
8.9k
4 Signs Your Business is Dying
shpigford
184
22k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Transcript
Mohit Sarveiya Building Android Testing Infrastructure @heyitsmohit
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Testing
Challenges Teams Monorepo
Arch Fragmentation Codebase Legacy code Feature A Feature B
Testing Codebase 50% Code coverage 20% Code coverage
Test Coverage Gradle Builds Database UI API
Testing Infra Goals • Automate • Scale • Testing Gradle,
Database, UI
UI Test Automation
Test Case Test Suite Test Case Test Case Test Case
Test Case Test Case Test Case Test Case
Test Case Test Case Test Case Test Case Firebase Test
Lab
gCloud Tool to run tests on Firebase Integrates with CI
~ gCloud firebase test android models list
MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS Nexus4 LG Nexus 4
VIRTUAL 2560 x 1600 19,21,22 sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26 ~ gCloud firebase test android models list
~ gCloud firebase test android run —type robo tests instrumentation
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk
—device model=Nexus6, version=21 local=en, orientation=portrait
Upload Google Cloud Storage
Google Cloud Storage Run Tests Report
None
~ gCloud firebase test android run —type robo
None
Get Report Test Lab gCloud Build APK CI Pipeline
UI Tests Time
Flank Run Android and iOS tests in parallel Uses Kotlin
Coroutines
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk
Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk device: - model: NexusLowRes
version: 28
Test Case Test Case Test Case Test Case Parallel
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2
Flank flank: ## test shards - the amount
of groups to split the test suite into max-test-shards: 2 ## shard time - the amount of time tests within a shard should take shard-time: 2
~ flank android test run
Test Case Test Case Test Case Test Case Parallel
Merged Results Shared 2 Parallel Shard 1
Test Case Test Case Test Case Test Case Problems Flaky
tests
Flaky Test Causes • Concurrency problems • Flaky third party
code • Resource idling.
Flank flank: ## Number of Flaky test attempts
num-flaky-test-attempts: 2
CI Setup Get Report Test Lab Flank Build APK
CI Setup Get Report Test Lab Flank Build APK Docker
Fladle Gradle plugin for using Flank Multi Module Testing support
Recipes Performance Regression
Recipes perfTests { devices.set([ ["model" : "Nexus5", "version" : "28"],
]) testTargets.set([ "class com.test.PerformanceTest" ]) }
Recipes regressionTests { devices.set([ [ "model" : "Nexus4", "version" :
"28"] ]) testTargets.set([ "class com.sample.MyRegressionTest" ]) }
Problems • Tests take too long to run • Flaky
tests • When do you run UI tests?
Nightly CI Job Running UI Tests Get Report Regression
Tests Fladle Build APK
Smoke Tests • Test basic functionality • Run on PRs
Smoke Tests @Retention(AnnotationRetention.RUNTIME) annotation class SmokeTest
Example - Firefox App • Open source • Nightly &
Smoke tests • Shards with Flank
https: / / github.com/mozilla-mobile/fenix
Firefox App Test Rails Smoke Tests Fladle Build APK
Test Rails • Documentation for your test suite • View
results over time • Collaborate with QA
None
None
UI Tests Infra • Run tests in parallel • Use
Flank with Firebase test lab • Flaky tests
Unit Testing Infrastructure
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
How do we structure unit tests?
Problem class Presenter( val repo: Repo, ... )
Problem interface Repo { fun getData(): Data
... }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Structure class PresenterTest { val fakeRepo = FakeRepository()
val presenter = Presenter(fakeRepo) }
Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)
Dev Tools • Auto Generate module scaffolding • Enforce rules
on how modules depends on each other
Unit Testing Lint Rules • Enforce best practices
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules
Unit Testing Lint Rules • Enforce best practices • Coroutine
lint test rules • Do not mock data classes
Unit Tests Time
Problem bdae142 .. main 50157a .. d89f145 .. f0ddfb ..
None
Run only affected unit tests
Affected Module Dectector Gradle plugin for determine which files changed
Run only affected tests
Affected Module Detector App Module A Module B Module C
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" }
Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =
"PreviousCommit" } Fork Commit SpecifiedBranchCommit
Affected Module Detector App Module A Module B Module C
assembleAndroidDebugTest connectedAndroidDebugTest testDebug
Affected Module Detector ~ ./gradlew runAffectedUnitTests // Runs jvm tests
Affected Module Detector ~ ./gradlew runAffectedAndroidTests // Runs UI tests
Run only affected unit tests
Unit Testing Integration E2E Slower, more expensive Faster,
cheaper
Problem • Run only affected unit tests • Snapshot testing
with unit tests
Problem • UI Regressions • Example - Constraint layout changes
Paparazzi Gradle plugin to generate screenshot with unit tests Supports
Compose UI
Paparazzi @get:Rule val paparazzi = Paparazzi( deviceConfig = PIXEL_5,
theme = “android.Theme.Material.Light.NoActionBar” )
Paparazzi @get:Rule val paparazzi = Paparazzi( ... ) @Test fun
testView() { paparazzi.snapshot { UiView(uiState) } }
Paparazzi Git (LFS) Snapshot
Run Paparazzi tests
Paparazzi ~ ./gradlew app:recordPaparazziDebug // Generate Report
Paparazzi ~ ./gradlew app:verifyPaparazziDebug // Run again previously recorded
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Database Migrations
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table
Database Migrations id user_name user_email 1 User 1
[email protected]
2
User 2
[email protected]
User Table Rename to “email”
Database Migrations @RenameColumn( tableName = "users", fromColumnName =
"user_email", toColumnName = "email" ) class RenameFromUserAddressToAddress : AutoMigrationSpec
Database Migrations @Database( version = 2, autoMigrations = [ AutoMigration(
from = 1, to = 2, spec = UserDatabase.RenameFromUserAddressToAddress :: class ), ], )
How do we test migrations?
Test Migrations @get:Rule val helper: MigrationTestHelper = MigrationTestHelper( UserDatabase ::
class.java, listOf( UserDatabase.RenameFromUserAddressToAddress() ), )
Test Migrations @Test fun migrate1To2() { db = helper.createDatabase(TEST_DB, 1).apply
{ execSQL( """ INSERT INTO users VALUES (1, ‘User 1', ‘
[email protected]
') """.trimIndent() ) close() } }
Test Migrations @Test fun migrate1To2() { db = helper.runMigrationsAndValidate(TEST_DB,
2, true) }
Test Migrations @Test fun migrate1To2() { val resultCursor =
db.query("SELECT * FROM users”) // Perform assertions }
Problem • Test migrations with unit tests
Database Migrations JDBC SQLite Driver
DB Tools Room for Android Test Room with databases Migration
testing
https: // github.com/jeffdcamp/dbtools-room
Test Migrations class MigrationTest: BaseMigrationTest<UserDatabase>()
Test Migrations val db = Room.databaseBuilder( ... ) .openHelperFactory(JdbcSQLiteOpenHelperFactory(…)) .build()
Test Migrations fun testMigration(fromVersion: Int, toVersion: Int) { migrationTestExtension.createDatabase(name, fromVersion)
migrationTestExtension.runMigrationsAndValidate( name, toVersion, *migrations ) }
Problem • Run only affected unit tests • Creating snapshot
with unit testing • Unit testing database migrations
Test Coverage Gradle Builds Database UI API
Gradle Testing
Problem • Detect build regressions
Gradle Profiler Change
Gradle Profiler 1. Write a performance scenario 2. Specify number
of iterations
Gradle Profiler Iteration 1 Iteration 2 Iteration 3
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Abi Change
Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Result
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” }
Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” apply-android-resource-change-to = “strings.xml”
tasks = ["assemble"] }
Gradle Profiler gradle-profiler —benchmark —iterations=10 —warmups=6
Gradle Profiler Mean: 348 ms Min: 319 ms P25: 330
ms Median: 341 ms P75: 368 ms Std dev: 22.71 ms
Gradle Profiler
How do we automate profiling on CI?
CI Pipeline Docker image with Gradle Profiler
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios
CI Pipeline Docker image with Gradle Profiler Performance
Scenarios Run nightly
Use Cases • Introducing Anvil
Benchmarking Build • Benchmark with Anvil change • Benchmark without
Anvil change (Baseline) • Compare results
Results Module A Module B Module C Baseline Anvil
Building Android Testing Infrastructure • Automation Tools • Building Infra
pipelines • Snapshot Testing • Gradle Regressions
Thank You! www.codingwithmohit.com @heyitsmohit