Mohit SarveiyaBuilding Android Testing Infrastructure @heyitsmohit
View Slide
Building Android Testing Infrastructure● Automation Tools● Building Infra pipelines● Snapshot Testing● Gradle Testing
ChallengesTeamsMonorepo
Arch FragmentationCodebaseLegacy codeFeature AFeature B
TestingCodebase 50% Code coverage20% Code coverage
Test CoverageGradle BuildsDatabaseUIAPI
Testing Infra Goals● Automate● Scale● Testing Gradle, Database, UI
UI Test Automation
Test CaseTest SuiteTest Case Test Case Test Case
Test CaseTest CaseTest CaseTest Case
Test CaseTest CaseTest CaseTest CaseFirebase Test Lab
gCloudTool to run tests on FirebaseIntegrates with CI
~ gCloud firebase test android models list
MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OSNexus4 LG Nexus 4 VIRTUAL 2560 x 1600 19,21,22sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26~ gCloud firebase test android models list
~ gCloud firebase test android run—typerobo testsinstrumentation
~ gCloud firebase test android run—type instrumentation—app app-debug-unaligned.apk
~ gCloud firebase test android run—type instrumentation—app app-debug-unaligned.apk—device model=Nexus6, version=21 local=en, orientation=portrait
UploadGoogle Cloud Storage
Google Cloud StorageRun TestsReport
~ gCloud firebase test android run—type robo
Get ReportTest LabgCloudBuild APKCI Pipeline
UI TestsTime
FlankRun Android and iOS tests in parallelUses Kotlin Coroutines
Flankflank:app: ./app-debug.apktest: ./app-debug-androidTest.apk
Flankflank:app: ./app-debug.apktest: ./app-debug-androidTest.apkdevice:- model: NexusLowResversion: 28
Test CaseTest CaseTest CaseTest CaseParallel
Flankflank: ##test shards - the amount of groups to split the test suite intomax-test-shards: 2
Flankflank: ##test shards - the amount of groups to split the test suite intomax-test-shards: 2##shard time - the amount of time tests within a shard should take shard-time: 2
~ flank android test run
Merged ResultsShared 2ParallelShard 1
Test CaseTest CaseTest CaseTest CaseProblemsFlaky tests
Flaky Test Causes● Concurrency problems● Flaky third party code● Resource idling.
Flankflank: ##Number of Flaky test attemptsnum-flaky-test-attempts: 2
CI SetupGet ReportTest LabFlankBuild APK
CI SetupGet ReportTest LabFlankBuild APKDocker
FladleGradle plugin for using FlankMulti Module Testing support
RecipesPerformanceRegression
RecipesperfTests {devices.set([["model" : "Nexus5", "version" : "28"], ])testTargets.set(["class com.test.PerformanceTest"])}
RecipesregressionTests {devices.set([[ "model" : "Nexus4", "version" : "28"]])testTargets.set(["class com.sample.MyRegressionTest"])}
Problems● Tests take too long to run● Flaky tests● When do you run UI tests?
Nightly CI JobRunning UI TestsGet ReportRegression TestsFladleBuild APK
Smoke Tests● Test basic functionality● Run on PRs
Smoke Tests@Retention(AnnotationRetention.RUNTIME)annotation class SmokeTest
Example - Firefox App● Open source● Nightly & Smoke tests● Shards with Flank
https://github.com/mozilla-mobile/fenix
Firefox AppTest RailsSmoke TestsFladleBuild APK
Test Rails● Documentation for your test suite● View results over time● Collaborate with QA
UI Tests Infra● Run tests in parallel● Use Flank with Firebase test lab● Flaky tests
Unit Testing Infrastructure
Unit TestingIntegrationE2ESlower, more expensiveFaster, cheaper
How do we structure unit tests?
Problemclass Presenter(val repo: Repo, ...)
Probleminterface Repo { fun getData(): Data ...}
StructureFeature (folder)Public (Module)Impl (Module)Fakes (Module)
Structureclass PresenterTest { val fakeRepo = FakeRepository() val presenter = Presenter(fakeRepo) }
Dev Tools● Auto Generate module scaffolding● Enforce rules on how modules depends on each other
Unit Testing Lint Rules● Enforce best practices
Unit Testing Lint Rules● Enforce best practices● Coroutine lint test rules
Unit Testing Lint Rules● Enforce best practices● Coroutine lint test rules● Do not mock data classes
Unit TestsTime
Problembdae142..main50157a..d89f145..f0ddfb..
Run only affected unit tests
Affected Module DectectorGradle plugin for determine which files changedRun only affected tests
Affected Module DetectorAppModule A Module BModule C
Affected Module DetectoraffectedModuleDetector {baseDir = "${project.rootDir}"compareFrom = "PreviousCommit"}
Affected Module DetectoraffectedModuleDetector {baseDir = "${project.rootDir}"compareFrom = "PreviousCommit"}Fork CommitSpecifiedBranchCommit
Affected Module DetectorAppModule A Module BModule CassembleAndroidDebugTestconnectedAndroidDebugTesttestDebug
Affected Module Detector~ ./gradlew runAffectedUnitTests//Runs jvm tests
Affected Module Detector~ ./gradlew runAffectedAndroidTests//Runs UI tests
Problem● Run only affected unit tests● Snapshot testing with unit tests
Problem● UI Regressions● Example - Constraint layout changes
PaparazziGradle plugin to generate screenshot with unit testsSupports Compose UI
Paparazzi@get:Ruleval paparazzi = Paparazzi( deviceConfig = PIXEL_5, theme = “android.Theme.Material.Light.NoActionBar”)
Paparazzi@get:Ruleval paparazzi = Paparazzi(...)@Testfun testView() {paparazzi.snapshot {UiView(uiState)} }
PaparazziGit (LFS)Snapshot
Run Paparazzi tests
Paparazzi~ ./gradlew app:recordPaparazziDebug//Generate Report
Paparazzi~ ./gradlew app:verifyPaparazziDebug//Run again previously recorded
Problem● Run only affected unit tests● Creating snapshot with unit testing● Unit testing database migrations
Database Migrations
Database Migrationsid user_name user_email1 User 1 [email protected]2 User 2 [email protected]User Table
Database Migrationsid user_name user_email1 User 1 [email protected]2 User 2 [email protected]User TableRename to “email”
Database Migrations@RenameColumn( tableName = "users", fromColumnName = "user_email", toColumnName = "email" )class RenameFromUserAddressToAddress : AutoMigrationSpec
Database Migrations@Database(version = 2,autoMigrations = [AutoMigration(from = 1, to = 2,spec = UserDatabase.RenameFromUserAddressToAddress::class),],)
How do we test migrations?
Test Migrations@get:Ruleval helper: MigrationTestHelper = MigrationTestHelper(UserDatabase::class.java,listOf(UserDatabase.RenameFromUserAddressToAddress()),)
Test Migrations@Testfun migrate1To2() {db = helper.createDatabase(TEST_DB, 1).apply {execSQL("""INSERT INTO users VALUES (1, ‘User 1', ‘[email protected]')""".trimIndent())close()}}
Test Migrations@Testfun migrate1To2() { db = helper.runMigrationsAndValidate(TEST_DB, 2, true)}
Test Migrations@Testfun migrate1To2() { val resultCursor = db.query("SELECT * FROM users”)//Perform assertions}
Problem● Test migrations with unit tests
Database MigrationsJDBC SQLite Driver
DB Tools Room for AndroidTest Room with databasesMigration testing
https://github.com/jeffdcamp/dbtools-room
Test Migrationsclass MigrationTest: BaseMigrationTest()
Test Migrationsval db = Room.databaseBuilder(...).openHelperFactory(JdbcSQLiteOpenHelperFactory(…)).build()
Test Migrationsfun testMigration(fromVersion: Int, toVersion: Int) {migrationTestExtension.createDatabase(name, fromVersion)migrationTestExtension.runMigrationsAndValidate(name, toVersion, *migrations )}
Gradle Testing
Problem● Detect build regressions
Gradle ProfilerChange
Gradle Profiler1. Write a performance scenario2. Specify number of iterations
Gradle ProfilerIteration 1Iteration 2Iteration 3
Gradle ProfilerIteration 1Iteration 2Iteration 3Abi Change
Gradle ProfilerIteration 1Iteration 2Iteration 3Result
Performance Scenarioincremental_build {apply-abi-change-to = “src/main/java/StringUtils.kt”}
Performance Scenarioincremental_build {apply-abi-change-to = “src/main/java/StringUtils.kt”apply-android-resource-change-to = “strings.xml”tasks = ["assemble"]}
Gradle Profilergradle-profiler —benchmark —iterations=10 —warmups=6
Gradle ProfilerMean: 348 msMin: 319 msP25: 330 msMedian: 341 msP75: 368 msStd dev: 22.71 ms
Gradle Profiler
How do we automate profiling on CI?
CI PipelineDocker image with Gradle Profiler
CI PipelineDocker image with Gradle ProfilerPerformance Scenarios
CI PipelineDocker image with Gradle ProfilerPerformance ScenariosRun nightly
Use Cases• Introducing Anvil
Benchmarking Build• Benchmark with Anvil change• Benchmark without Anvil change (Baseline)• Compare results
ResultsModule A Module B Module CBaseline Anvil
Building Android Testing Infrastructure● Automation Tools● Building Infra pipelines● Snapshot Testing● Gradle Regressions
Thank You!www.codingwithmohit.com@heyitsmohit
heyitsmohit
gotok365
kinocoboy2
treastrain
jmatsu
nsuz
mukky620
yui_knk
yosuke_furukawa
koba04
tomokusaba
dalinaum
pineapplecandy
keithpitt
malarkey
akmur
thoeni
samanthasiow
philhawksworth
jrom
aki_iinuma
paigeccino
chrisshort
bkeepers
rasmusluckow