Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Android Testing Infrastructure

Mohit S
August 02, 2022

Building Android Testing Infrastructure

Mohit S

August 02, 2022
Tweet

More Decks by Mohit S

Other Decks in Programming

Transcript

  1. Mohit Sarveiya Building Android Testing Infrastructure @heyitsmohit

  2. Building Android Testing Infrastructure • Automation Tools • Building Infra

    pipelines • Snapshot Testing • Gradle Testing
  3. Challenges Teams Monorepo

  4. Arch Fragmentation Codebase Legacy code Feature A Feature B

  5. Testing Codebase 50% Code coverage 20% Code coverage

  6. Test Coverage Gradle Builds Database UI API

  7. Testing Infra Goals • Automate • Scale • Testing Gradle,

    Database, UI
  8. UI Test Automation

  9. Test Case Test Suite Test Case Test Case Test Case

  10. Test Case Test Case Test Case Test Case

  11. Test Case Test Case Test Case Test Case Firebase Test

    Lab
  12. gCloud Tool to run tests on Firebase Integrates with CI

  13. ~ gCloud firebase test android models list

  14. MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS Nexus4 LG Nexus 4

    VIRTUAL 2560 x 1600 19,21,22 sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26 ~ gCloud firebase test android models list
  15. ~ gCloud firebase test android run —type robo tests instrumentation

  16. ~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk

  17. ~ gCloud firebase test android run —type instrumentation —app app-debug-unaligned.apk

    —device model=Nexus6, version=21 local=en, orientation=portrait
  18. Upload Google Cloud Storage

  19. Google Cloud Storage Run Tests Report

  20. None
  21. ~ gCloud firebase test android run —type robo

  22. None
  23. Get Report Test Lab gCloud Build APK CI Pipeline

  24. UI Tests Time

  25. Flank Run Android and iOS tests in parallel Uses Kotlin

    Coroutines
  26. Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk

  27. Flank flank: app: ./app-debug.apk test: ./app-debug-androidTest.apk device: - model: NexusLowRes

    version: 28
  28. Test Case Test Case Test Case Test Case Parallel

  29. Flank flank: 
 
 ## test shards - the amount

    of groups to split the test suite into max-test-shards: 2
  30. Flank flank: 
 
 ## test shards - the amount

    of groups to split the test suite into max-test-shards: 2 ## shard time - the amount of time tests within a shard should take shard-time: 2
  31. ~ flank android test run

  32. Test Case Test Case Test Case Test Case Parallel

  33. Merged Results Shared 2 Parallel Shard 1

  34. Test Case Test Case Test Case Test Case Problems Flaky

    tests
  35. Flaky Test Causes • Concurrency problems • Flaky third party

    code • Resource idling.
  36. Flank flank: 
 
 ## Number of Flaky test attempts

    num-flaky-test-attempts: 2
  37. CI Setup Get Report Test Lab Flank Build APK

  38. CI Setup Get Report Test Lab Flank Build APK Docker

  39. Fladle Gradle plugin for using Flank Multi Module Testing support

  40. Recipes Performance Regression

  41. Recipes perfTests { devices.set([ ["model" : "Nexus5", "version" : "28"],

    ]) testTargets.set([ "class com.test.PerformanceTest" ]) }
  42. Recipes regressionTests { devices.set([ [ "model" : "Nexus4", "version" :

    "28"] ]) testTargets.set([ "class com.sample.MyRegressionTest" ]) }
  43. Problems • Tests take too long to run • Flaky

    tests • When do you run UI tests?
  44. Nightly CI Job Running UI Tests Get Report Regression 


    Tests Fladle Build APK
  45. Smoke Tests • Test basic functionality • Run on PRs

  46. Smoke Tests @Retention(AnnotationRetention.RUNTIME) annotation class SmokeTest

  47. Example - Firefox App • Open source • Nightly &

    Smoke tests • Shards with Flank
  48. https: / / github.com/mozilla-mobile/fenix

  49. Firefox App Test Rails Smoke 
 Tests Fladle Build APK

  50. Test Rails • Documentation for your test suite • View

    results over time • Collaborate with QA
  51. None
  52. None
  53. UI Tests Infra • Run tests in parallel • Use

    Flank with Firebase test lab • Flaky tests
  54. Unit Testing Infrastructure

  55. Unit Testing Integration E2E Slower, 
 more expensive Faster, 


    cheaper
  56. How do we structure unit tests?

  57. Problem class Presenter( val repo: Repo, 
 ... )

  58. Problem interface Repo { 
 
 fun getData(): Data 


    
 ... }
  59. Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)

  60. Structure class PresenterTest { 
 val fakeRepo = FakeRepository() 


    
 val presenter = Presenter(fakeRepo) 
 }
  61. Structure Feature (folder) Public (Module) Impl (Module) Fakes (Module)

  62. Dev Tools • Auto Generate module scaffolding • Enforce rules

    on how modules depends on each other
  63. Unit Testing Lint Rules • Enforce best practices

  64. Unit Testing Lint Rules • Enforce best practices • Coroutine

    lint test rules
  65. Unit Testing Lint Rules • Enforce best practices • Coroutine

    lint test rules • Do not mock data classes
  66. Unit Tests Time

  67. Problem bdae142 .. main 50157a .. d89f145 .. f0ddfb ..

  68. None
  69. Run only affected unit tests

  70. Affected Module Dectector Gradle plugin for determine which files changed

    Run only affected tests
  71. Affected Module Detector App Module A Module B Module C

  72. Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =

    "PreviousCommit" }
  73. Affected Module Detector affectedModuleDetector { baseDir = "${project.rootDir}" compareFrom =

    "PreviousCommit" } Fork Commit SpecifiedBranchCommit
  74. Affected Module Detector App Module A Module B Module C

    assembleAndroidDebugTest connectedAndroidDebugTest testDebug
  75. Affected Module Detector ~ ./gradlew runAffectedUnitTests // Runs jvm tests

  76. Affected Module Detector ~ ./gradlew runAffectedAndroidTests // Runs UI tests

  77. Run only affected unit tests

  78. Unit Testing Integration E2E Slower, 
 more expensive Faster, 


    cheaper
  79. Problem • Run only affected unit tests • Snapshot testing

    with unit tests
  80. Problem • UI Regressions • Example - Constraint layout changes

  81. Paparazzi Gradle plugin to generate screenshot with unit tests Supports

    Compose UI
  82. Paparazzi @get:Rule val paparazzi = Paparazzi( 
 deviceConfig = PIXEL_5,

    
 theme = “android.Theme.Material.Light.NoActionBar” )
  83. Paparazzi @get:Rule val paparazzi = Paparazzi( ... ) @Test fun

    testView() { paparazzi.snapshot { UiView(uiState) } }
  84. Paparazzi Git (LFS) Snapshot

  85. Run Paparazzi tests

  86. Paparazzi ~ ./gradlew app:recordPaparazziDebug // Generate Report

  87. Paparazzi ~ ./gradlew app:verifyPaparazziDebug // Run again previously recorded

  88. Problem • Run only affected unit tests • Creating snapshot

    with unit testing • Unit testing database migrations
  89. Database Migrations

  90. Database Migrations id user_name user_email 1 User 1 user1@gmail.com 2

    User 2 user2@gmail.com User Table
  91. Database Migrations id user_name user_email 1 User 1 user1@gmail.com 2

    User 2 user2@gmail.com User Table Rename to “email”
  92. Database Migrations @RenameColumn( 
 tableName = "users", 
 fromColumnName =

    "user_email", 
 toColumnName = "email" 
 ) class RenameFromUserAddressToAddress : AutoMigrationSpec
  93. Database Migrations @Database( version = 2, autoMigrations = [ AutoMigration(

    from = 1, to = 2, spec = UserDatabase.RenameFromUserAddressToAddress :: class ), ], )
  94. How do we test migrations?

  95. Test Migrations @get:Rule val helper: MigrationTestHelper = MigrationTestHelper( UserDatabase ::

    class.java, listOf( UserDatabase.RenameFromUserAddressToAddress() ), )
  96. Test Migrations @Test fun migrate1To2() { db = helper.createDatabase(TEST_DB, 1).apply

    { execSQL( """ INSERT INTO users VALUES (1, ‘User 1', ‘user1@google.com') """.trimIndent() ) close() } }
  97. Test Migrations @Test fun migrate1To2() { 
 db = helper.runMigrationsAndValidate(TEST_DB,

    2, true) }
  98. Test Migrations @Test fun migrate1To2() { 
 val resultCursor =

    db.query("SELECT * FROM users”) // Perform assertions }
  99. Problem • Test migrations with unit tests

  100. Database Migrations JDBC SQLite Driver

  101. DB Tools Room for Android Test Room with databases Migration

    testing
  102. https: // github.com/jeffdcamp/dbtools-room

  103. Test Migrations class MigrationTest: BaseMigrationTest<UserDatabase>()

  104. Test Migrations val db = Room.databaseBuilder( ... ) .openHelperFactory(JdbcSQLiteOpenHelperFactory(…)) .build()

  105. Test Migrations fun testMigration(fromVersion: Int, toVersion: Int) { migrationTestExtension.createDatabase(name, fromVersion)

    migrationTestExtension.runMigrationsAndValidate( name, toVersion, 
 *migrations 
 ) }
  106. Problem • Run only affected unit tests • Creating snapshot

    with unit testing • Unit testing database migrations
  107. Test Coverage Gradle Builds Database UI API

  108. Gradle Testing

  109. Problem • Detect build regressions

  110. Gradle Profiler Change

  111. Gradle Profiler 1. Write a performance scenario 2. Specify number

    of iterations
  112. Gradle Profiler Iteration 1 Iteration 2 Iteration 3

  113. Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Abi Change

  114. Gradle Profiler Iteration 1 Iteration 2 Iteration 3 Result

  115. Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” }

  116. Performance Scenario incremental_build { apply-abi-change-to = “src/main/java/StringUtils.kt” apply-android-resource-change-to = “strings.xml”

    tasks = ["assemble"] }
  117. Gradle Profiler gradle-profiler —benchmark —iterations=10 —warmups=6

  118. Gradle Profiler Mean: 348 ms Min: 319 ms P25: 330

    ms Median: 341 ms P75: 368 ms Std dev: 22.71 ms
  119. Gradle Profiler

  120. How do we automate profiling on CI?

  121. CI Pipeline Docker image with 
 Gradle Profiler

  122. CI Pipeline Docker image with 
 Gradle Profiler Performance 


    Scenarios
  123. CI Pipeline Docker image with 
 Gradle Profiler Performance 


    Scenarios Run 
 nightly
  124. Use Cases • Introducing Anvil

  125. Benchmarking Build • Benchmark with Anvil change • Benchmark without

    Anvil change (Baseline) • Compare results
  126. Results Module A Module B Module C Baseline Anvil

  127. Building Android Testing Infrastructure • Automation Tools • Building Infra

    pipelines • Snapshot Testing • Gradle Regressions
  128. Thank You! www.codingwithmohit.com @heyitsmohit