Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building Android Testing Infrastructure

Mohit S
August 02, 2022

Building Android Testing Infrastructure

Mohit S

August 02, 2022
Tweet

More Decks by Mohit S

Other Decks in Programming

Transcript

  1. Mohit Sarveiya
    Building Android Testing Infrastructure

    @heyitsmohit

    View full-size slide

  2. Building Android Testing Infrastructure
    ● Automation Tools

    ● Building Infra pipelines

    ● Snapshot Testing

    ● Gradle Testing

    View full-size slide

  3. Challenges
    Teams
    Monorepo

    View full-size slide

  4. Arch Fragmentation
    Codebase
    Legacy code
    Feature A
    Feature B

    View full-size slide

  5. Testing
    Codebase 50% Code coverage
    20% Code coverage

    View full-size slide

  6. Test Coverage
    Gradle Builds
    Database
    UI
    API

    View full-size slide

  7. Testing Infra Goals
    ● Automate

    ● Scale

    ● Testing Gradle, Database, UI

    View full-size slide

  8. UI Test Automation

    View full-size slide

  9. Test Case
    Test Suite
    Test Case Test Case Test Case

    View full-size slide

  10. Test Case
    Test Case
    Test Case
    Test Case

    View full-size slide

  11. Test Case
    Test Case
    Test Case
    Test Case
    Firebase Test Lab

    View full-size slide

  12. gCloud
    Tool to run tests on Firebase
    Integrates with CI

    View full-size slide

  13. ~ gCloud firebase test android models list

    View full-size slide

  14. MODEL_ID MAKE MODEL_NAME FORM RESOLUTION OS
    Nexus4 LG Nexus 4 VIRTUAL 2560 x 1600 19,21,22
    sailfish Google Pixel PHYSICAL 1080 x 1920 25, 26
    ~ gCloud firebase test android models list

    View full-size slide

  15. ~ gCloud firebase test android run
    —type
    robo tests
    instrumentation

    View full-size slide

  16. ~ gCloud firebase test android run
    —type instrumentation
    —app app-debug-unaligned.apk

    View full-size slide

  17. ~ gCloud firebase test android run
    —type instrumentation
    —app app-debug-unaligned.apk
    —device model=Nexus6, version=21 local=en, orientation=portrait

    View full-size slide

  18. Upload
    Google Cloud Storage

    View full-size slide

  19. Google Cloud Storage
    Run Tests
    Report

    View full-size slide

  20. ~ gCloud firebase test android run
    —type robo

    View full-size slide

  21. Get Report
    Test Lab
    gCloud
    Build APK
    CI Pipeline

    View full-size slide

  22. UI Tests
    Time

    View full-size slide

  23. Flank
    Run Android and iOS tests in parallel
    Uses Kotlin Coroutines

    View full-size slide

  24. Flank
    flank:

    app: ./app-debug.apk

    test: ./app-debug-androidTest.apk

    View full-size slide

  25. Flank
    flank:

    app: ./app-debug.apk

    test: ./app-debug-androidTest.apk

    device:

    - model: NexusLowRes

    version: 28

    View full-size slide

  26. Test Case
    Test Case
    Test Case
    Test Case
    Parallel

    View full-size slide

  27. Flank
    flank:


    ##
    test shards - the amount of groups to split the test suite into

    max-test-shards: 2

    View full-size slide

  28. Flank
    flank:


    ##
    test shards - the amount of groups to split the test suite into

    max-test-shards: 2

    ##
    shard time - the amount of time tests within a shard should take

    shard-time: 2

    View full-size slide

  29. ~ flank android test run

    View full-size slide

  30. Test Case
    Test Case
    Test Case
    Test Case
    Parallel

    View full-size slide

  31. Merged Results
    Shared 2
    Parallel
    Shard 1

    View full-size slide

  32. Test Case
    Test Case
    Test Case
    Test Case
    Problems
    Flaky tests

    View full-size slide

  33. Flaky Test Causes
    ● Concurrency problems

    ● Flaky third party code

    ● Resource idling.

    View full-size slide

  34. Flank
    flank:


    ##
    Number of Flaky test attempts

    num-flaky-test-attempts: 2

    View full-size slide

  35. CI Setup
    Get Report
    Test Lab
    Flank
    Build APK

    View full-size slide

  36. CI Setup
    Get Report
    Test Lab
    Flank
    Build APK
    Docker

    View full-size slide

  37. Fladle
    Gradle plugin for using Flank
    Multi Module Testing support

    View full-size slide

  38. Recipes
    Performance
    Regression

    View full-size slide

  39. Recipes
    perfTests {

    devices.set([

    ["model" : "Nexus5", "version" : "28"],

    ])

    testTargets.set([

    "class com.test.PerformanceTest"

    ])

    }

    View full-size slide

  40. Recipes
    regressionTests {

    devices.set([

    [ "model" : "Nexus4", "version" : "28"]

    ])

    testTargets.set([

    "class com.sample.MyRegressionTest"

    ])

    }

    View full-size slide

  41. Problems
    ● Tests take too long to run

    ● Flaky tests

    ● When do you run UI tests?

    View full-size slide

  42. Nightly CI Job
    Running UI Tests
    Get Report
    Regression

    Tests
    Fladle
    Build APK

    View full-size slide

  43. Smoke Tests
    ● Test basic functionality

    ● Run on PRs

    View full-size slide

  44. Smoke Tests
    @Retention(AnnotationRetention.RUNTIME)

    annotation class SmokeTest

    View full-size slide

  45. Example - Firefox App
    ● Open source

    ● Nightly & Smoke tests

    ● Shards with Flank

    View full-size slide

  46. https:
    /
    /
    github.com/mozilla-mobile/fenix

    View full-size slide

  47. Firefox App
    Test Rails
    Smoke

    Tests
    Fladle
    Build APK

    View full-size slide

  48. Test Rails
    ● Documentation for your test suite

    ● View results over time

    ● Collaborate with QA

    View full-size slide

  49. UI Tests Infra
    ● Run tests in parallel

    ● Use Flank with Firebase test lab

    ● Flaky tests

    View full-size slide

  50. Unit Testing Infrastructure

    View full-size slide

  51. Unit Testing
    Integration
    E2E
    Slower,

    more expensive
    Faster,

    cheaper

    View full-size slide

  52. How do we structure unit tests?

    View full-size slide

  53. Problem
    class Presenter(

    val repo: Repo,

    ...


    )

    View full-size slide

  54. Problem
    interface Repo {


    fun getData(): Data

    
 ...


    }

    View full-size slide

  55. Structure
    Feature (folder)
    Public (Module)
    Impl (Module)
    Fakes (Module)

    View full-size slide

  56. Structure
    class PresenterTest {


    val fakeRepo = FakeRepository()


    val presenter = Presenter(fakeRepo)

    }

    View full-size slide

  57. Structure
    Feature (folder)
    Public (Module)
    Impl (Module)
    Fakes (Module)

    View full-size slide

  58. Dev Tools
    ● Auto Generate module scaffolding

    ● Enforce rules on how modules depends on each other

    View full-size slide

  59. Unit Testing Lint Rules
    ● Enforce best practices

    View full-size slide

  60. Unit Testing Lint Rules
    ● Enforce best practices

    ● Coroutine lint test rules

    View full-size slide

  61. Unit Testing Lint Rules
    ● Enforce best practices

    ● Coroutine lint test rules

    ● Do not mock data classes

    View full-size slide

  62. Unit Tests
    Time

    View full-size slide

  63. Problem
    bdae142
    ..
    main
    50157a
    ..
    d89f145
    ..
    f0ddfb
    ..

    View full-size slide

  64. Run only affected unit tests

    View full-size slide

  65. Affected Module Dectector
    Gradle plugin for determine which files changed
    Run only affected tests

    View full-size slide

  66. Affected Module Detector
    App
    Module A Module B
    Module C

    View full-size slide

  67. Affected Module Detector
    affectedModuleDetector {

    baseDir = "${project.rootDir}"

    compareFrom = "PreviousCommit"

    }

    View full-size slide

  68. Affected Module Detector
    affectedModuleDetector {

    baseDir = "${project.rootDir}"

    compareFrom = "PreviousCommit"

    }

    Fork Commit

    SpecifiedBranchCommit

    View full-size slide

  69. Affected Module Detector
    App
    Module A Module B
    Module C
    assembleAndroidDebugTest

    connectedAndroidDebugTest

    testDebug

    View full-size slide

  70. Affected Module Detector
    ~ ./gradlew runAffectedUnitTests
    //
    Runs jvm tests

    View full-size slide

  71. Affected Module Detector
    ~ ./gradlew runAffectedAndroidTests
    //
    Runs UI tests

    View full-size slide

  72. Run only affected unit tests

    View full-size slide

  73. Unit Testing
    Integration
    E2E
    Slower,

    more expensive
    Faster,

    cheaper

    View full-size slide

  74. Problem
    ● Run only affected unit tests

    ● Snapshot testing with unit tests

    View full-size slide

  75. Problem
    ● UI Regressions

    ● Example - Constraint layout changes

    View full-size slide

  76. Paparazzi
    Gradle plugin to generate screenshot with unit tests
    Supports Compose UI

    View full-size slide

  77. Paparazzi
    @get:Rule

    val paparazzi = Paparazzi(

    deviceConfig = PIXEL_5,

    theme = “android.Theme.Material.Light.NoActionBar”

    )

    View full-size slide

  78. Paparazzi
    @get:Rule

    val paparazzi = Paparazzi(
    ...
    )

    @Test

    fun testView() {

    paparazzi.snapshot {

    UiView(uiState)

    }

    }

    View full-size slide

  79. Paparazzi
    Git (LFS)
    Snapshot

    View full-size slide

  80. Run Paparazzi tests

    View full-size slide

  81. Paparazzi
    ~ ./gradlew app:recordPaparazziDebug
    //
    Generate Report

    View full-size slide

  82. Paparazzi
    ~ ./gradlew app:verifyPaparazziDebug
    //
    Run again previously recorded

    View full-size slide

  83. Problem
    ● Run only affected unit tests

    ● Creating snapshot with unit testing

    ● Unit testing database migrations

    View full-size slide

  84. Database Migrations

    View full-size slide

  85. Database Migrations
    id user_name user_email
    1 User 1 [email protected]
    2 User 2 [email protected]
    User Table

    View full-size slide

  86. Database Migrations
    id user_name user_email
    1 User 1 [email protected]
    2 User 2 [email protected]
    User Table
    Rename to “email”

    View full-size slide

  87. Database Migrations
    @RenameColumn(

    tableName = "users",

    fromColumnName = "user_email",

    toColumnName = "email"

    )

    class RenameFromUserAddressToAddress : AutoMigrationSpec

    View full-size slide

  88. Database Migrations
    @Database(

    version = 2,

    autoMigrations = [

    AutoMigration(

    from = 1, to = 2,

    spec = UserDatabase.RenameFromUserAddressToAddress
    ::
    class

    ),

    ],

    )

    View full-size slide

  89. How do we test migrations?

    View full-size slide

  90. Test Migrations
    @get:Rule

    val helper: MigrationTestHelper = MigrationTestHelper(

    UserDatabase
    ::
    class.java,

    listOf(

    UserDatabase.RenameFromUserAddressToAddress()

    ),

    )

    View full-size slide

  91. Test Migrations
    @Test

    fun migrate1To2() {

    db = helper.createDatabase(TEST_DB, 1).apply {

    execSQL(

    """

    INSERT INTO users VALUES (1, ‘User 1', ‘[email protected]')
    """.trimIndent()

    )

    close()

    }

    }

    View full-size slide

  92. Test Migrations
    @Test

    fun migrate1To2() {


    db = helper.runMigrationsAndValidate(TEST_DB, 2, true)

    }

    View full-size slide

  93. Test Migrations
    @Test

    fun migrate1To2() {


    val resultCursor = db.query("SELECT * FROM users”)

    //
    Perform assertions

    }

    View full-size slide

  94. Problem
    ● Test migrations with unit tests

    View full-size slide

  95. Database Migrations
    JDBC SQLite Driver

    View full-size slide

  96. DB Tools Room for Android
    Test Room with databases
    Migration testing

    View full-size slide

  97. https:
    //
    github.com/jeffdcamp/dbtools-room

    View full-size slide

  98. Test Migrations
    class MigrationTest: BaseMigrationTest()

    View full-size slide

  99. Test Migrations
    val db = Room.databaseBuilder(
    ...
    )

    .openHelperFactory(JdbcSQLiteOpenHelperFactory(…))

    .build()

    View full-size slide

  100. Test Migrations
    fun testMigration(fromVersion: Int, toVersion: Int) {

    migrationTestExtension.createDatabase(name, fromVersion)

    migrationTestExtension.runMigrationsAndValidate(

    name,

    toVersion,

    *migrations

    )

    }

    View full-size slide

  101. Problem
    ● Run only affected unit tests

    ● Creating snapshot with unit testing

    ● Unit testing database migrations

    View full-size slide

  102. Test Coverage
    Gradle Builds
    Database
    UI
    API

    View full-size slide

  103. Gradle Testing

    View full-size slide

  104. Problem
    ● Detect build regressions

    View full-size slide

  105. Gradle Profiler
    Change

    View full-size slide

  106. Gradle Profiler
    1. Write a performance scenario

    2. Specify number of iterations

    View full-size slide

  107. Gradle Profiler
    Iteration 1
    Iteration 2
    Iteration 3

    View full-size slide

  108. Gradle Profiler
    Iteration 1
    Iteration 2
    Iteration 3
    Abi Change

    View full-size slide

  109. Gradle Profiler
    Iteration 1
    Iteration 2
    Iteration 3
    Result

    View full-size slide

  110. Performance Scenario
    incremental_build {

    apply-abi-change-to = “src/main/java/StringUtils.kt”

    }

    View full-size slide

  111. Performance Scenario
    incremental_build {

    apply-abi-change-to = “src/main/java/StringUtils.kt”

    apply-android-resource-change-to = “strings.xml”

    tasks = ["assemble"]

    }

    View full-size slide

  112. Gradle Profiler
    gradle-profiler —benchmark —iterations=10 —warmups=6

    View full-size slide

  113. Gradle Profiler
    Mean: 348 ms

    Min: 319 ms

    P25: 330 ms

    Median: 341 ms

    P75: 368 ms

    Std dev: 22.71 ms

    View full-size slide

  114. Gradle Profiler

    View full-size slide

  115. How do we automate profiling on CI?

    View full-size slide

  116. CI Pipeline
    Docker image with

    Gradle Profiler

    View full-size slide

  117. CI Pipeline
    Docker image with

    Gradle Profiler
    Performance

    Scenarios

    View full-size slide

  118. CI Pipeline
    Docker image with

    Gradle Profiler
    Performance

    Scenarios
    Run

    nightly

    View full-size slide

  119. Use Cases
    • Introducing Anvil

    View full-size slide

  120. Benchmarking Build
    • Benchmark with Anvil change

    • Benchmark without Anvil change (Baseline)

    • Compare results

    View full-size slide

  121. Results
    Module A Module B Module C
    Baseline Anvil

    View full-size slide

  122. Building Android Testing Infrastructure
    ● Automation Tools

    ● Building Infra pipelines

    ● Snapshot Testing

    ● Gradle Regressions

    View full-size slide

  123. Thank You!
    www.codingwithmohit.com
    @heyitsmohit

    View full-size slide