More Effective Testing on Android Devices

Slide 1

Slide 1 text

More Effective Testing on Android Devices by Aurimas Liutikas / Google

Slide 2

Slide 2 text

More Effective Testing on Android Devices of Code That Will Run on Android

Slide 3

Slide 3 text

Don’t - if you can avoid it

Slide 4

Slide 4 text

Best - JUnit tests on JVM Pros ● Cacheable in most build systems ● Multiple orders of magnitude faster ● Nudges tests to unit test scope Cons ● Requires refactoring to pure JVM projects / isolation from android.* APIs

Slide 5

Slide 5 text

Best - JUnit tests on JVM Tips ● Run multiple tests at once maxParallelForks ● Gradle Enterprise test distribution

Slide 6

Slide 6 text

Robolectric A framework for running Android tests on JVM. Built from Android source code with additional fakes. Google-employee maintainers, but not an Google-owned product

Slide 7

Slide 7 text

Good - Robolectric tests Pros ● Cacheable ● Multiple orders of magnitude faster ● Able to test components that have Android tie-ins ● Easily fake system state (e.g. WiFi off) Cons ● Not an accurate representation of a real Android device ● Google support is shaky

Slide 8

Slide 8 text

Good - Robolectric tests Tips ● Cache system image downloads in CI ● Try to minimize Android API usage ● 4.10 support @GraphicsMode(NATIVE)

Slide 9

Slide 9 text

Okay - Activity-less on device Pros ● Can be <100ms per test method ● Testing real Android behavior Cons ● No caching* unless using Gradle Managed Devices (GMD) or custom runner ● Sharding on through multiple connected devices ● Flaky due to device instability

Slide 10

Slide 10 text

If you must - with Activity on device Pros ● Testing real Android behavior Cons ● Really slow ● No caching* unless using GMD or custom runner ● Sharding on through multiple connected devices ● Flaky due to device instability

Slide 11

Slide 11 text

Robolectric tests 3 JUnit tests on JVM 4 UI tests 1 Activity-less tests 2 Fidelity Performance

Slide 12

Slide 12 text

Robolectric tests 3 JUnit tests on JVM 4 UI tests 1 Activity-less tests 2

Slide 13

Slide 13 text

Test Stability Highly Important ● Flaky JVM tests are bad, flaky Android tests are worse ● Disable/delete flaky tests as running them has high costs ● State clean-up (e.g. @After) ● Factory reset or Android User Profiles in custom lab

Slide 14

Slide 14 text

On Device Tips

Slide 15

Slide 15 text

Only run what you need ● AOSP system images ○ Disable noisy applications (adb shell pm disable-user) ● Automated Test Devices (ATD) images

Slide 16

Slide 16 text

Modularize Tests Along With Features ● Splitting tests allows to shard ● Less interference between tests

Slide 17

Slide 17 text

AndroidX case study

Slide 18

Slide 18 text

Build Time at Bay https://dpesummit.com/chasing-the-speed-of-gradle-builds/

Slide 19

Slide 19 text

Test Time Continuing to Grow

Slide 20

Slide 20 text

Key Insight on APK checksums Test results don’t change if both application and test APKs are the same Combined with modularization → higher hit rate

Slide 21

Slide 21 text

Unstable APK generation ● baseline.profm (issuetracker.google.com/issues/231837768) ● shadow jar including incremental kotlin data (r.android.com/2089482) ● AndroidManifest.xml android:compileSdkVersionCodename (issuetracker.google.com/issues/277836549) ● r8 + API 34 record types regression

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

Slide 25

Slide 25 text

Migration From Custom Lab to Firebase Test Lab Caching APK checksum result caching Sharding from n devices to run m APK sets → 1:1 Isolation multiple APKs sets per device →dedicated device per APK set

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Effects on 95th Percentile

Slide 29

Slide 29 text

Effects on Mean Time

Slide 30

Slide 30 text

What’s next? ● Replace FTL shard retries to per method retries ● Emulator stability work

Slide 31

Slide 31 text

Thanks!