Slide 1

Slide 1 text

Marcel Schnelle @marcelschnelle A NEW ERA OF TESTING

Slide 2

Slide 2 text

JUnit 5

Slide 3

Slide 3 text

JUnit • Successor to JUnit 4, the standard for testing on Android • First public release: September 11, 2016 • Rebuilt from scratch and divided into three components • Jupiter (for writing tests) • Platform (for running tests) • Vintage (for backwards compatibility)

Slide 4

Slide 4 text

A simple JUnit 4 test case import org.junit.Assert.assertEquals import org.junit.Test class ExampleUnitTest { @Test fun test() { assertEquals(4, 2 + 2) } }

Slide 5

Slide 5 text

A simple JUnit 5 test case import org.junit.jupiter.api.Assertions.assertEquals import org.junit.jupiter.api.Test class ExampleUnitTest { @Test fun test() { assertEquals(4, 2 + 2) } }

Slide 6

Slide 6 text

What is the current state of testing with JUnit 5 for Android?

Slide 7

Slide 7 text

How about Unit Tests? It works!

Slide 8

Slide 8 text

android-junit • mannodermaus/android-junit5 • Developer tools for using JUnit 5 with Android • Gradle plugin + libraries for instrumentation tests • First public release: September 11, 2017

Slide 9

Slide 9 text

Setting up Unit Tests // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } 1. Add dependencies to project // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" }

Slide 10

Slide 10 text

Setting up Unit Tests // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } 1. Add dependencies to project // libs.versions.toml [versions] junit4 = “4.13.2” [libraries] junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } For JUnit 4

Slide 11

Slide 11 text

Setting up Unit Tests // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } dependencies { testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) } 2. Prepare module build script file // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } dependencies { testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) }

Slide 12

Slide 12 text

Setting up Unit Tests // app/build.gradle.kts dependencies { testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) } 2. Prepare module build script file // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) For JUnit 4

Slide 13

Slide 13 text

Setting up Unit Tests import org.junit.Assert.assertEquals import org.junit.Test class JUnit4Test { @Test fun `test running with junit4`() { assertEquals(4, 2 + 2) } } import org.junit.jupiter.api.Assertions.assertEquals import org.junit.jupiter.api.RepeatedTest import org.junit.jupiter.api.RepetitionInfo import org.junit.jupiter.api.Test class JUnit5Test { @Test fun `test running with junit5`() { assertEquals(4, 2 + 2) } @RepeatedTest(2) fun `repeated test`(info: RepetitionInfo) { val i = info.currentRepetition assertEquals(i * 2, i + i) } }

Slide 14

Slide 14 text

Setting up Unit Tests

Slide 15

Slide 15 text

How about instrumentation tests? It works! (on API 26 and above)

Slide 16

Slide 16 text

Android Studio (New Project Wizard) Android API Levels (https://apilevels.com)

Slide 17

Slide 17 text

Android Studio (New Project Wizard) Android API Levels (https://apilevels.com)

Slide 18

Slide 18 text

About the minSdk requirement • JUnit 5 uses several Java 8 APIs (SimpleFileVisitor, java.nio.file.*) • Only available on API 26 (Android 8.0) and higher • Desugaring does not include these features • Result: Instrumentation tests are skipped on API < 26

Slide 19

Slide 19 text

About the minSdk requirement

Slide 20

Slide 20 text

Setting up Instrumentation Tests // app/build.gradle.kts dependencies { androidTestImplementation(libs.junit.jupiter.api) androidTestImplementation(libs.junit.jupiter.params) } 1. Add dependencies to module’s build script file

Slide 21

Slide 21 text

Setting up Instrumentation Tests 2. There is no step 2 • Plugin automatically provides new APIs to you • Let's write some JUnit 5 tests!

Slide 22

Slide 22 text

Simple: Test with ActivityScenario // ... import de.mannodermaus.junit5.ActivityScenarioExtension import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.RegisterExtension class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch() @Test fun test() { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }

Slide 23

Slide 23 text

Fancy: Test with ActivityScenario class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch() @ValueSource(strings = ["Login", "Register"]) @ParameterizedTest fun test(buttonLabel: String) { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withText(buttonLabel)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }

Slide 24

Slide 24 text

Fancy: Test with ActivityScenario class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch() @ValueSource(strings = ["Login", "Register"]) @ParameterizedTest fun test(buttonLabel: String) { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withText(buttonLabel)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }

Slide 25

Slide 25 text

Parameterized Argument Sources @ArgumentsSource @NullAndEmpty Source @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource

Slide 26

Slide 26 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource @ValueSource(ints = [1, 2, 3]) @ParameterizedTest fun testInts(value: Int) { // ... } @ValueSource(classes = [Foo::class, Bar::class, Baz::class]) @ParameterizedTest fun testClasses(value: Class<*>) { // ... }

Slide 27

Slide 27 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource @ValueSource(ints = [1, 2, 3]) @ParameterizedTest fun testInts(value: Int) { // ... } @ValueSource(classes = [Foo::class, Bar::class, Baz::class]) @ParameterizedTest fun testClasses(value: Class<*>) { // ... }

Slide 28

Slide 28 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource enum class Direction { North, East, South, West } @EnumSource(Direction::class) @ParameterizedTest fun testEnum(direction: Direction) { // ... }

Slide 29

Slide 29 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource enum class Direction { North, East, South, West } @EnumSource(Direction::class) @ParameterizedTest fun testEnum(direction: Direction) { // ... }

Slide 30

Slide 30 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource companion object { @JvmStatic fun arguments() = listOf( Arguments.of("First", Foo()), Arguments.of("Second", Bar()), Arguments.of("Third", Baz()), ) } @MethodSource("arguments") @ParameterizedTest fun testMethod(label: String, model: Base) { // ... }

Slide 31

Slide 31 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource companion object { @JvmStatic fun arguments() = listOf( Arguments.of("First", Foo()), Arguments.of("Second", Bar()), Arguments.of("Third", Baz()), ) } @MethodSource("arguments") @ParameterizedTest fun testMethod(label: String, model: Base) { // ... }

Slide 32

Slide 32 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource companion object { @JvmStatic val argumentsField = listOf(1234, 777, 573) } @FieldSource("argumentsField") @ParameterizedTest fun testField(number: Int) { // ... }

Slide 33

Slide 33 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource companion object { @JvmStatic val argumentsField = listOf(1234, 777, 573) } @FieldSource("argumentsField") @ParameterizedTest fun testField(number: Int) { // ... }

Slide 34

Slide 34 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @CsvSource( "Alice, 34, f, true", "Bob, 16, m, false", "Charlie, 45, m, true", ) @ParameterizedTest fun testCsv(name: String, age: Int, gender: Char, expectedAdult: Boolean) { val person = Person(name, age, gender) assertEquals(expectedAdult, person.isAdult) }

Slide 35

Slide 35 text

@ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @CsvSource( "Alice, 34, f, true", "Bob, 16, m, false", "Charlie, 45, m, true", ) @ParameterizedTest fun testCsv(name: String, age: Int, gender: Char, expectedAdult: Boolean) { val person = Person(name, age, gender) assertEquals(expectedAdult, person.isAdult) }

Slide 36

Slide 36 text

Testing Compose UI works, too.

Slide 37

Slide 37 text

Compose UI Test with JUnit 5 @RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() } @RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() }

Slide 38

Slide 38 text

@RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() } Compose UI Test with JUnit 5

Slide 39

Slide 39 text

Conditional Test Execution

Slide 40

Slide 40 text

Conditional Test Execution • Tests can be skipped when certain conditions apply, e.g.: • Environment variables • System properties • Operating system • android-junit5 adds more conditions, e.g.: • Build manufacturer • BuildConfig value • API level

Slide 41

Slide 41 text

Conditional Annotations @Test @DisabledIfEnvironmentVariable(named = "CI", matches = "true") fun testOnlyOnLocal() { // ... } @Test @EnabledIfEnvironmentVariable(named = "CI", matches = "true") fun testOnlyOnCI() { // ... }

Slide 42

Slide 42 text

Conditional Annotations @Test @EnabledOnManufacturer(["samsung"]) fun testOnlyOnSamsungDevices() { // ... } @Test @DisabledOnManufacturer(["google"]) fun testNotOnGoogleDevices() { // ... }

Slide 43

Slide 43 text

Conditional Annotations @Test @EnabledIfBuildConfigValue(named = "DEBUG", matches = "true") fun testInDebugBuilds() { // ... } @Test @DisabledIfBuildConfigValue(named = "FLAVOR", matches = "prod") fun testNotInProd() { // ... }

Slide 44

Slide 44 text

Conditional Annotations @Test @EnabledOnSdkVersion(from = 26, until = 30) fun testFromAndroid8UntilAndroid11() { // ... } @Test @DisabledOnSdkVersion(from = 33) fun testOnlyBelowAndroid13() { // ... }

Slide 45

Slide 45 text

Parallel Test Execution

Slide 46

Slide 46 text

Parallel Test Execution • Can greatly reduce test execution time • All unit tests can (theoretically) be parallelized • For instrumentation tests • UI tests (Espresso, Compose) are not supported • Non-UI tests (e.g. Realm database) are supported

Slide 47

Slide 47 text

Slow: Without Parallel Tests class ParallelTest { @RepeatedTest(5) fun test(info: RepetitionInfo) { val i = info.currentRepetition Thread.sleep(i * 1_000L) } }

Slide 48

Slide 48 text

Setting up Parallel Test Execution // app/build.gradle.kts junitPlatform { configurationParameter( "junit.jupiter.execution.parallel.enabled", "true" ) } // app/build.gradle.kts junitPlatform { configurationParameter( "junit.jupiter.execution.parallel.enabled", "true" ) }

Slide 49

Slide 49 text

Fast(er): With Parallel Tests @Execution(ExecutionMode.CONCURRENT) class ParallelTest { @RepeatedTest(5) fun test(info: RepetitionInfo) { val i = info.currentRepetition Thread.sleep(i * 1_000L) } }

Slide 50

Slide 50 text

How about Robolectric? It works! (but there is much to do)

Slide 51

Slide 51 text

Robolectric: An overview • Framework for running unit tests with Android APIs • Physical device not necessary • Simulation of hardware events for better tests • Other libraries (Roborazzi, Robospec) are built on top

Slide 52

Slide 52 text

Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 53

Slide 53 text

Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 54

Slide 54 text

Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner @RunWith(RobolectricTestRunner::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 55

Slide 55 text

Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner @RunWith(RobolectricTestRunner::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 56

Slide 56 text

How does Robolectric work? • Robolectric injects a modified version of android.jar into the runtime classpath • When the ClassLoader loads an Android API, it uses Robolectric's version instead of the default stub • RobolectricTestRunner takes care of this injection

Slide 57

Slide 57 text

Problem: JUnit 5 did not support custom class loaders. For a long time.

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

uT

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

junit. -robolectric-extension • apter-tech/junit5-robolectric-extension • Class loader replacement logic for Robolectric & JUnit 5 • Still highly experimental, but works for basic cases

Slide 62

Slide 62 text

Robolectric with JUnit 5 import android.content.Intent import android.net.Uri import org.junit.jupiter.api.Assertions.assertNotNull import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.ExtendWith import tech.apter.junit.jupiter.robolectric.RobolectricExtension @ExtendWith(RobolectricExtension::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 63

Slide 63 text

Robolectric with JUnit 5 import android.content.Intent import android.net.Uri import org.junit.jupiter.api.Assertions.assertNotNull import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.ExtendWith import tech.apter.junit.jupiter.robolectric.RobolectricExtension @ExtendWith(RobolectricExtension::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }

Slide 64

Slide 64 text

How about the future? We’re working on it!

Slide 65

Slide 65 text

What doesn't work (1) • junit5-robolectric-extension issue with Compose unit tests @ExtendWith(RobolectricExtension::class) class RobolectricComposeJUnit5Test { @RegisterExtension val composeExtension = createComposeExtension() @Test fun `robolectric compose test running with junit5`() { composeExtension.use { setContent { Text("Test") } onNodeWithText("Test").assertIsDisplayed() } } }

Slide 66

Slide 66 text

What doesn't work (1) • junit5-robolectric-extension issue with Compose unit tests @ExtendWith(RobolectricExtension::class) class RobolectricComposeJUnit5Test { @RegisterExtension val composeExtension = createComposeExtension() @Test fun `robolectric compose test running with junit5`() { composeExtension.use { setContent { Text("Test") } onNodeWithText("Test").assertIsDisplayed() } } }

Slide 67

Slide 67 text

What doesn't work (2) • Using Espresso in multiple unit tests causes a deadlock @ExtendWith(RobolectricExtension::class) class RobolectricEspressoJUnit5Test { @Test fun `robolectric espresso test #1 running with junit5`() { val scenario = ActivityScenario.launch(ViewActivity::class.java) onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) scenario.close() } @Test fun `robolectric espresso test #2 running with junit5`() { // ... } }

Slide 68

Slide 68 text

What doesn't work (2) • Using Espresso in multiple unit tests causes a deadlock @ExtendWith(RobolectricExtension::class) class RobolectricEspressoJUnit5Test { @Test fun `robolectric espresso test #1 running with junit5`() { val scenario = ActivityScenario.launch(ViewActivity::class.java) onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) scenario.close() } @Test fun `robolectric espresso test #2 running with junit5`() { // ... } } will run forever...

Slide 69

Slide 69 text

What doesn't work (3) • Roborazzi & Robospec are not yet supported with JUnit 5

Slide 70

Slide 70 text

My testing recommendation in 2024 Unit Tests Instrumentation Tests Robolectric, Espresso, Compose Other Tests minSdk < 26 minSdk >= 26 (w/ Vintage Engine) + (slow adoption) 4 4

Slide 71

Slide 71 text

Marcel Schnelle @marcelschnelle A NEW ERA OF TESTING