A New Era of Testing

Transcript

1. Marcel Schnelle @marcelschnelle A NEW ERA OF TESTING
2. JUnit 5
3. JUnit • Successor to JUnit 4, the standard for testing on Android • First public release: September 11, 2016 • Rebuilt from scratch and divided into three components • Jupiter (for writing tests) • Platform (for running tests) • Vintage (for backwards compatibility)
4. A simple JUnit 4 test case import org.junit.Assert.assertEquals import org.junit.Test class ExampleUnitTest { @Test fun test() { assertEquals(4, 2 + 2) } }
5. A simple JUnit 5 test case import org.junit.jupiter.api.Assertions.assertEquals import org.junit.jupiter.api.Test class ExampleUnitTest { @Test fun test() { assertEquals(4, 2 + 2) } }
6. What is the current state of testing with JUnit 5 for Android?
7. How about Unit Tests? It works!
8. android-junit • mannodermaus/android-junit5 • Developer tools for using JUnit 5 with Android • Gradle plugin + libraries for instrumentation tests • First public release: September 11, 2017
9. Setting up Unit Tests // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } 1. Add dependencies to project // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" }
10. Setting up Unit Tests // libs.versions.toml [versions] androidJUnit5Plugin = “1.11.0.0” junit5 = “5.11.0” junit4 = “4.13.2” [plugins] android-junit5 = { id = "de.mannodermaus.android-junit5", version.ref = "androidJUnit5Plugin" } [libraries] junit-jupiter-api = { module = "org.junit.jupiter:junit-jupiter-api", version.ref = "junit5" } junit-jupiter-engine = { module = "org.junit.jupiter:junit-jupiter-engine", version.ref = "junit5" } junit-jupiter-params = { module = “org.junit.jupiter:junit-jupiter-params", version.ref = "junit5" } junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } 1. Add dependencies to project // libs.versions.toml [versions] junit4 = “4.13.2” [libraries] junit4 = { module = “junit:junit", version.ref = "junit4" } junit-vintage-engine = { module = “org.junit.vintage:junit-vintage-engine”, version.ref = “junit5" } For JUnit 4
11. Setting up Unit Tests // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } dependencies { testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) } 2. Prepare module build script file // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } dependencies { testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) }
12. Setting up Unit Tests // app/build.gradle.kts dependencies { testImplementation(libs.junit4) testRuntimeOnly(libs.junit.vintage.engine) } 2. Prepare module build script file // app/build.gradle.kts plugins { alias(libs.plugins.android.junit5) } testImplementation(libs.junit.jupiter.api) testImplementation(libs.junit.jupiter.params) testRuntimeOnly(libs.junit.jupiter.engine) For JUnit 4
13. Setting up Unit Tests import org.junit.Assert.assertEquals import org.junit.Test class JUnit4Test { @Test fun `test running with junit4`() { assertEquals(4, 2 + 2) } } import org.junit.jupiter.api.Assertions.assertEquals import org.junit.jupiter.api.RepeatedTest import org.junit.jupiter.api.RepetitionInfo import org.junit.jupiter.api.Test class JUnit5Test { @Test fun `test running with junit5`() { assertEquals(4, 2 + 2) } @RepeatedTest(2) fun `repeated test`(info: RepetitionInfo) { val i = info.currentRepetition assertEquals(i * 2, i + i) } }
14. Setting up Unit Tests
15. How about instrumentation tests? It works! (on API 26 and above)
16. Android Studio (New Project Wizard) Android API Levels (https://apilevels.com)
17. Android Studio (New Project Wizard) Android API Levels (https://apilevels.com)
18. About the minSdk requirement • JUnit 5 uses several Java 8 APIs (SimpleFileVisitor, java.nio.file.*) • Only available on API 26 (Android 8.0) and higher • Desugaring does not include these features • Result: Instrumentation tests are skipped on API < 26
19. About the minSdk requirement
20. Setting up Instrumentation Tests // app/build.gradle.kts dependencies { androidTestImplementation(libs.junit.jupiter.api) androidTestImplementation(libs.junit.jupiter.params) } 1. Add dependencies to module’s build script file
21. Setting up Instrumentation Tests 2. There is no step 2 • Plugin automatically provides new APIs to you • Let's write some JUnit 5 tests!
22. Simple: Test with ActivityScenario // ... import de.mannodermaus.junit5.ActivityScenarioExtension import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.RegisterExtension class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch<MyActivity>() @Test fun test() { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }
23. Fancy: Test with ActivityScenario class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch<MyActivity>() @ValueSource(strings = ["Login", "Register"]) @ParameterizedTest fun test(buttonLabel: String) { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withText(buttonLabel)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }
24. Fancy: Test with ActivityScenario class JUnit5ActivityScenarioTest { @RegisterExtension val scenarioExtension = ActivityScenarioExtension.launch<MyActivity>() @ValueSource(strings = ["Login", "Register"]) @ParameterizedTest fun test(buttonLabel: String) { onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withText(buttonLabel)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) } }
25. Parameterized Argument Sources @ArgumentsSource @NullAndEmpty Source @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource
26. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource @ValueSource(ints = [1, 2, 3]) @ParameterizedTest fun testInts(value: Int) { // ... } @ValueSource(classes = [Foo::class, Bar::class, Baz::class]) @ParameterizedTest fun testClasses(value: Class<*>) { // ... }
27. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource @ValueSource @ValueSource(ints = [1, 2, 3]) @ParameterizedTest fun testInts(value: Int) { // ... } @ValueSource(classes = [Foo::class, Bar::class, Baz::class]) @ParameterizedTest fun testClasses(value: Class<*>) { // ... }
28. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource enum class Direction { North, East, South, West } @EnumSource(Direction::class) @ParameterizedTest fun testEnum(direction: Direction) { // ... }
29. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource @EnumSource enum class Direction { North, East, South, West } @EnumSource(Direction::class) @ParameterizedTest fun testEnum(direction: Direction) { // ... }
30. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource companion object { @JvmStatic fun arguments() = listOf( Arguments.of("First", Foo()), Arguments.of("Second", Bar()), Arguments.of("Third", Baz()), ) } @MethodSource("arguments") @ParameterizedTest fun testMethod(label: String, model: Base) { // ... }
31. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource @MethodSource companion object { @JvmStatic fun arguments() = listOf( Arguments.of("First", Foo()), Arguments.of("Second", Bar()), Arguments.of("Third", Baz()), ) } @MethodSource("arguments") @ParameterizedTest fun testMethod(label: String, model: Base) { // ... }
32. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource companion object { @JvmStatic val argumentsField = listOf(1234, 777, 573) } @FieldSource("argumentsField") @ParameterizedTest fun testField(number: Int) { // ... }
33. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @FieldSource companion object { @JvmStatic val argumentsField = listOf(1234, 777, 573) } @FieldSource("argumentsField") @ParameterizedTest fun testField(number: Int) { // ... }
34. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @CsvSource( "Alice, 34, f, true", "Bob, 16, m, false", "Charlie, 45, m, true", ) @ParameterizedTest fun testCsv(name: String, age: Int, gender: Char, expectedAdult: Boolean) { val person = Person(name, age, gender) assertEquals(expectedAdult, person.isAdult) }
35. @ArgumentsSource @NullAndEmpty Source Parameterized Argument Sources @NullSource @EmptySource @CsvFileSource @CsvSource @CsvSource( "Alice, 34, f, true", "Bob, 16, m, false", "Charlie, 45, m, true", ) @ParameterizedTest fun testCsv(name: String, age: Int, gender: Char, expectedAdult: Boolean) { val person = Person(name, age, gender) assertEquals(expectedAdult, person.isAdult) }
36. Testing Compose UI works, too.
37. Compose UI Test with JUnit 5 @RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() } @RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() }
38. @RegisterExtension val composeExtension = createComposeExtension() @CsvSource( // Dark mode | Font scale " false, 1.0", " false, 2.0", " true, 1.0", " true, 2.0", ) @ParameterizedTest(name = "dark={0}, font={1}") fun test(darkTheme: Boolean, fontScale: Float) = composeExtension.use { setContent { DK24ShowcaseTheme( darkTheme = darkTheme, fontScale = fontScale, ) { Greeting("DroidKaigi", onNameUpdated = {}) } } onNodeWithText("Hello DroidKaigi").assertExists() } Compose UI Test with JUnit 5
39. Conditional Test Execution
40. Conditional Test Execution • Tests can be skipped when certain conditions apply, e.g.: • Environment variables • System properties • Operating system • android-junit5 adds more conditions, e.g.: • Build manufacturer • BuildConfig value • API level
41. Conditional Annotations @Test @DisabledIfEnvironmentVariable(named = "CI", matches = "true") fun testOnlyOnLocal() { // ... } @Test @EnabledIfEnvironmentVariable(named = "CI", matches = "true") fun testOnlyOnCI() { // ... }
42. Conditional Annotations @Test @EnabledOnManufacturer(["samsung"]) fun testOnlyOnSamsungDevices() { // ... } @Test @DisabledOnManufacturer(["google"]) fun testNotOnGoogleDevices() { // ... }
43. Conditional Annotations @Test @EnabledIfBuildConfigValue(named = "DEBUG", matches = "true") fun testInDebugBuilds() { // ... } @Test @DisabledIfBuildConfigValue(named = "FLAVOR", matches = "prod") fun testNotInProd() { // ... }
44. Conditional Annotations @Test @EnabledOnSdkVersion(from = 26, until = 30) fun testFromAndroid8UntilAndroid11() { // ... } @Test @DisabledOnSdkVersion(from = 33) fun testOnlyBelowAndroid13() { // ... }
45. Parallel Test Execution
46. Parallel Test Execution • Can greatly reduce test execution time • All unit tests can (theoretically) be parallelized • For instrumentation tests • UI tests (Espresso, Compose) are not supported • Non-UI tests (e.g. Realm database) are supported
47. Slow: Without Parallel Tests class ParallelTest { @RepeatedTest(5) fun test(info: RepetitionInfo) { val i = info.currentRepetition Thread.sleep(i * 1_000L) } }
48. Setting up Parallel Test Execution // app/build.gradle.kts junitPlatform { configurationParameter( "junit.jupiter.execution.parallel.enabled", "true" ) } // app/build.gradle.kts junitPlatform { configurationParameter( "junit.jupiter.execution.parallel.enabled", "true" ) }
49. Fast(er): With Parallel Tests @Execution(ExecutionMode.CONCURRENT) class ParallelTest { @RepeatedTest(5) fun test(info: RepetitionInfo) { val i = info.currentRepetition Thread.sleep(i * 1_000L) } }
50. How about Robolectric? It works! (but there is much to do)
51. Robolectric: An overview • Framework for running unit tests with Android APIs • Physical device not necessary • Simulation of hardware events for better tests • Other libraries (Roborazzi, Robospec) are built on top
52. Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
53. Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
54. Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner @RunWith(RobolectricTestRunner::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
55. Robolectric: An overview For JUnit 4 import android.content.Intent import android.net.Uri import org.junit.Assert.assertNotNull import org.junit.Test import org.junit.runner.RunWith import org.robolectric.RobolectricTestRunner @RunWith(RobolectricTestRunner::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
56. How does Robolectric work? • Robolectric injects a modified version of android.jar into the runtime classpath • When the ClassLoader loads an Android API, it uses Robolectric's version instead of the default stub • RobolectricTestRunner takes care of this injection
57. Problem: JUnit 5 did not support custom class loaders. For a long time.
59. uT
61. junit. -robolectric-extension • apter-tech/junit5-robolectric-extension • Class loader replacement logic for Robolectric & JUnit 5 • Still highly experimental, but works for basic cases
62. Robolectric with JUnit 5 import android.content.Intent import android.net.Uri import org.junit.jupiter.api.Assertions.assertNotNull import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.ExtendWith import tech.apter.junit.jupiter.robolectric.RobolectricExtension @ExtendWith(RobolectricExtension::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
63. Robolectric with JUnit 5 import android.content.Intent import android.net.Uri import org.junit.jupiter.api.Assertions.assertNotNull import org.junit.jupiter.api.Test import org.junit.jupiter.api.extension.ExtendWith import tech.apter.junit.jupiter.robolectric.RobolectricExtension @ExtendWith(RobolectricExtension::class) class HogeTest { @Test fun checkIntent() { val intent = Intent().setData(Uri.parse("app://main?campaign=1")) assertNotNull(intent) } }
64. How about the future? We’re working on it!
65. What doesn't work (1) • junit5-robolectric-extension issue with Compose unit tests @ExtendWith(RobolectricExtension::class) class RobolectricComposeJUnit5Test { @RegisterExtension val composeExtension = createComposeExtension() @Test fun `robolectric compose test running with junit5`() { composeExtension.use { setContent { Text("Test") } onNodeWithText("Test").assertIsDisplayed() } } }
66. What doesn't work (1) • junit5-robolectric-extension issue with Compose unit tests @ExtendWith(RobolectricExtension::class) class RobolectricComposeJUnit5Test { @RegisterExtension val composeExtension = createComposeExtension() @Test fun `robolectric compose test running with junit5`() { composeExtension.use { setContent { Text("Test") } onNodeWithText("Test").assertIsDisplayed() } } }
67. What doesn't work (2) • Using Espresso in multiple unit tests causes a deadlock @ExtendWith(RobolectricExtension::class) class RobolectricEspressoJUnit5Test { @Test fun `robolectric espresso test #1 running with junit5`() { val scenario = ActivityScenario.launch(ViewActivity::class.java) onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) scenario.close() } @Test fun `robolectric espresso test #2 running with junit5`() { // ... } }
68. What doesn't work (2) • Using Espresso in multiple unit tests causes a deadlock @ExtendWith(RobolectricExtension::class) class RobolectricEspressoJUnit5Test { @Test fun `robolectric espresso test #1 running with junit5`() { val scenario = ActivityScenario.launch(ViewActivity::class.java) onView(withId(R.id.textView)).check(matches(isDisplayed())) onView(withId(R.id.button)).perform(click()) onView(withText("Updated")).check(matches(isDisplayed())) scenario.close() } @Test fun `robolectric espresso test #2 running with junit5`() { // ... } } will run forever...
69. What doesn't work (3) • Roborazzi & Robospec are not yet supported with JUnit 5
70. My testing recommendation in 2024 Unit Tests Instrumentation Tests Robolectric, Espresso, Compose Other Tests minSdk < 26 minSdk >= 26 (w/ Vintage Engine) + (slow adoption) 4 4
71. Marcel Schnelle @marcelschnelle A NEW ERA OF TESTING