Modern Android Testing

A top quality app is one that is well tested. Test verify the behavior you'd expect from the app and prevent regression as it evolves. A spin on the modern android development (MAD), this talk will focus on modern testing tooling and frameworks. You will learn about testing strategies and various types of testing, and tools that you can utilize to empower your testing strategy.

Transcript

1. Modern Android Testing (Vincent) Aung Kyaw Paing Dev @ thoughtworks Android GDE
2. Android Testing Program Input Output
3. Android Testing Program Input Output [ Henry, Nicole, James ] “Henry, Nicole and James”
4. Android Testing Program Input Output [ Henry, Nicole, James ] “Henry, Nicole and James” Arrange Act Assert
5. @Test fun `test returns readbale string given a list of name` { // Arrange val input = listOf("Henry", "Nicole", "James") val expected = "Henry, Nicole and James" // Act val output = formatNames(input) // Assert assertEqual(expected, output) }
6. Type of testing Android Testing
7. Testing Pyramid Type of Testing closer to real world Slower, more $$$ E2E UI Test Integration Test Unit Test Manual Test
8. Testing Pyramid Type of Testing closer to real world Slower, more $$$ A11y Test E2E Test Exploratory Test Smoke Test Screenshot Test Contract Tests Unit Tests Form Factor Tests I18n test Performance Test
9. Unit Tests Type of Testing
10. Unit Tests Unit Test Test smallest unit of your software. A unit could be a function, a class, or a component Sign up Button Login Page Username Text Field Forgot password button Login Component Password Text Field Login Button Input Fields
11. Unit Tests Unit Test Test smallest unit of your software. A unit could be a function, a class, or a component Sign up Button Login Page Username Text Field Forgot password button Login Component Password Text Field Login Button Input Fields
12. Unit Tests Unit Test Test smallest unit of your software. A unit could be a function, a class, or a component Sign up Button Login Page Username Text Field Forgot password button Login Component Password Text Field Login Button Input Fields
13. class LoginButton( val viewModel: ViewModel ) { val status = state("unknown") Button( onClick= { viewModel.login().catch { status = "Failure" } status = "Success" } ) Text(status) }
14. @Test fun `test login press success state` { // TODO: Arrange // Act performClick() // Assert assertIsDisplayed("Success") }
15. Test Double Unit Test
16. Mocking Test Double Technique to replacing a implementation of its dependencies to achieve a behavior you want
17. class LoginViewModel() : ViewModel() { fun onLoginClick() { //... } }
18. interface LoginViewModel { fun onLoginClick() }
19. @Test fun `test login press success state` { val mockViewModel = object: LoginViewModel { fun onLoginClick() { // Do Nothing } } // Act performClick() // Assert assertIsDisplayed("Success") }
20. @Test fun `test login press success state` { val mockViewModel = object: LoginViewModel { fun onLoginClick() { // Do Nothing } } // Act performClick() // Assert assertIsDisplayed("Success") } class LoginButton( val viewModel: ViewModel ) { val status = state("unknown") Button( onClick= { viewModel.login().catch { status = "Failure" } status = "Success" } ) Text(status) }
21. @Test fun `test login press success state` { val mockViewModel = object: LoginViewModel { fun onLoginClick() { // Do Nothing } } // Act render(LoginButton(mockViewModel)) performClick() // Assert assertIsDisplayed("Success") }
22. interface LoginViewModel { fun onLoginClick() }
23. interface LoginViewModel { fun onLoginClick() fun onSignUpClick() fun onNavigateBack() fun onOpenTermsAndConditions() ... }
24. val mockViewModel = object: LoginViewModel { fun onLoginClick() { ... } fun onSignUpClick() { ... } fun onNavigateBack() { ... } fun onOpenTermsAndConditions() { ... } }
25. @Test fun `test login press success state` { val mockViewModel = object: LoginViewModel { fun onLoginClick() { ... } fun onSignUpClick() { ... } fun onNavigateBack() { ... } fun onOpenTermsAndConditions() { ... } // Act render(LoginButton(mockViewModel)) performClick() // Assert assertIsDisplayed("Success") }
26. @Test fun `test login press success state` { val mockViewModel = mockk<LoginViewModel>() every { mockViewModel.onLoginClick() } answers { // Do Nothing } // Act render(LoginButton(mockViewModel)) performClick() // Assert assertIsDisplayed("Success") }
27. Mocking’s Drawbacks Test Double Every time a dependencies change the contract, the test breaks Brittle
28. class LoginViewModel { fun setUserName(username: String) { ... } fun setPassword(password: String) { ... } fun onLoginClick() { //... } }
29. @Test fun `test call login api on press` { val mockViewModel = object: LoginViewModel{ fun setUsername(username: String) { ... } fun setPassword(password: String) { ... } fun onLoginClick() { ... } } ... verify(times = 1) { mockLoginApi.execute(...) } }
30. class LoginViewModel { fun onLoginClick( username: String password: String ) { //... } }
31. @Test fun `test call login api on press` { val mockViewModel = object: LoginViewModel{ fun setUsername(username: String) { ... } fun setPassword(password: String) { ... } fun onLoginClick() { ... } } ... verify(times = 1) { mockLoginApi.execute(...) } }
32. Mocking Drawbacks Test Double Every time a dependencies change the contract, the test breaks Brittle Implementation Leak A test should avoid testing specific implementation but mocking do exactly that
33. @Test fun `test login press` { val mockViewModel = object: LoginViewModel{ fun setUsername(username: String) { ... } fun setPassword(password: String) { ... } fun onLoginClick() { ... } } ... verify(times = 1) { mockLoginApi.execute(...) } }
34. Mocking Drawbacks Unit Tests Every time a dependencies change the contract, the test breaks Brittle Implementation Leak Too much = Bad A test should avoid testing specific implementation but mocking do exactly that Mocking too much could leads to you not really testing anything at all since everything is a mock
35. class Auth( networkClient: NetworkClient, userStore: UserStore ) { fun login() { val res = networkClient.login() // Save locally userStore.put(res) } } @Test fun `test auth login`() { val auth = Auth( mockNetworkClient mockUserCache ) auth.login() // Assert verify { mockUserCache.put(mockRes) } }
36. Classicist Approach Unit Tests An approach to inside-out testing where we use test doubles(fake) instead of mocks to replicate actual behavior
37. class LoginButton( val viewModel: ViewModel ) { val status = state("unknown") Button( onClick= { viewModel.login().catch { status = "Failure" } status = "Success" } ) Text(status) }
38. class LoginButton( val viewModel: ViewModel ) { val status = state("unknown") Button( onClick= { viewModel.login().catch { status = "Failure" } status = "Success" } ) Text(status) }
42. class FakeLoginApi: LoginApi { val userList = listOf( User(id=0, name = "abc", pass="abc") ) fun login(name : String, pass: String) : User? { return userList.find { user -> user.name == name && user.pass == pass } } }
43. class FakeLoginApi: LoginApi { val userList = mutabeListOf( User(name = "abc", pass="abc") ) fun login(name : String, pass: String) : User? { return userList.find { user -> user.name == name && user.pass == pass } } fun signUp(name: Sring, pass: String) { userList.add(name, pass) } }
44. @Test fun `test login press success state` { val fakeLoginApi = FakeLoginApi() val viewModel = LoginViewModel(fakeLoginApi) // Act render(LoginButton(viewModel)) performClick() // Assert assertIsDisplayed("Success") }
45. Classicist Tradeoffs Unit Tests Test covers larger areas but missing out on details Larger test surface Flakiness Complex setup Fakes often replicate actual behavior, leading to unreliable tests if not used properly (e.g threading) Faking a part of your system is not easy and could take a chunk of your efforts
46. DEVICE FRAME - PIXELBOOK Mockist or Classicist It’s up to whatever approach make your team have confident in shipping your app Good reads: https://martinfowler.com/articles/mocksArentStubs.html https://martinfowler.com/articles/2021-test-shapes.html
47. Unit Tests Common Mistakes Unit Test More specific tests make the tests susceptible to breaking when production code changes with end result still being the same (e.g 5x2 = 10 and 7+3= 10) Testing specific implementation
48. Unit Tests Common Mistakes Unit Test More specific tests make the tests susceptible to breaking when production code changes with end result still being the same (e.g 5x2 = 10 and 7+3= 10) Testing specific implementation Testing libraries’ code Assume libraries are already tested. We should only test the code that we own.
49. Unit Tests Common Mistakes Unit Test More specific tests make the tests susceptible to breaking when production code changes with end result still being the same (e.g 5x2 = 10 and 7+3= 10) Testing specific implementation Testing libraries’ code Testing Configs Assume libraries are already tested. We should only test the code that we own. You are prone to making human mistakes the same way you’d make when writing a config file
50. Integration Tests Type of Testing
51. Integration Tests Type of Testing Test the interaction between your components and the Android system
52. Integration Tests Type of Testing Test the interaction between your components and the Android system Persistence Storage Notification Scheduler I18n Alarms
54. @RunWith(AndroidJUnit4::class) @Config(sdk = { MARSHPMALLOW, OREO }) class UserCacheTest { ... }
55. @RunWith(AndroidJUnit4::class) class UserCacheTest { @Test fun `save cache in json"() { // Arrange val sharedPref = getContext().sharedPref("user") val user = User() // Act userCache.put(user) // Assert val expected = user.toJson() val actual = sharedPref.get(user) Assert.assertEquals(expected, actual) } }
56. UI Tests Type of Testing
57. UI Tests Type of Testing Test and verify your UI interactions are correct. Most of the time, these are higher level tests. You’d want to use fakes instead of mocks
58. @RunWith(AndroidJUnit4::class) class LoginScreenTest { @Test fun `show success on pressing Login`() { // Arrange composeTestRule.setContent { LoginScreen() } // Act composeTestRule.onNodeWithText("Login").performClick() // Assert composeTestRule.onNodeWithText("Login Success!").assertIsDisplayed() } }
59. @RunWith(AndroidJUnit4::class) class LoginScreenTest { @Test fun `show success on pressing Login`() { // Arrange composeTestRule.setContent { LoginScreen() } // Act composeTestRule.onNodeWithText("Login").performClick() // Assert Espresso.onView(withText("Login Success!")).check(matches(isDisplayed())) } }
62. src/test/*** src/androidTest/*** Directory for Unit Tests & Robolectric Test Directory for instrumentation tests that execute on real devices (or emulators)
63. src/test/LoginScreenTest.kt @RunWith(AndroidJUnit4::class) class LoginScreenTest { @Test fun `show success on pressing Login`() { // Arrange composeTestRule.setContent { LoginScreen() } // Act composeTestRule.onNodeWithText("Login").performClick() // Assert composeTestRule.onNodeWithText("Login Success!").assertIsDisplayed() } }
64. src/androidTest/LoginScreenTest.kt class LoginScreenTest { @Test fun `show success on pressing Login`() { // Arrange composeTestRule.setContent { LoginScreen() } // Act composeTestRule.onNodeWithText("Login").performClick() // Assert composeTestRule.onNodeWithText("Login Success!").assertIsDisplayed() } }
65. UI Tests Type of Testing Real Device Fast to run Far from real world Slow to run Closet to real world
66. UI Tests Type of Testing Real Device Fast to run Far from real world Slow to run Closet to real world
67. Gradle Managed Devices Unit Tests Run tests in a scaleable way by utilizing gradle to manage the lifecycle of virtual devices. Available for API 27+ 🖥 PC 🐘 Gradle 📱Emulator
68. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
69. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
70. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
71. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
72. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
73. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "aosp" } } } } }
74. android { testOptions { managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel5api30").apply { device = "Pixel 5" apiLevel = 30 systemImageSource = "google" } } } } }
75. ./gradlew {Name}{BuildType}AndroidTest ./gradlew pixel5api30DebugAndroidTest
76. managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel3api30").apply { ... } maybeCreate<.ManagedVirtualDevice>("pixelTabletApi30").apply { ... } } groups { maybeCreate("phoneAndTablet").apply { targetDevices.add(devices["pixel3api30"]) targetDevices.add(devices["pixelTabletApi30"]) } } }
77. managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel3api30").apply { ... } maybeCreate<.ManagedVirtualDevice>("pixelTabletApi30").apply { ... } } groups { maybeCreate("phoneAndTablet").apply { targetDevices.add(devices["pixel3api30"]) targetDevices.add(devices["pixelTabletApi30"]) } } }
78. managedDevices { devices { maybeCreate<ManagedVirtualDevice>("pixel3api30").apply { ... } maybeCreate<.ManagedVirtualDevice>("pixelTabletApi30").apply { ... } } groups { maybeCreate("phoneAndTablet").apply { targetDevices.add(devices["pixel3api30"]) targetDevices.add(devices["pixelTabletApi30"]) } } }
79. ./gradlew {Name}Group{BuildType}AndroidTest ./gradlew phoneAndTabletGroupDebugAndroidTest
80. Automated Test Device (ATD) UI Tests Reduces CPU and Memory usage by avoiding rendering to a screen. However, it still supports screenshot testing Lightweight Optimized Consistent Apps and services that are not likely to be used are removed to improve performance. It’s still an emulator device and built from the same images, that you use in Android Studio, so test results are consistent across local and CI.
81. UI Tests
82. Firebase Test Lab UI Tests Scale your instrumentation tests with Firebase test matrix Instrumentation Robo Test Game Loop Automatically run tests as how user would behave with no code Run test in your native game engine
83. Firebase Console Firebase Test Labs
84. From Android Studio Firebase Test Labs
85. With Gradle Managed Devices Firebase Test Labs An experimental API that runs the ATDs on Firebase Test Labs with real devices instead of emulators. Available on Android Studio Canary Build
86. plugins { ... id("com.google.firebase.testlab") }
87. firebaseTestLab { managedDevices { create("ftlDevice") { // Use device from Firebase device = "Pixel3" apiLevel = 30 } } }
88. ./gradlew {Name}{BuildType}AndroidTest ./gradlew ftlDeviceDebugAndroidTest
89. Firebase Test Labs
90. Screenshot Test Type of Testing Also known as snapshot testing, these tests captures a copy of screenshot of the app and matches it against the copy to verify that UI has not broken yet. These tests allow you to iterate faster and make sure it cover edge cases like styling. screenshot-tests-for-android happo.io
91. @Test fun testScreenshot() { val scenario = launchActivity<MainActivity>() scenario.onActivity { compareScreenshot(it) } }
92. https://developer.android.com/studio/preview Android Gradle Plugin 8.2
93. E2E Tests Type of Testing
94. E2E Tests Type of Testing Verify user flow is working as intended from beginning to end on a real device in a production or production-like environment. This is the most expensive form of testing and most often used as a form of regression testing. Black Box Testing
95. UIAutomator Type of Testing Testing framework for cross-app testing across the system and installed apps. Most modern end-to-end testing frameworks use UIAutomator API underneath the hood.
96. val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()) // Go back to home device.pressHome() val gmail: UiObject2 = device.findObject(By.text("Gmail")) // Perform a click and wait until the app is opened. val opened: Boolean = gmail.clickAndWait(Until.newWindow(), 3000) assertThat(opened).isTrue()
97. E2E Tests Type of Testing wdio cypress detox maestro
98. appId: com.example.login --- - launchApp - tapOn: "username" - inputText: "Aung" - tapOn: "password" - inputText: "abcedfg" - tapOn: "Login"
99. Firebase Test Lab UI Tests Robo Test Automatically run tests as how user would behave with no code
100. E2E UI Test Integration Test Unit Test Manual Test
101. E2E UI Test Integration Test Unit Test Manual Test JUnit, Mockk, Mockito Local Location API
102. E2E UI Test Integration Test Unit Test Manual Test Robolectric Local JUnit, Mockk, Mockito Local Location API
103. E2E UI Test Integration Test Unit Test Manual Test Espresso, Compose UI Test Local through Gradle Managed Device / Firebase Test Lab Robolectric Local JUnit, Mockk, Mockito Local Location API
104. E2E UI Test Integration Test Unit Test Manual Test UIAutomator, Robo Script Real Device / Firebase Test Lab Espresso, Compose UI Test Local through Gradle Managed Device / Firebase Test Lab Robolectric Local JUnit, Mockk, Mockito Local Location API
105. Manual Testing Type of Testing
106. Firebase App Distribution Manual Testing Streamline your testing process by releasing builds through Firebase App Distribution and collect feedback easily
107. Accessibility test Manual Testing Accessibility, by nature, is a user experience and you can’t test user experience with a machine or an automated tool. These should be tested by an individual, preferably a person with disabilities A11y scanner
108. DEVICE FRAME - PIXELBOOK Checks Make sure your app stay compliant and up to date with policy changes in your market Sign up for early access at checks.google.com
109. Thank You aungkyawpaing.dev @vincentpaing