Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Modern Testing on Android

Modern Testing on Android

If you still test your apps manually, when will you have any time left to develop features? Android is present in so many flavors and devices, that relying solely on manual testing is practically an act of faith. Luckily, the testing landscape is also evolving so it's getting easier to implement a solid testing strategy.

In this talk we’ll cover the modern way of testing your Android application. We will outline the different types of tests and when to use which, show how to implement your tests using Jetpack Compose, discuss synchronization issues and how to solve them, and show how you can run your tests automatically using a Continuous Integration system.

Jolanda Verhoef

July 11, 2022
Tweet

Transcript

  1. on Android Modern Testing Jolanda Verhoef Developer Relations Engineer Google

    she/her 
 @lojanda Jose Alcérreca Developer Relations Engineer Google he/him 
 @ppvi
  2. Agenda ANSI/IEEE 1059 A process of a software item analyzing

    in order to detect the discrepancies between actual and required conditions (that is errors/bugs/defects) and to estimate the software item features. • Business process-based testing • Operational profile testing • Requirements-based testing • Incident management tools • Non-functional test design techniques • Glass box testing
  3. Horror stories Modern Testing Strategies Regressions Manual QA Unit tests

    UI tests Screenshots Sync issues Using fakes Where to start
  4. fun isValid(emailAddress: String): Boolean { return Pattern.compile( "(?:[a-z0-9!#\$%&'*+/=?^_`{|}~-]+" + "(?:\\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*"

    + "|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@" + "(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+" + "[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:" + "[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])" ).matcher(emailAddress).find() } Email validator
  5. fun isValid(emailAddress: String): Boolean { return Pattern.compile( "(?:[a-z0-9!#\$%&'*+/=?^_`{|}~-]+" + "(?:\\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*"

    + "|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@" + "(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+" + "[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:" + "[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])" ).matcher(emailAddress).find() } Email validator fun isValid(emailAddress: String): Boolean { return Pattern.compile( "(?:[a-z0-9!#\$%&'*+/=?^_`{|}~-]+" + "(?:\\.[a-z0-9!#\$%&'*+/=?^_`{|}~-]+)*" + "|\"(?:[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21\\x23-\\x5b\\x5d-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])*\")@" + "(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\\.)+" + "[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\\[(?:" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}" + "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:" + "[\\x01-\\x08\\x0b\\x0c\\x0e-\\x1f\\x21-\\x5a\\x53-\\x7f]|\\" + "\\[\\x01-\\x09\\x0b\\x0c\\x0e-\\x7f])+)\\])" ).matcher(emailAddress).find() }
  6. 1 2 3 4 5 Jul 2 Jul 3 Jul

    4 Jul 5 Jul 6 Google Play Rating
  7. class EmailValidatorTest { @Test fun emailAddress_default_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test

    fun emailAddress_withDotsInLocalPart_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test fun emailAddress_withoutDomain_isInvalid() { assertFalse(EmailValidator().isValid("person@example")) } }
  8. class EmailValidatorTest { @Test fun emailAddress_default_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test

    fun emailAddress_withDotsInLocalPart_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test fun emailAddress_withoutDomain_isInvalid() { assertFalse(EmailValidator().isValid("person@example")) } @Test fun emailAddress_withMultiplePartsDomain_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } }
  9. Local Instrumented Smaller, faster Bigger, higher fidelity Integration tests Semantic

    UI tests Screenshot UI tests Performance tests Accessibility tests Unit tests
  10. Local Instrumented Smaller, faster Bigger, higher fidelity Integration tests Semantic

    UI tests Screenshot UI tests Performance tests Accessibility tests Unit tests
  11. class EmailValidatorTest { @Test fun emailAddress_default_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test

    fun emailAddress_withDotsInLocalPart_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } @Test fun emailAddress_withoutDomain_isInvalid() { assertFalse(EmailValidator().isValid("person@example")) } @Test fun emailAddress_withMultiplePartsDomain_isValid() { assertTrue(EmailValidator().isValid("[email protected]")) } }
  12. class LoginViewModelTest { @Test fun login_withValidEmail_returnsValidEmailUIState() { val viewModel =

    LoginViewModel( FakeEmailValidator(alwaysValid = true) ) viewModel.login(email = "whatevs", passwd = "hunter2") Assert.assertTrue(viewModel.uiState.validEmail) } }
  13. Local Instrumented Smaller, faster Bigger, higher fidelity Semantic UI tests

    Screenshot UI tests Performance tests Accessibility tests Unit tests Integration tests
  14. Local Instrumented Smaller, faster Bigger, higher fidelity Semantic UI tests

    Screenshot UI tests Performance tests Accessibility tests Unit tests Integration tests
  15. App

  16. API 27 3.19.4 API 28 3.22.0 API 30 3.28.0 API

    33 3.32.2 Integration test: SQLite compatibility
  17. Local Instrumented Smaller, faster Bigger, higher fidelity Screenshot UI tests

    Performance tests Accessibility tests Unit tests Integration tests Semantic UI tests
  18. Local Instrumented Smaller, faster Bigger, higher fidelity Screenshot UI tests

    Performance tests Accessibility tests Unit tests Integration tests Semantic UI tests
  19. Small • Render part of UI • Verify properties Large

    • Render app • Verify properties • Interact • Verify properties • Interact • Verify properties • Interact • Verify properties Medium • Render one screen • Verify properties • Interact • Verify properties
  20. Large • Render app • Verify properties • Interact •

    Verify properties • Interact • Verify properties • Interact • Verify properties
  21. class LoginScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun

    invalidEmail_disablesButton() { composeTestRule.setContent { MySootheTheme { LoginScreen() } } } }
  22. val composeTestRule = createComposeRule() @Test fun invalidEmail_disablesButton() { composeTestRule.setContent {

    MySootheTheme { LoginScreen() } } composeTestRule .onNodeWithText("LOG IN") .assertIsEnabled() } }
  23. composeTestRule.setContent { MySootheTheme { LoginScreen() } } composeTestRule .onNodeWithText("LOG IN")

    .assertIsEnabled() composeTestRule .onNodeWithText("Email address") .performTextInput("person@example") } }
  24. class LoginScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun

    invalidEmail_disablesButton() { composeTestRule.setContent { MySootheTheme { LoginScreen() } } composeTestRule .onNodeWithText("LOG IN") .assertIsEnabled() // src/androidTest/java/com/example/soothe
  25. class LoginScreenTest { @get:Rule val composeTestRule = createComposeRule() @Test fun

    invalidEmail_disablesButton() { composeTestRule.setContent { MySootheTheme { LoginScreen() } } composeTestRule .onNodeWithText("LOG IN") .assertIsEnabled() // src/test/java/com/example/soothe @RunWith(RobolectricTestRunner::class)
  26. Local Instrumented Integration tests Performance tests Accessibility tests Semantic UI

    tests Screenshot UI tests Unit tests Smaller, faster Bigger, higher fidelity Robolectric Compose Test APIs Espresso
  27. Local Instrumented Smaller, faster Bigger, higher fidelity Performance tests Accessibility

    tests Unit tests Integration tests Semantic UI tests Screenshot UI tests
  28. @Composable fun MyCard() { Surface( color = Color.Red, modifier =

    Modifier.testTag("Green background") // Lies ) { } }
  29. @Composable fun MyCard() { Surface( color = Color.Red, modifier =

    Modifier.testTag("Green background") // Lies ) { } } // LoginScreenTest.kt @Test fun loginButton_onStart_backgroundIsGreen() { composeTestRule.onNodeWithTag("Green background") }
  30. @Composable fun MyCard() { Surface( color = Color.Red, modifier =

    Modifier.semantics { customBackgroundProperty = Color.Green // Lies } ) { } }
  31. Local Instrumented Smaller, faster Bigger, higher fidelity Screenshot UI tests

    Performance tests Accessibility tests Unit tests Integration tests Semantic UI tests Compatibility testing
  32. class TopicScreenScreenshotTests { @get:Rule val composeTestRule = createAndroidComposeRule<ComponentActivity>() @get:Rule val

    dropshots = Dropshots() @Test fun test1() { composeTestRule.setContent { LoginScreen() } } }
  33. class TopicScreenScreenshotTests { @get:Rule val composeTestRule = createAndroidComposeRule<ComponentActivity>() @get:Rule val

    dropshots = Dropshots() @Test fun test1() { composeTestRule.setContent { LoginScreen() } dropshots.assertSnapshot(composeTestRule.activity, "test1") } }
  34. Local Instrumented Smaller, faster Bigger, higher fidelity Screenshot UI tests

    Performance tests Accessibility tests Unit tests Integration tests Semantic UI tests Compatibility testing
  35. Local Instrumented Smaller, faster Bigger, higher fidelity Screenshot UI tests

    Performance tests Accessibility tests Unit tests Integration tests Semantic UI tests Regression testing
  36. @Test fun invalidEmail_disablesButton() { composeTestRule .onNodeWithText("LOG IN") .assertIsEnabled() composeTestRule .onNodeWithText("Email

    address") .performTextInput("person@example") composeTestRule .onNodeWithText("LOG IN") .assertIsNotEnabled() } }
  37. 1. Idling Resources 2. Waiting for things 3. Replace components

    1. Idling Resources class AuthRepository(val authService: AuthService) { suspend fun login(email: String, password: String) { authService.login(email, password) } }
  38. 1. Idling Resources 2. Waiting for things 3. Replace components

    1. Idling Resources class AuthRepository(val authService: AuthService) { private val idlingResource = SimpleIdlingResource() suspend fun login(email: String, password: String) { authService.login(email, password) } }
  39. 1. Idling Resources 2. Waiting for things 3. Replace components

    1. Idling Resources class AuthRepository(val authService: AuthService) { private val idlingResource = SimpleIdlingResource() suspend fun login(email: String, password: String) { idlingResource.setIdleState(false) authService.login(email, password) idlingResource.setIdleState(true) } }
  40. 1. Idling Resources 2. Waiting for things 3. Replace components

    1. Idling Resources class AuthRepository(val authService: AuthService) { private val idlingResource = SimpleIdlingResource() suspend fun login(email: String, password: String) { idlingResource.setIdleState(false) authService.login(email, password) idlingResource.setIdleState(true) } } goo.gle/espresso-idling-resources
  41. 1. Idling Resources 2. Waiting for things 3. Replace components

    2. Waiting for things composeTestRule .onNodeWithText("LOG IN") .performClick()
  42. 1. Idling Resources 2. Waiting for things 3. Replace components

    2. Waiting for things composeTestRule .onNodeWithText("LOG IN") .performClick() // wait... composeTestRule .onNodeWithText("LOG IN") .assertIsNotEnabled()
  43. 1. Idling Resources 2. Waiting for things 3. Replace components

    2. Waiting for things composeTestRule .onNodeWithText("LOG IN") .performClick() Thread.sleep(2000) // don't do this composeTestRule .onNodeWithText("LOG IN") .assertIsNotEnabled()
  44. 1. Idling Resources 2. Waiting for things 3. Replace components

    2. Waiting for things composeTestRule .onNodeWithText("LOG IN") .performClick() // Wait until there's one element with a "LOG IN" text composeTestRule.waitUntil { composeTestRule .onAllNodesWithText("LOG IN") .fetchSemanticsNodes().size == 1 } composeTestRule .onNodeWithText("LOG IN") .assertIsNotEnabled()
  45. 1. Idling Resources 2. Waiting for things 3. Replace components

    2. Waiting for things composeTestRule .onNodeWithText("LOG IN") .performClick() composeTestRule.waitUntilExists(hasText("LOG IN")) composeTestRule .onNodeWithText("LOG IN") .assertIsNotEnabled() goo.gle/compose-test-sync
  46. 1. Idling Resources 2. Waiting for things 3. Replace components

    3. Replace components Server Email validator LoginViewModel
  47. 1. Idling Resources 2. Waiting for things 3. Replace components

    3. Replace components LoginViewModel Fake Email validator
  48. class MyTestSuite { @get:Rule var activityScenarioRule = activityScenarioRule<MyActivity>() @Test fun

    testEvent() { val scenario = activityScenarioRule.scenario ... } }
  49. @Module @TestInstallIn( components = [SingletonComponent::class], replaces = [AuthenticationModule::class] ) abstract

    class FakeAuthenticationModule { @Singleton @Binds abstract fun bindAuthService( fakeAuthService: FakeAuthService ): AuthService }
  50. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 1. Implement app architecture d.android.com/topic/architecture
  51. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 2. Configure CI service Build Local tests Instrumented tests CI Service
  52. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 3. Write unit tests UI Layer UI elements State holders Data Layer Repositories Data sources Domain Layer Use cases
  53. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 4. Screenshot tests for Previews Golden New Difference
  54. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 5. Write smoke tests Large • Render app • Verify properties • Interact • Verify properties • Interact • Verify properties • Interact • Verify properties
  55. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 6. Write tests for bugs Blimey! I can't log into the app. Rubbish. D. Turner July 6th, 2022
  56. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 6. Write tests for bugs class EmailValidatorTest { }
  57. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 6. Write tests for bugs class EmailValidatorTest { emailAddress_withMultiplePartsDomain_isValid() { assertTrue( EmailValidator().isValid(“[email protected]") ) } }
  58. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 6. Write tests for bugs class EmailValidatorTest { emailAddress_withMultiplePartsDomain_isValid() { assertTrue( EmailValidator().isValid(“[email protected]") ) } }
  59. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 7. Test everything else Step 1: Draw circles and lines Step 2: Finish drawing
  60. 1. Implement app architecture 2. Configure CI service 3. Write

    unit tests 4. Screenshot tests for Previews 5. Write smoke tests 6. Write tests for bugs 7. Test everything else 7. Test everything else Element Class Method Line home 80% 78% 69% login 95% 92% 88% profile 90% 83% 79% ui 75% 75% 70% Test coverage
  61. d.android.com/training/testing Testing documentation Testing tools documentation Testing Compose Compose testing

    cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Testing documentation
  62. d.android.com/studio/test Testing documentation Testing tools documentation Testing Compose Compose testing

    cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Testing tools documentation
  63. d.android.com/jetpack/compose/testing Testing documentation Testing tools documentation Testing your Compose layout

    Compose testing cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Testing Compose
  64. Testing documentation Testing tools documentation Testing your Compose layout Compose

    testing cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Compose testing cheat sheet
  65. onView(ViewMatcher) .perform(ViewAction) .check(ViewAssertion); isDisplayed() isCompletelyDisplayed() isEnabled() hasFocus() isClickable() isChecked() isNotChecked()

    withEffectiveVisibility(...) isSelected() UI PROPERTIES allOf(Matchers) anyOf(Matchers) is(...) not(...) endsWith(String) startsWith(String) OBJECT MATCHER withId(...) withText(...) withTagKey(...) withTagValue(...) hasContentDescription(...) withContentDescription(...) withHint(...) withSpinnerText(...) hasLinks() hasEllipsizedText() hasMultilineTest() View Matchers USER PROPERTIES withParent(Matcher) withChild(Matcher) hasDescendant(Matcher) isDescendantOfA(Matcher) hasSibling(Matcher) isRoot() HIERARCHY supportsInputMethods(...) hasIMEAction(...) INPUT isAssignableFrom(...) withClassName(...) CLASS isFocusable() isTouchable() isDialog() withDecorView() isPlatformPopup() ROOT MATCHERS Preference matchers Cursor matchers Layout matchers SEE ALSO click() doubleClick() longClick() pressBack() pressIMEActionButton() pressKey([int/EspressoKey]) pressMenuKey() closeSoftKeyboard() openLink() View Actions CLICK/PRESS scrollTo() swipeLeft() swipeRight() swipeUp() swipeDown() GESTURES clearText() typeText(String) typeTextIntoFocusedView(String) replaceText(String) TEXT matches(Matcher) doesNotExist() View Assertions POSITION ASSERTIONS onData(ObjectMatcher) .DataOptions .perform(ViewAction) .check(ViewAssertion); inAdapterView(Matcher) atPosition(Integer) onChildView(Matcher) Data Options Testing documentation Testing tools documentation Testing your Compose layout Compose testing cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Espresso cheat sheet
  66. Testing documentation Testing tools documentation Testing your Compose layout Compose

    testing cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Compose testing codelab
  67. Testing documentation Testing tools documentation Testing your Compose layout Compose

    testing cheat sheet Espresso cheat sheet Compose testing codelab Other testing code labs Other testing codelabs