Effective Android Testing - A value driven approach

Companion slides from my talk with same name.

Delivered in the following events

- PicPay Special Webinar (Online) - October/2021

Transcript

1. A value-driven approach Ubiratan Soares October / 2021 EFFECTIVE ANDROID TESTING
2. CHALLENGES FROM TESTING ANDROID APPS
3. Testing can be hard
4. Testing can be hard For Android projects, it can be even harder
8. CONCEPTS REVIEW
9. Dimensions of testing • Which type of code a test touches • Which boundaries of the system it touches • Which real flows a test exercises • Which level of isolation are in place (if any) • Etc
10. Unit Integration E2E + + Value Ensured Complexity Involved
11. Subject Under Test Input Output Classical understanding Single Class or Single Function
12. Entry point Exit point Not necessarily 1:1 relationship to classes / functions SUT
13. https://youtu.be/6ndAWzc2F-I
14. Entry point Exit point SUT Subject under test is not supposed to be the whole thing
15. Entry point Exit point SUT Boundary Owned by Others Owned by Others Abstractions you own within the app boundaries Boundary
16. Unit tests • Well-defined single entry point • Well-defined single exit point • Usually a small subject • Exercise abstractions you own • Not necessary 1:1 relationship to classes / functions • Never touches infrastructure ⚠
18. Entry point Boundary Owned by Others Owned by You Exit point
19. Entry point Boundary Owned by Others Owned by You Boundary Owned by Others Exit point
20. Entry point Boundary Owned by Others Owned by You Boundary Owned by Others Exit point
21. Entry point Boundary Owned by Others Owned by You Boundary Owned by Others Exit point
23. Integration tests • Exercise entry points and/or exit points beyond your application boundaries • Always enforce some degree of isolation (ideally fully isolated) • Categorized according the number of integrations under test • Categorized according user-value under test
24. Integration Level Rationale Single Exercises only one well-defined system boundary Multiple Exercises several well-defined system boundaries Component Exercises one well-defined component (user perspective) Acceptance Exercises several components with real user flows
25. E2E tests • Black-box instrumentation over a deliverable artefact • Exercise user flows over a pre-production environment • The most expensive ones to write + maintain
26. AN EFFECTIVE STRATEGY
27. Expectations 😎 Code
28. Reality 😅 Code
29. Bussines Rules App Micro service Nano service Monolith
30. App Realization of value Enabler of value != To avoid
31. app Product onboarding login search … Features Platform core-persistance core-rest core-location core-translation common-android common-di common-assets …
32. Entrypoint / Glue Features Platform Codebase contribution ** Practical observations from previous projects
33. “If the vast majority of a codebase from a Mobile product deals with coordination over middleware and/or infrastructure, how do we do unit testing ?”
34. “Write tests. Not too many. Mostly integration” Kent Dodds https://kentcdodds.com/blog/write-tests
35. https://youtu.be/Fha2bVoC8SE
36. https://martinfowler.com/articles/2021-test-shapes.html
37. Integration Unit E2E Contribution to Product Codebase Contribution to Test Suite
38. Product Features Platform Unit tests (stable core) Integration tests Unit tests (on-demand) Contract tests Component tests Screenshot tests E2E tests Acceptance tests
39. Unit Tests that bring value Tests for your implementation for HTTP authentication (eg, Oauth)
40. Unit Tests that bring value Tests for your retry- strategy over HTTP (eg, exponential backoff)
41. Unit Tests that bring value Tests over non-trivial validations over data provided by users via your App (eg, regexes)
42. Unit Tests that bring value Tests over non-trivial data parsing that happens at Mobile level (eg URI / DeepLinks)
43. Unit Tests that don’t bring value Tests over REST integration by mocking a Retrofit API
44. Unit Tests that don’t bring value Tests over ViewModels by mocking its colaborators
45. Unit Tests that don’t bring value Tests over trivial data mappers used within your abstractions
46. Unit Tests that don’t bring value Tests over sandwitch layers
47. Are you ready to let Mockito go ??? Android Engineer
48. Integrated tests that bring value Comprehenstive tests over data sources, exercising thirdy-party transformation / serialization engines alongside with your own error handlers and/or data transformers
49. class FactsDataSourceTests { @get:Rule val restInfrastructure = RestInfrastructureRule() private lateinit var dataSource: FactsDataSource @Before fun `before each test`() { val api = restInfrastructure.server.wireRestApi() dataSource = FactsDataSource(api) } @Test fun `should handle no results properly`() { restInfrastructure.restScenario( status = 200, response = loadFile("200_search_no_results.json") ) val noFacts = emptyList<ChuckNorrisFact>() assertThat(simpleSearch()).isEqualTo(noFacts) } https://github.com/dotanuki-labs/norris
50. class FactsDataSourceTests { @get:Rule val restInfrastructure = RestInfrastructureRule() private lateinit var dataSource: FactsDataSource @Before fun `before each test`() { val api = restInfrastructure.server.wireRestApi() dataSource = FactsDataSource(api) } @Test fun `should handle no results properly`() { restInfrastructure.restScenario( status = 200, response = loadFile("200_search_no_results.json") ) val noFacts = emptyList<ChuckNorrisFact>() assertThat(simpleSearch()).isEqualTo(noFacts) } https://github.com/dotanuki-labs/norris
51. class FactsDataSourceTests { @get:Rule val restInfrastructure = RestInfrastructureRule() private lateinit var dataSource: FactsDataSource @Before fun `before each test`() { val api = restInfrastructure.server.wireRestApi() dataSource = FactsDataSource(api) } @Test fun `should handle no results properly`() { restInfrastructure.restScenario( status = 200, response = loadFile("200_search_no_results.json") ) val noFacts = emptyList<ChuckNorrisFact>() assertThat(simpleSearch()).isEqualTo(noFacts) } https://github.com/dotanuki-labs/norris
52. @Test fun `should handle downstream error`() { restInfrastructure.restScenario(status = 500) assertErrorTransformed( whenRunning = this::simpleSearch, expected = RemoteServiceIntegrationError.RemoteSystem ) } https://github.com/dotanuki-labs/norris
53. @Test fun `should fetch facts with valid query term`() { restInfrastructure.restScenario( status = 200, response = loadFile("200_search_with_results.json") ) val facts = listOf( ChuckNorrisFact( id = "2wzginmks8azrbaxnamxdw", shareableUrl = "https://api.chucknorris.io/jokes/2wzginmks", textual = "Chuck Norris divides by zero”, ) ) assertThat(simpleSearch()).isEqualTo(facts) } } https://github.com/dotanuki-labs/norris
54. Integrated tests that bring value Component tests over entire screens, capturing the asynchonicity of data operations without coupling with implementation details (eg, Coroutines)
55. Activity View
56. Activity Screen Interface View or Wrapper
57. Activity Screen Interface View or Wrapper Test Abstraction
58. @RunWith(AndroidJUnit4::class) class FactsActivityTests { private lateinit var screen: FakeFactsScreen @get:Rule val restInfrastructure = RestInfrastructureRule() @Before fun `before each test`() { // Switching over fake screen wrapper with DI support val testApp = TestApplication.setupWith( factsModule, factsTestModule, RestInfrastructureTestModule(restInfrastructure.server) ) screen = testApp.factsScreen() } https://github.com/dotanuki-labs/norris
59. @Test fun `at first lunch, should start on empty state`() { whenActivityResumed<FactsActivity> { assertThat(screen.isLinked).isTrue() val expectedStates = listOf(Idle, Loading, FactsScreenState.Empty) assertThat(screen.trackedStates).isEqualTo(expectedStates) } } // More tests @Test fun `when remote service fails, should display the error`() { restInfrastructure.restScenario(status = 503) PersistanceHelper.registerNewSearch("code") whenActivityResumed<FactsActivity> { val error = Failed(RemoteServiceIntegrationError.RemoteSystem) val expectedStates = listOf(Idle, Loading, error) assertThat(screen.trackedStates).isEqualTo(expectedStates) } } } https://github.com/dotanuki-labs/norris
60. @Test fun `at first lunch, should start on empty state`() { whenActivityResumed<FactsActivity> { assertThat(screen.isLinked).isTrue() val expectedStates = listOf(Idle, Loading, FactsScreenState.Empty) assertThat(screen.trackedStates).isEqualTo(expectedStates) } } // More tests @Test fun `when remote service fails, should display the error`() { restInfrastructure.restScenario(status = 503) PersistanceHelper.registerNewSearch("code") whenActivityResumed<FactsActivity> { val error = Failed(RemoteServiceIntegrationError.RemoteSystem) val expectedStates = listOf(Idle, Loading, error) assertThat(screen.trackedStates).isEqualTo(expectedStates) } } } https://github.com/dotanuki-labs/norris
61. Screenshots tests over your UI delivery, decoupled from any data operations Integrated tests that bring value
62. Activity Screen Interface View or Wrapper Test Abstraction
63. Activity Screen Interface View or Wrapper Test Abstraction TestActivity
64. @LargeTest class FactsScreenshotTests : ScreenshotTest { private fun checkScreenshot(targetState: FactsScreenState) { val testActivity = prepareToCaptureScreenshot<FactsTestActivity> { target -> listOf(FactsScreenState.Idle, targetState).forEach { target.screen.updateWith(it) } } compareScreenshot(testActivity) } https://github.com/dotanuki-labs/norris
65. @Test fun emptyState() { checkScreenshot(Empty) } @Test fun errorState() { val error = RemoteServiceIntegrationError.RemoteSystem checkScreenshot(Failed(error)) } @Test fun successState() { val facts = listOf( FactDisplayRow( url = "https://chucknorris.io/jokes/998877", fact = "Chuck Norris merges before commit anything on Git", displayWithSmallerFontSize = false ) ) val presentation = FactsPresentation("humor", facts) checkScreenshot(Success(presentation)) } } https://github.com/dotanuki-labs/norris
66. @Test fun emptyState() { checkScreenshot(Empty) } @Test fun errorState() { val error = RemoteServiceIntegrationError.RemoteSystem checkScreenshot(Failed(error)) } @Test fun successState() { val facts = listOf( FactDisplayRow( url = "https://chucknorris.io/jokes/998877", fact = "Chuck Norris merges before commit anything on Git", displayWithSmallerFontSize = false ) ) val presentation = FactsPresentation("humor", facts) checkScreenshot(Success(presentation)) } } https://github.com/dotanuki-labs/norris
67. Integrated tests that bring value Acceptance tests over a fake ready-to-deliver artefact, exercising real user flows in isolation
68. @RunWith(AndroidJUnit4!"class) class NorrisAcceptanceTests { private val restInfrastructure = RestInfrastructureRule(customPort = 4242) private val stressedExecution = FlakyTestRule() @get:Rule val rules: RuleChain = RuleChain.outerRule(stressedExecution).around(restInfrastructure) @Before fun beforeEachTest() { PersistanceHelper.clearStorage() } https://github.com/dotanuki-labs/norris
69. @Repeat @Test fun shouldPerformFirstSearch_AfterFirstLunch_ByTypingATerm() { val searchTerm = "math" val suggestions = listOf("career", "dev", "humor") val fact = "Chuck Norris can divide by zero" val suggestionsPayload = RestDataBuilder.suggestionsPayload(suggestions) val factsPayload = RestDataBuilder.factsPayload(searchTerm, fact) restInfrastructure.run { restScenario(200, suggestionsPayload) restScenario(200, factsPayload) } https://github.com/dotanuki-labs/norris
70. startingFrom<FactsActivity> { checkEmptyState() clickOnSearchIcon() awaitTransition() checkSuggestions(suggestions) performSearch(searchTerm) awaitTransition() checkDisplayed(fact) } } } https://github.com/dotanuki-labs/norris
71. fun Project.applyAndroidApplicationConventions() { applyAndroidStandardConventions() val android = extensions.findByName("android") as ApplicationExtension android.apply { testBuildType = when { isTestMode() -> "release" else -> "debug" } defaultConfig { testInstrumentationRunner = AndroidDefinitions.instrumentationTestRunner } // More plugin-driven conventions } https://github.com/dotanuki-labs/norris
72. fun Project.applyAndroidApplicationConventions() { applyAndroidStandardConventions() val android = extensions.findByName("android") as ApplicationExtension android.apply { testBuildType = when { isTestMode() -> "release" else -> "debug" } defaultConfig { testInstrumentationRunner = AndroidDefinitions.instrumentationTestRunner } // More plugin-driven conventions } https://github.com/dotanuki-labs/norris
73. Integration Test Proposed Value Single integration over data sources Ensures data transformations + error handling over the target boundary Component Tests over entire screens Enables refactoring over target screen Screenshot Tests over entire screens Ensures association between state and Views without any assertions Acceptance Exercises several components with real user flows over a production-like artefact, emulating E2E at integration level
74. https://github.com/dotanuki-labs/norris
76. PLUS ULTRA !!!
77. Coverage on Sonarqube Actual tests
80. Standards help large teams since they reduce both accidental complexity and also cognitive load when switching between contexts inside the same codebase
81. Purpose Recommendation Reason Regular assertions Google Truth Max compatibility for JVM + Roboletric + Instrumentation Test Runner jUnit4 Junit5 is not supported for Android/Instrumentation Screenshot Tests Karumi/Shot Nice wrapper over Facebook tooling + useful add-ons Espresso Tests Barista Nice wrapper over Espresso tooling + useful add-ons Compose Tests Standard tooling Provided since day-zero Contract Tests Pact/JVM Most adopted solution E2E 👀 Up to you, as long it runs for as part of your pipeline
82. What really matters is focusing on writting good tests that allow Engineers to move faster by maximizing the correlation between test code and value proposition (ie, probability of catching bugs)
83. Software Engineer Quality Engineer
84. Team work makes the dream work!
85. FINAL REMARKS
86. TL;DR • Be aware of unit tests that don’t bring any value • If the case for you, invest energy on integration tests specially over Data Sources and Screen Components • There is no such thing like effective Espresso tests over non-release builds • Let Mocks fade away from your codebase (and your life!) • Mind the conciliation between your Test Strategy and your Quality Strategy
87. UBIRATAN SOARES Brazilian Computer Scientist Senior Software Engineer @ N26 GDE for Android and Kotlin @ubiratanfsoares ubiratansoares.dev
88. THANKS