Testing as a Culture

Automated testing is a very important aspect of building any kind of software, including Android apps. There are many resources on the Internet on how to test our app and everybody has their own opinion on what is the most effective method. In Android, automated testing does not only mean unit tests, as there are different ways to automate the testing of our application. When building an app, setting up automated testing should be a culture, instead of an afterthought. However, there are still some teams who have very minimal tests and some don't even have any tests at all for their projects.
In this talk, I will discuss the mindset for testing an Android application. A mental model that we can apply when writing and improving our automated testing. I will also explore the types of tests that are available in an Android project and then present some techniques and tools that can help us structure our tests, such as the testing pyramid, Project Nitrogen, and Firebase Robo Test. In addition, I will talk about testing conventions and how it can help us write cleaner, more readable, and more maintainable test code. Lastly, I will share my experience of migrating a legacy codebase to a new architecture that is more scalable and testable.

Transcript

1. ©2020 Wantedly, Inc. Photo by Rami Al-zayat on Unsplash Testing as a Culture DroidKaigi 2020 Malvin Sutanto | @malvinsutanto
2. ©2020 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android, Kotlin. Twitter/Medium: @malvinsutanto
3. ©2020 Wantedly, Inc. Android application
4. ©2020 Wantedly, Inc. Android application over the years !
5. ©2020 Wantedly, Inc. ! Android application over the years
6. ©2020 Wantedly, Inc. !" Android application over the years
7. ©2020 Wantedly, Inc. "# Android application over the years
8. ©2020 Wantedly, Inc. !$%& Android application over the years
9. ©2020 Wantedly, Inc. Can I release this? What if it crashes? Will there be any bugs? Will there be any problems? Android application over the years
10. ©2020 Wantedly, Inc. Who should test your app?
11. ©2020 Wantedly, Inc. Manual QA team Page Title Page Subtitle
12. ©2020 Wantedly, Inc. Follow scenarios and validate the outcomes Time consuming Not very scalable Hard to test different devices and features Hard to simulate edge cases How to simulate network errors? Who should test your app? Manual QA team
13. ©2020 Wantedly, Inc. Engineers )* Page Title Page Subtitle
14. ©2020 Wantedly, Inc. Know the code best Manual tests are necessary Is it productive? Still difficult to simulate edge cases Need to build and ship product to the users Who should test your app? Engineers
15. ©2020 Wantedly, Inc. End users + Page Title Page Subtitle
16. ©2020 Wantedly, Inc. + Page Title Page Subtitle
17. ©2020 Wantedly, Inc. Automated test for an Android app
18. ©2020 Wantedly, Inc. Difficult to set up correctly Especially in an existing project Rewriting test code for any changes Slowing you down Automated test for an Android app
19. ©2020 Wantedly, Inc. Why automated tests
20. ©2020 Wantedly, Inc. Scalable Page Title Page Subtitle
21. ©2020 Wantedly, Inc. Write once, run forever However many times you want at any time Automate process! Benefits of automated tests Scalable
22. ©2020 Wantedly, Inc. Consider this screen EditText for a message and a “Send” Button Type into the EditText and when the “Send” Button is pressed, send the message inside the EditText. If EditText is empty, the Button will be disabled. Benefits of automated tests Scalable
23. ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Scalable
24. ©2020 Wantedly, Inc. 10 0 Mins Mins Automated test Manual test Writing time Writing time Benefits of automated tests Scalable
25. ©2020 Wantedly, Inc. 10 2 Mins Mins Automated test Manual test Writing time Writing time + 1x execution time Benefits of automated tests Scalable
26. ©2020 Wantedly, Inc. 11 2 Mins Mins Automated test Manual test Writing time + 1x execution time Writing time + 1x execution time Benefits of automated tests Scalable
27. ©2020 Wantedly, Inc. 12 2 Mins Mins Automated test Manual test Writing time + 2x execution time Writing time + 1x execution time Benefits of automated tests Scalable
28. ©2020 Wantedly, Inc. 12 4 Mins Mins Automated test Manual test Writing time + 2x execution time Writing time + 2x execution time Benefits of automated tests Scalable
29. ©2020 Wantedly, Inc. 14 8 Mins Mins Automated test Manual test Writing time + 4x execution time Writing time + 4x execution time Benefits of automated tests Scalable
30. ©2020 Wantedly, Inc. 20 20 Mins Mins Automated test Manual test Writing time + 10x execution time Writing time + 10x execution time Benefits of automated tests Scalable
31. ©2020 Wantedly, Inc. 110 200 Mins Mins Automated test Manual test Writing time + 100x execution time Writing time + 100x execution time Benefits of automated tests Scalable
32. ©2020 Wantedly, Inc. Benefits of automated tests Scalable Time Number of executions Automated tests Manual tests
33. ©2020 Wantedly, Inc. Simulate edge cases Page Title Page Subtitle
34. ©2020 Wantedly, Inc. Reliably test network errors Especially with proper test doubles Simulate error responses Simulate response timing Benefits of automated tests Simulate edge cases
35. ©2020 Wantedly, Inc. Documentation and information sharing Page Title Page Subtitle
36. ©2020 Wantedly, Inc. Structured and descriptive test cases to describe use cases Benefits of automated tests Documentation and information sharing
37. ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Documentation and information sharing
38. ©2020 Wantedly, Inc. Structured and descriptive test cases to describe use cases Benefits of automated tests Documentation and information sharing Prevent silos of information Aid in code reviews
39. ©2020 Wantedly, Inc. Write better software Page Title Page Subtitle
40. ©2020 Wantedly, Inc. Decouple based on usage Reinforce single responsibility pattern Shorten the feedback loop Increase development velocity Benefits of automated tests Write better software
41. ©2020 Wantedly, Inc. Release updates with confidence Page Title Page Subtitle
42. ©2020 Wantedly, Inc. Automated test for Android projects Tools and frameworks
43. ©2020 Wantedly, Inc. Local unit test
44. ©2020 Wantedly, Inc. Located in src/test/java Runs in local machine’s JVM Relatively fast No Android framework’s API Automated tests for Android projects Local unit test
45. ©2020 Wantedly, Inc. Instrumented test
46. ©2020 Wantedly, Inc. Located in src/androidTest/java Runs on a device or an emulator Combined with Espresso Tests and validates UI components Relatively slow Automated tests for Android projects Instrumented test
47. ©2020 Wantedly, Inc. Robolectric
48. ©2020 Wantedly, Inc. Uses Android framework’s API in local JVM environment 4.0+ supports JetPack’s androidx.tests APIs Shadow implementation of Android Framework APIs See: https://github.com/robolectric/robolectric Automated tests for Android projects Robolectric
49. ©2020 Wantedly, Inc. Project Nitrogen Page Title Page Subtitle
50. ©2020 Wantedly, Inc. Write once, run everywhere Choose whether to run your test code As a local unit test As an instrumented test Automated tests for Android projects Project Nitrogen
51. ©2020 Wantedly, Inc. Project Nitrogen Setting up shared tests https://www.youtube.com/watch?v=VJi2vmaQe6w Automated tests for Android projects Project Nitrogen build.gradle android { sourceSets { String sharedTestDir = ‘src/sharedTest/java’ test { java.srcDir sharedTestDir } androidTest { java.srcDir sharedTestDir } } }
52. ©2020 Wantedly, Inc. import androidx.test.ext.junit.runners.AndroidJUnit4 !!... @RunWith(AndroidJUnit4!::class) class SampleFragmentTest { !!... } Automated tests for Android projects Project Nitrogen See: https://github.com/android/testing-samples/blob/master/ui/espresso/FragmentScenarioSample
53. ©2020 Wantedly, Inc. UIAutomator
54. ©2020 Wantedly, Inc. High level, black-box testing Set of instructions to interact with the device Access device’s state, click at views, press the back button, rotate screen, etc. Tests a device as a whole Access home screen and app drawers, or interact with other apps Automated tests for Android projects UIAutomator
55. ©2020 Wantedly, Inc. Firebase Robo Test
56. ©2020 Wantedly, Inc. Simulates real user interactions Tests “flows” of an application Guide the interactions with roboscript and/or robo directives Automated tests for Android projects Firebase Robo Test https://medium.com/wantedly-engineering/ android-application-testing-with-firebase-robo-test-c674e1754298
57. ©2020 Wantedly, Inc. Structuring test suite For an Android project
58. ©2020 Wantedly, Inc. Testing Pyramid
59. ©2020 Wantedly, Inc. Number of components Fidelity Execution time Maintenance effort Testing Pyramid Structuring test suite
60. ©2020 Wantedly, Inc. Number of tests Testing Pyramid Structuring test suite
61. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Integration tests Unit tests
62. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Integration tests Repositories and utility classes
63. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Fragments and ViewModels Repositories and utility classes
64. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite Firebase Robo Test or UI Automator Fragments and ViewModels Repositories and utility classes
65. ©2020 Wantedly, Inc. How?
66. ©2020 Wantedly, Inc. Determine effort and type of tests Create a balanced test suite Increase developer’s productivity Find the correct amount of tests Less exhaustive tests at the top How? Structuring test suite
67. ©2020 Wantedly, Inc. Structuring code for testability
68. ©2020 Wantedly, Inc. Test doubles Fakes and mocks
69. ©2020 Wantedly, Inc. Replacement objects With controlled behavior Create a predictable condition Fake and mock classes Structuring code for testability Test doubles
70. ©2020 Wantedly, Inc. Fake classes Easiest to create manually Actual object with “shortcut” implementations to act like the real implementation. interface UserRepository { suspend fun getUser(userId: Long): User } class FakeUserRepository: UserRepository { override suspend fun getUser(userId: Long): User { return User( id = userId, firstName = "John", lastName = "Smith" ) } } Structuring code for testability Test doubles
71. ©2020 Wantedly, Inc. Mock classes Answer to set calls Created using Mockito or MockK Respond to a set of method calls whose answers have been defined. Throw an exception if it encounters any unexpected calls that have not been set. Structuring code for testability Test doubles interface UserRepository { suspend fun getUser(userId: Long): User suspend fun deleteUser(userId: Long) } "// Create a mock UserRepository with MockK val mockRepo = mockk<UserRepository>() coEvery { mockRepo.getUser(any()) } answers { User( id = firstArg(), firstName = "John", lastName = "Smith" ) }
72. ©2020 Wantedly, Inc. Structuring code for testability Test doubles - Mock classes !//Usage launch { mockRepo.getUser(1L) "// User(1L, "John", "Smith") mockRepo.deleteUser(1L) "// Throws MockKException: no answer found for UserRepository.deleteUser }
73. ©2020 Wantedly, Inc. Independent and isolated classes
74. ©2020 Wantedly, Inc. Decouple classes Allow for replacement with test doubles Use constructor’s arguments to pass in other classes or interfaces Avoid static getInstance methods Use Dependency Injection Framework Dagger Structuring code for testability Independent and isolated classes
75. ©2020 Wantedly, Inc. Structuring code for testability Independent and isolated classes val navController = mockk<NavController>() ""... fun launchFragment(): FragmentScenario<SampleFragment> { return launchFragmentInContainer(themeResId = R.style.AppTheme) { SampleFragment() }.onFragment { Navigation.setViewNavController(it.requireView(), navController) } } ""... verify { navController.navigate(R.id.toOtherFragment) } Test in isolation
76. ©2020 Wantedly, Inc. Stateless tests Page Title Page Subtitle
77. ©2020 Wantedly, Inc. Start and finish at the same state Clean up DB, SharedPreferences, value flags, etc. Prevent side-effects and flakiness Might affect other test’s outcomes Properly set up and tear down tests Structuring code for testability Stateless tests
78. ©2020 Wantedly, Inc. class SampleFragmentTest { @Before fun setUp() {""...} @After fun tearDown() {""...} @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {""...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {""...} } Structuring code for testability Stateless tests }
79. ©2020 Wantedly, Inc. class SampleFragmentTest { @Rule @JvmField val testSchedulerRule = TestSchedulerRule() ""... () () () } Structuring code for testability Stateless tests }
80. ©2020 Wantedly, Inc. class TestSchedulerRule : TestRule { val testScheduler = TestScheduler() override fun apply(base: Statement, description: Description): Statement { return object : Statement() { override fun evaluate() { RxJavaPlugins.setIoSchedulerHandler { testScheduler } !!... try { base.evaluate() "// Test code will be run here. } finally { RxJavaPlugins.reset() ""... } } } } } Structuring code for testability Stateless tests Set up Test Tear down
81. ©2020 Wantedly, Inc. Synchronous tests
82. ©2020 Wantedly, Inc. Avoid multi-threading Can cause side-effects and increase execution time significantly Testing with 3rd party libraries TestScheduler for RxJava InstantTaskExecutorRule for Architecture Components TestDispatcher and runBlockingTest for coroutines Structuring code for testability Synchronous tests
83. ©2020 Wantedly, Inc. Testing conventions
84. ©2020 Wantedly, Inc. A set of rules or patterns for writing tests for a project Page Title Page Subtitle
85. ©2020 Wantedly, Inc. Type of test Test method naming Usage of test doubles Rules to apply etc Testing conventions
86. ©2020 Wantedly, Inc. Can be different for each projects Page Title Page Subtitle
87. ©2020 Wantedly, Inc. But everybody agrees on the same conventions Page Title Page Subtitle
88. ©2020 Wantedly, Inc. Consistency Easier to understand Page Title Page Subtitle easier to identify breaking changes higher productivity
89. ©2020 Wantedly, Inc. Allows you to scale your team Page Title Page Subtitle
90. ©2020 Wantedly, Inc. Example test convention For describing test case with method name Page Title Page Subtitle
91. ©2020 Wantedly, Inc. Testing conventions Describes what is being tested Not too long Test one aspect of your app at a time Use words like given, when, then, etc Example
92. ©2020 Wantedly, Inc. fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() { … } Testing conventions Example Initial state Action Outcome
93. ©2020 Wantedly, Inc. CI/CD Pipeline Automate test further
94. ©2020 Wantedly, Inc. Run tests against the latest code Automatically detect breaking changes
95. ©2020 Wantedly, Inc. Ci/CD Pipeline Automate test further PR Trigger Run
96. ©2020 Wantedly, Inc. Other considerations When writing tests
97. ©2020 Wantedly, Inc. Time⏳ Page Title Page Subtitle Writing and execution time
98. ©2020 Wantedly, Inc. Slow test can decrease productivity Long CI/CD pipeline execution time Can have financial impact Don’t try aim for 100% coverage Focus on testing the most important parts Other considerations Time
99. ©2020 Wantedly, Inc. Do your manual test! Manual testing is still important Page Title Page Subtitle
100. ©2020 Wantedly, Inc. Good for user interactions and animations Find edge cases, incorrect test doubles, etc Other considerations Manual test is important!
101. ©2020 Wantedly, Inc. Making legacy code testable What we did and what we've learnt
102. ©2020 Wantedly, Inc. Understanding the problems
103. ©2020 Wantedly, Inc. Difficult to maintain Too complex, each part of the app required different test setup A lot of “magic” to inject test doubles Tightly coupled Hard to test in isolation Rewrite is not possible Making legacy code testable Understanding the problems
104. ©2020 Wantedly, Inc. Create a plan Determine the type of test and decide on a convention Page Title Page Subtitle
105. ©2020 Wantedly, Inc. Making legacy code testable Create a plan Firebase Robo Test Fragments and ViewModels ViewModels, Repositories, Helper classes
106. ©2020 Wantedly, Inc. Descriptive test method name Develop branch is always shippable Every Pull Request needs to have added/removed test code Test libraries for easier setup AndroidX Test, Robolectric, MockK, Custom JUnit rules, etc. Making legacy code testable Create a plan
107. ©2020 Wantedly, Inc. Migration process Page Title Page Subtitle
108. ©2020 Wantedly, Inc. MFHBDZ BQQ Making legacy code testable Legacy module
109. ©2020 Wantedly, Inc. BQQ MFHBDZ Making legacy code testable Legacy module
110. ©2020 Wantedly, Inc. TFBSDI POCPBSEJOH BQQ MFHBDZ Making legacy code testable Legacy module
111. ©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code testable Legacy module
112. ©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code testable Legacy module
113. ©2020 Wantedly, Inc. CI/CD infrastructure setup Page Title Page Subtitle
114. ©2020 Wantedly, Inc. + Making legacy code testable CI/CD pipeline setup Fastlane
115. ©2020 Wantedly, Inc. Making legacy code testable CI/CD pipeline setup + Fastlane Local Unit Tests JVM Instrumented Tests Firebase Test Lab Firebase Robo Test JaCoCo + Codecov Pull Request Nightly
116. ©2020 Wantedly, Inc. ~60% Code coverage1 Within a year Consistent releases New features and updates Making legacy code testable CI/CD pipeline setup -65% Monthly fatal crashes 1: New and migrated code
117. ©2020 Wantedly, Inc. Summary
118. ©2020 Wantedly, Inc. Automated tests make development faster Makes team and project scalable Create test strategy and test conventions Everybody follows the same strategy and conventions Focus on writing maintainable tests Plan testable architecture early or create a migration plan. Projects will always get more complicated! Summary
119. ©2020 Wantedly, Inc. Thank you! Page Title Page Subtitle