Testing as a Culture

Testing as a Culture

Automated testing is a very important aspect of building any kind of software, including Android apps. There are many resources on the Internet on how to test our app and everybody has their own opinion on what is the most effective method. In Android, automated testing does not only mean unit tests, as there are different ways to automate the testing of our application. When building an app, setting up automated testing should be a culture, instead of an afterthought. However, there are still some teams who have very minimal tests and some don't even have any tests at all for their projects.
In this talk, I will discuss the mindset for testing an Android application. A mental model that we can apply when writing and improving our automated testing. I will also explore the types of tests that are available in an Android project and then present some techniques and tools that can help us structure our tests, such as the testing pyramid, Project Nitrogen, and Firebase Robo Test. In addition, I will talk about testing conventions and how it can help us write cleaner, more readable, and more maintainable test code. Lastly, I will share my experience of migrating a legacy codebase to a new architecture that is more scalable and testable.

6338c8fa4e2e6325094fe30b1e9f9443?s=128

Malvin Sutanto

February 20, 2020
Tweet

Transcript

  1. 1.

    ©2020 Wantedly, Inc. Photo by Rami Al-zayat on Unsplash Testing

    as a Culture DroidKaigi 2020 Malvin Sutanto | @malvinsutanto
  2. 9.

    ©2020 Wantedly, Inc. Can I release this? What if it

    crashes? Will there be any bugs? Will there be any problems? Android application over the years
  3. 12.

    ©2020 Wantedly, Inc. Follow scenarios and validate the outcomes Time

    consuming Not very scalable Hard to test different devices and features Hard to simulate edge cases How to simulate network errors? Who should test your app? Manual QA team
  4. 14.

    ©2020 Wantedly, Inc. Know the code best Manual tests are

    necessary Is it productive? Still difficult to simulate edge cases Need to build and ship product to the users Who should test your app? Engineers
  5. 18.

    ©2020 Wantedly, Inc. Difficult to set up correctly Especially in

    an existing project Rewriting test code for any changes Slowing you down Automated test for an Android app
  6. 21.

    ©2020 Wantedly, Inc. Write once, run forever However many times

    you want at any time Automate process! Benefits of automated tests Scalable
  7. 22.

    ©2020 Wantedly, Inc. Consider this screen EditText for a message

    and a “Send” Button Type into the EditText and when the “Send” Button is pressed, send the message inside the EditText. If EditText is empty, the Button will be disabled. Benefits of automated tests Scalable
  8. 23.

    ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI()

    {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Scalable
  9. 24.

    ©2020 Wantedly, Inc. 10 0 Mins Mins Automated test Manual

    test Writing time Writing time Benefits of automated tests Scalable
  10. 25.

    ©2020 Wantedly, Inc. 10 2 Mins Mins Automated test Manual

    test Writing time Writing time + 1x execution time Benefits of automated tests Scalable
  11. 26.

    ©2020 Wantedly, Inc. 11 2 Mins Mins Automated test Manual

    test Writing time + 1x execution time Writing time + 1x execution time Benefits of automated tests Scalable
  12. 27.

    ©2020 Wantedly, Inc. 12 2 Mins Mins Automated test Manual

    test Writing time + 2x execution time Writing time + 1x execution time Benefits of automated tests Scalable
  13. 28.

    ©2020 Wantedly, Inc. 12 4 Mins Mins Automated test Manual

    test Writing time + 2x execution time Writing time + 2x execution time Benefits of automated tests Scalable
  14. 29.

    ©2020 Wantedly, Inc. 14 8 Mins Mins Automated test Manual

    test Writing time + 4x execution time Writing time + 4x execution time Benefits of automated tests Scalable
  15. 30.

    ©2020 Wantedly, Inc. 20 20 Mins Mins Automated test Manual

    test Writing time + 10x execution time Writing time + 10x execution time Benefits of automated tests Scalable
  16. 31.

    ©2020 Wantedly, Inc. 110 200 Mins Mins Automated test Manual

    test Writing time + 100x execution time Writing time + 100x execution time Benefits of automated tests Scalable
  17. 32.
  18. 34.

    ©2020 Wantedly, Inc. Reliably test network errors Especially with proper

    test doubles Simulate error responses Simulate response timing Benefits of automated tests Simulate edge cases
  19. 36.

    ©2020 Wantedly, Inc. Structured and descriptive test cases to describe

    use cases Benefits of automated tests Documentation and information sharing
  20. 37.

    ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI()

    {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Documentation and information sharing
  21. 38.

    ©2020 Wantedly, Inc. Structured and descriptive test cases to describe

    use cases Benefits of automated tests Documentation and information sharing Prevent silos of information Aid in code reviews
  22. 40.

    ©2020 Wantedly, Inc. Decouple based on usage Reinforce single responsibility

    pattern Shorten the feedback loop Increase development velocity Benefits of automated tests Write better software
  23. 44.

    ©2020 Wantedly, Inc. Located in src/test/java Runs in local machine’s

    JVM Relatively fast No Android framework’s API Automated tests for Android projects Local unit test
  24. 46.

    ©2020 Wantedly, Inc. Located in src/androidTest/java Runs on a device

    or an emulator Combined with Espresso Tests and validates UI components Relatively slow Automated tests for Android projects Instrumented test
  25. 48.

    ©2020 Wantedly, Inc. Uses Android framework’s API in local JVM

    environment 4.0+ supports JetPack’s androidx.tests APIs Shadow implementation of Android Framework APIs See: https://github.com/robolectric/robolectric Automated tests for Android projects Robolectric
  26. 50.

    ©2020 Wantedly, Inc. Write once, run everywhere Choose whether to

    run your test code As a local unit test As an instrumented test Automated tests for Android projects Project Nitrogen
  27. 51.

    ©2020 Wantedly, Inc. Project Nitrogen Setting up shared tests https://www.youtube.com/watch?v=VJi2vmaQe6w

    Automated tests for Android projects Project Nitrogen build.gradle android { sourceSets { String sharedTestDir = ‘src/sharedTest/java’ test { java.srcDir sharedTestDir } androidTest { java.srcDir sharedTestDir } } }
  28. 52.

    ©2020 Wantedly, Inc. import androidx.test.ext.junit.runners.AndroidJUnit4 !!... @RunWith(AndroidJUnit4!::class) class SampleFragmentTest {

    !!... } Automated tests for Android projects Project Nitrogen See: https://github.com/android/testing-samples/blob/master/ui/espresso/FragmentScenarioSample
  29. 54.

    ©2020 Wantedly, Inc. High level, black-box testing Set of instructions

    to interact with the device Access device’s state, click at views, press the back button, rotate screen, etc. Tests a device as a whole Access home screen and app drawers, or interact with other apps Automated tests for Android projects UIAutomator
  30. 56.

    ©2020 Wantedly, Inc. Simulates real user interactions Tests “flows” of

    an application Guide the interactions with roboscript and/or robo directives Automated tests for Android projects Firebase Robo Test https://medium.com/wantedly-engineering/ android-application-testing-with-firebase-robo-test-c674e1754298
  31. 59.
  32. 61.
  33. 62.

    ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to

    end tests Integration tests Repositories and utility classes
  34. 63.

    ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to

    end tests Fragments and ViewModels Repositories and utility classes
  35. 64.

    ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite Firebase Robo

    Test or UI Automator Fragments and ViewModels Repositories and utility classes
  36. 66.

    ©2020 Wantedly, Inc. Determine effort and type of tests Create

    a balanced test suite Increase developer’s productivity Find the correct amount of tests Less exhaustive tests at the top How? Structuring test suite
  37. 69.

    ©2020 Wantedly, Inc. Replacement objects With controlled behavior Create a

    predictable condition Fake and mock classes Structuring code for testability Test doubles
  38. 70.

    ©2020 Wantedly, Inc. Fake classes Easiest to create manually Actual

    object with “shortcut” implementations to act like the real implementation. interface UserRepository { suspend fun getUser(userId: Long): User } class FakeUserRepository: UserRepository { override suspend fun getUser(userId: Long): User { return User( id = userId, firstName = "John", lastName = "Smith" ) } } Structuring code for testability Test doubles
  39. 71.

    ©2020 Wantedly, Inc. Mock classes Answer to set calls Created

    using Mockito or MockK Respond to a set of method calls whose answers have been defined. Throw an exception if it encounters any unexpected calls that have not been set. Structuring code for testability Test doubles interface UserRepository { suspend fun getUser(userId: Long): User suspend fun deleteUser(userId: Long) } "// Create a mock UserRepository with MockK val mockRepo = mockk<UserRepository>() coEvery { mockRepo.getUser(any()) } answers { User( id = firstArg(), firstName = "John", lastName = "Smith" ) }
  40. 72.

    ©2020 Wantedly, Inc. Structuring code for testability Test doubles -

    Mock classes !//Usage launch { mockRepo.getUser(1L) "// User(1L, "John", "Smith") mockRepo.deleteUser(1L) "// Throws MockKException: no answer found for UserRepository.deleteUser }
  41. 74.

    ©2020 Wantedly, Inc. Decouple classes Allow for replacement with test

    doubles Use constructor’s arguments to pass in other classes or interfaces Avoid static getInstance methods Use Dependency Injection Framework Dagger Structuring code for testability Independent and isolated classes
  42. 75.

    ©2020 Wantedly, Inc. Structuring code for testability Independent and isolated

    classes val navController = mockk<NavController>() ""... fun launchFragment(): FragmentScenario<SampleFragment> { return launchFragmentInContainer(themeResId = R.style.AppTheme) { SampleFragment() }.onFragment { Navigation.setViewNavController(it.requireView(), navController) } } ""... verify { navController.navigate(R.id.toOtherFragment) } Test in isolation
  43. 77.

    ©2020 Wantedly, Inc. Start and finish at the same state

    Clean up DB, SharedPreferences, value flags, etc. Prevent side-effects and flakiness Might affect other test’s outcomes Properly set up and tear down tests Structuring code for testability Stateless tests
  44. 78.

    ©2020 Wantedly, Inc. class SampleFragmentTest { @Before fun setUp() {""...}

    @After fun tearDown() {""...} @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {""...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {""...} } Structuring code for testability Stateless tests }
  45. 79.

    ©2020 Wantedly, Inc. class SampleFragmentTest { @Rule @JvmField val testSchedulerRule

    = TestSchedulerRule() ""... () () () } Structuring code for testability Stateless tests }
  46. 80.

    ©2020 Wantedly, Inc. class TestSchedulerRule : TestRule { val testScheduler

    = TestScheduler() override fun apply(base: Statement, description: Description): Statement { return object : Statement() { override fun evaluate() { RxJavaPlugins.setIoSchedulerHandler { testScheduler } !!... try { base.evaluate() "// Test code will be run here. } finally { RxJavaPlugins.reset() ""... } } } } } Structuring code for testability Stateless tests Set up Test Tear down
  47. 82.

    ©2020 Wantedly, Inc. Avoid multi-threading Can cause side-effects and increase

    execution time significantly Testing with 3rd party libraries TestScheduler for RxJava InstantTaskExecutorRule for Architecture Components TestDispatcher and runBlockingTest for coroutines Structuring code for testability Synchronous tests
  48. 84.

    ©2020 Wantedly, Inc. A set of rules or patterns for

    writing tests for a project Page Title Page Subtitle
  49. 85.

    ©2020 Wantedly, Inc. Type of test Test method naming Usage

    of test doubles Rules to apply etc Testing conventions
  50. 88.

    ©2020 Wantedly, Inc. Consistency Easier to understand Page Title Page

    Subtitle easier to identify breaking changes higher productivity
  51. 90.
  52. 91.

    ©2020 Wantedly, Inc. Testing conventions Describes what is being tested

    Not too long Test one aspect of your app at a time Use words like given, when, then, etc Example
  53. 98.

    ©2020 Wantedly, Inc. Slow test can decrease productivity Long CI/CD

    pipeline execution time Can have financial impact Don’t try aim for 100% coverage Focus on testing the most important parts Other considerations Time
  54. 99.

    ©2020 Wantedly, Inc. Do your manual test! Manual testing is

    still important Page Title Page Subtitle
  55. 100.

    ©2020 Wantedly, Inc. Good for user interactions and animations Find

    edge cases, incorrect test doubles, etc Other considerations Manual test is important!
  56. 103.

    ©2020 Wantedly, Inc. Difficult to maintain Too complex, each part

    of the app required different test setup A lot of “magic” to inject test doubles Tightly coupled Hard to test in isolation Rewrite is not possible Making legacy code testable Understanding the problems
  57. 104.

    ©2020 Wantedly, Inc. Create a plan Determine the type of

    test and decide on a convention Page Title Page Subtitle
  58. 105.

    ©2020 Wantedly, Inc. Making legacy code testable Create a plan

    Firebase Robo Test Fragments and ViewModels ViewModels, Repositories, Helper classes
  59. 106.

    ©2020 Wantedly, Inc. Descriptive test method name Develop branch is

    always shippable Every Pull Request needs to have added/removed test code Test libraries for easier setup AndroidX Test, Robolectric, MockK, Custom JUnit rules, etc. Making legacy code testable Create a plan
  60. 115.

    ©2020 Wantedly, Inc. Making legacy code testable CI/CD pipeline setup

    + Fastlane Local Unit Tests JVM Instrumented Tests Firebase Test Lab Firebase Robo Test JaCoCo + Codecov Pull Request Nightly
  61. 116.

    ©2020 Wantedly, Inc. ~60% Code coverage1 Within a year Consistent

    releases New features and updates Making legacy code testable CI/CD pipeline setup -65% Monthly fatal crashes 1: New and migrated code
  62. 118.

    ©2020 Wantedly, Inc. Automated tests make development faster Makes team

    and project scalable Create test strategy and test conventions Everybody follows the same strategy and conventions Focus on writing maintainable tests Plan testable architecture early or create a migration plan. Projects will always get more complicated! Summary