Testing as a Culture

Testing as a Culture

Automated testing is a very important aspect of building any kind of software, including Android apps. There are many resources on the Internet on how to test our app and everybody has their own opinion on what is the most effective method. In Android, automated testing does not only mean unit tests, as there are different ways to automate the testing of our application. When building an app, setting up automated testing should be a culture, instead of an afterthought. However, there are still some teams who have very minimal tests and some don't even have any tests at all for their projects.
In this talk, I will discuss the mindset for testing an Android application. A mental model that we can apply when writing and improving our automated testing. I will also explore the types of tests that are available in an Android project and then present some techniques and tools that can help us structure our tests, such as the testing pyramid, Project Nitrogen, and Firebase Robo Test. In addition, I will talk about testing conventions and how it can help us write cleaner, more readable, and more maintainable test code. Lastly, I will share my experience of migrating a legacy codebase to a new architecture that is more scalable and testable.

6338c8fa4e2e6325094fe30b1e9f9443?s=128

Malvin Sutanto

February 20, 2020
Tweet

Transcript

  1. ©2020 Wantedly, Inc. Photo by Rami Al-zayat on Unsplash Testing

    as a Culture DroidKaigi 2020 Malvin Sutanto | @malvinsutanto
  2. ©2020 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android,

    Kotlin. Twitter/Medium: @malvinsutanto
  3. ©2020 Wantedly, Inc. Android application

  4. ©2020 Wantedly, Inc. Android application over the years !

  5. ©2020 Wantedly, Inc. ! Android application over the years

  6. ©2020 Wantedly, Inc. !" Android application over the years

  7. ©2020 Wantedly, Inc. "# Android application over the years

  8. ©2020 Wantedly, Inc. !$%& Android application over the years

  9. ©2020 Wantedly, Inc. Can I release this? What if it

    crashes? Will there be any bugs? Will there be any problems? Android application over the years
  10. ©2020 Wantedly, Inc. Who should test your app?

  11. ©2020 Wantedly, Inc. Manual QA team Page Title Page Subtitle

  12. ©2020 Wantedly, Inc. Follow scenarios and validate the outcomes Time

    consuming Not very scalable Hard to test different devices and features Hard to simulate edge cases How to simulate network errors? Who should test your app? Manual QA team
  13. ©2020 Wantedly, Inc. Engineers )* Page Title Page Subtitle

  14. ©2020 Wantedly, Inc. Know the code best Manual tests are

    necessary Is it productive? Still difficult to simulate edge cases Need to build and ship product to the users Who should test your app? Engineers
  15. ©2020 Wantedly, Inc. End users + Page Title Page Subtitle

  16. ©2020 Wantedly, Inc. + Page Title Page Subtitle

  17. ©2020 Wantedly, Inc. Automated test for an Android app

  18. ©2020 Wantedly, Inc. Difficult to set up correctly Especially in

    an existing project Rewriting test code for any changes Slowing you down Automated test for an Android app
  19. ©2020 Wantedly, Inc. Why automated tests

  20. ©2020 Wantedly, Inc. Scalable Page Title Page Subtitle

  21. ©2020 Wantedly, Inc. Write once, run forever However many times

    you want at any time Automate process! Benefits of automated tests Scalable
  22. ©2020 Wantedly, Inc. Consider this screen EditText for a message

    and a “Send” Button Type into the EditText and when the “Send” Button is pressed, send the message inside the EditText. If EditText is empty, the Button will be disabled. Benefits of automated tests Scalable
  23. ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI()

    {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Scalable
  24. ©2020 Wantedly, Inc. 10 0 Mins Mins Automated test Manual

    test Writing time Writing time Benefits of automated tests Scalable
  25. ©2020 Wantedly, Inc. 10 2 Mins Mins Automated test Manual

    test Writing time Writing time + 1x execution time Benefits of automated tests Scalable
  26. ©2020 Wantedly, Inc. 11 2 Mins Mins Automated test Manual

    test Writing time + 1x execution time Writing time + 1x execution time Benefits of automated tests Scalable
  27. ©2020 Wantedly, Inc. 12 2 Mins Mins Automated test Manual

    test Writing time + 2x execution time Writing time + 1x execution time Benefits of automated tests Scalable
  28. ©2020 Wantedly, Inc. 12 4 Mins Mins Automated test Manual

    test Writing time + 2x execution time Writing time + 2x execution time Benefits of automated tests Scalable
  29. ©2020 Wantedly, Inc. 14 8 Mins Mins Automated test Manual

    test Writing time + 4x execution time Writing time + 4x execution time Benefits of automated tests Scalable
  30. ©2020 Wantedly, Inc. 20 20 Mins Mins Automated test Manual

    test Writing time + 10x execution time Writing time + 10x execution time Benefits of automated tests Scalable
  31. ©2020 Wantedly, Inc. 110 200 Mins Mins Automated test Manual

    test Writing time + 100x execution time Writing time + 100x execution time Benefits of automated tests Scalable
  32. ©2020 Wantedly, Inc. Benefits of automated tests Scalable Time Number

    of executions Automated tests Manual tests
  33. ©2020 Wantedly, Inc. Simulate edge cases Page Title Page Subtitle

  34. ©2020 Wantedly, Inc. Reliably test network errors Especially with proper

    test doubles Simulate error responses Simulate response timing Benefits of automated tests Simulate edge cases
  35. ©2020 Wantedly, Inc. Documentation and information sharing Page Title Page

    Subtitle
  36. ©2020 Wantedly, Inc. Structured and descriptive test cases to describe

    use cases Benefits of automated tests Documentation and information sharing
  37. ©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI()

    {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Documentation and information sharing
  38. ©2020 Wantedly, Inc. Structured and descriptive test cases to describe

    use cases Benefits of automated tests Documentation and information sharing Prevent silos of information Aid in code reviews
  39. ©2020 Wantedly, Inc. Write better software Page Title Page Subtitle

  40. ©2020 Wantedly, Inc. Decouple based on usage Reinforce single responsibility

    pattern Shorten the feedback loop Increase development velocity Benefits of automated tests Write better software
  41. ©2020 Wantedly, Inc. Release updates with confidence Page Title Page

    Subtitle
  42. ©2020 Wantedly, Inc. Automated test for Android projects Tools and

    frameworks
  43. ©2020 Wantedly, Inc. Local unit test

  44. ©2020 Wantedly, Inc. Located in src/test/java Runs in local machine’s

    JVM Relatively fast No Android framework’s API Automated tests for Android projects Local unit test
  45. ©2020 Wantedly, Inc. Instrumented test

  46. ©2020 Wantedly, Inc. Located in src/androidTest/java Runs on a device

    or an emulator Combined with Espresso Tests and validates UI components Relatively slow Automated tests for Android projects Instrumented test
  47. ©2020 Wantedly, Inc. Robolectric

  48. ©2020 Wantedly, Inc. Uses Android framework’s API in local JVM

    environment 4.0+ supports JetPack’s androidx.tests APIs Shadow implementation of Android Framework APIs See: https://github.com/robolectric/robolectric Automated tests for Android projects Robolectric
  49. ©2020 Wantedly, Inc. Project Nitrogen Page Title Page Subtitle

  50. ©2020 Wantedly, Inc. Write once, run everywhere Choose whether to

    run your test code As a local unit test As an instrumented test Automated tests for Android projects Project Nitrogen
  51. ©2020 Wantedly, Inc. Project Nitrogen Setting up shared tests https://www.youtube.com/watch?v=VJi2vmaQe6w

    Automated tests for Android projects Project Nitrogen build.gradle android { sourceSets { String sharedTestDir = ‘src/sharedTest/java’ test { java.srcDir sharedTestDir } androidTest { java.srcDir sharedTestDir } } }
  52. ©2020 Wantedly, Inc. import androidx.test.ext.junit.runners.AndroidJUnit4 !!... @RunWith(AndroidJUnit4!::class) class SampleFragmentTest {

    !!... } Automated tests for Android projects Project Nitrogen See: https://github.com/android/testing-samples/blob/master/ui/espresso/FragmentScenarioSample
  53. ©2020 Wantedly, Inc. UIAutomator

  54. ©2020 Wantedly, Inc. High level, black-box testing Set of instructions

    to interact with the device Access device’s state, click at views, press the back button, rotate screen, etc. Tests a device as a whole Access home screen and app drawers, or interact with other apps Automated tests for Android projects UIAutomator
  55. ©2020 Wantedly, Inc. Firebase Robo Test

  56. ©2020 Wantedly, Inc. Simulates real user interactions Tests “flows” of

    an application Guide the interactions with roboscript and/or robo directives Automated tests for Android projects Firebase Robo Test https://medium.com/wantedly-engineering/ android-application-testing-with-firebase-robo-test-c674e1754298
  57. ©2020 Wantedly, Inc. Structuring test suite For an Android project

  58. ©2020 Wantedly, Inc. Testing Pyramid

  59. ©2020 Wantedly, Inc. Number of components Fidelity Execution time Maintenance

    effort Testing Pyramid Structuring test suite
  60. ©2020 Wantedly, Inc. Number of tests Testing Pyramid Structuring test

    suite
  61. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to

    end tests Integration tests Unit tests
  62. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to

    end tests Integration tests Repositories and utility classes
  63. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to

    end tests Fragments and ViewModels Repositories and utility classes
  64. ©2020 Wantedly, Inc. Testing Pyramid Structuring test suite Firebase Robo

    Test or UI Automator Fragments and ViewModels Repositories and utility classes
  65. ©2020 Wantedly, Inc. How?

  66. ©2020 Wantedly, Inc. Determine effort and type of tests Create

    a balanced test suite Increase developer’s productivity Find the correct amount of tests Less exhaustive tests at the top How? Structuring test suite
  67. ©2020 Wantedly, Inc. Structuring code for testability

  68. ©2020 Wantedly, Inc. Test doubles Fakes and mocks

  69. ©2020 Wantedly, Inc. Replacement objects With controlled behavior Create a

    predictable condition Fake and mock classes Structuring code for testability Test doubles
  70. ©2020 Wantedly, Inc. Fake classes Easiest to create manually Actual

    object with “shortcut” implementations to act like the real implementation. interface UserRepository { suspend fun getUser(userId: Long): User } class FakeUserRepository: UserRepository { override suspend fun getUser(userId: Long): User { return User( id = userId, firstName = "John", lastName = "Smith" ) } } Structuring code for testability Test doubles
  71. ©2020 Wantedly, Inc. Mock classes Answer to set calls Created

    using Mockito or MockK Respond to a set of method calls whose answers have been defined. Throw an exception if it encounters any unexpected calls that have not been set. Structuring code for testability Test doubles interface UserRepository { suspend fun getUser(userId: Long): User suspend fun deleteUser(userId: Long) } "// Create a mock UserRepository with MockK val mockRepo = mockk<UserRepository>() coEvery { mockRepo.getUser(any()) } answers { User( id = firstArg(), firstName = "John", lastName = "Smith" ) }
  72. ©2020 Wantedly, Inc. Structuring code for testability Test doubles -

    Mock classes !//Usage launch { mockRepo.getUser(1L) "// User(1L, "John", "Smith") mockRepo.deleteUser(1L) "// Throws MockKException: no answer found for UserRepository.deleteUser }
  73. ©2020 Wantedly, Inc. Independent and isolated classes

  74. ©2020 Wantedly, Inc. Decouple classes Allow for replacement with test

    doubles Use constructor’s arguments to pass in other classes or interfaces Avoid static getInstance methods Use Dependency Injection Framework Dagger Structuring code for testability Independent and isolated classes
  75. ©2020 Wantedly, Inc. Structuring code for testability Independent and isolated

    classes val navController = mockk<NavController>() ""... fun launchFragment(): FragmentScenario<SampleFragment> { return launchFragmentInContainer(themeResId = R.style.AppTheme) { SampleFragment() }.onFragment { Navigation.setViewNavController(it.requireView(), navController) } } ""... verify { navController.navigate(R.id.toOtherFragment) } Test in isolation
  76. ©2020 Wantedly, Inc. Stateless tests Page Title Page Subtitle

  77. ©2020 Wantedly, Inc. Start and finish at the same state

    Clean up DB, SharedPreferences, value flags, etc. Prevent side-effects and flakiness Might affect other test’s outcomes Properly set up and tear down tests Structuring code for testability Stateless tests
  78. ©2020 Wantedly, Inc. class SampleFragmentTest { @Before fun setUp() {""...}

    @After fun tearDown() {""...} @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {""...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {""...} } Structuring code for testability Stateless tests }
  79. ©2020 Wantedly, Inc. class SampleFragmentTest { @Rule @JvmField val testSchedulerRule

    = TestSchedulerRule() ""... () () () } Structuring code for testability Stateless tests }
  80. ©2020 Wantedly, Inc. class TestSchedulerRule : TestRule { val testScheduler

    = TestScheduler() override fun apply(base: Statement, description: Description): Statement { return object : Statement() { override fun evaluate() { RxJavaPlugins.setIoSchedulerHandler { testScheduler } !!... try { base.evaluate() "// Test code will be run here. } finally { RxJavaPlugins.reset() ""... } } } } } Structuring code for testability Stateless tests Set up Test Tear down
  81. ©2020 Wantedly, Inc. Synchronous tests

  82. ©2020 Wantedly, Inc. Avoid multi-threading Can cause side-effects and increase

    execution time significantly Testing with 3rd party libraries TestScheduler for RxJava InstantTaskExecutorRule for Architecture Components TestDispatcher and runBlockingTest for coroutines Structuring code for testability Synchronous tests
  83. ©2020 Wantedly, Inc. Testing conventions

  84. ©2020 Wantedly, Inc. A set of rules or patterns for

    writing tests for a project Page Title Page Subtitle
  85. ©2020 Wantedly, Inc. Type of test Test method naming Usage

    of test doubles Rules to apply etc Testing conventions
  86. ©2020 Wantedly, Inc. Can be different for each projects Page

    Title Page Subtitle
  87. ©2020 Wantedly, Inc. But everybody agrees on the same conventions

    Page Title Page Subtitle
  88. ©2020 Wantedly, Inc. Consistency Easier to understand Page Title Page

    Subtitle easier to identify breaking changes higher productivity
  89. ©2020 Wantedly, Inc. Allows you to scale your team Page

    Title Page Subtitle
  90. ©2020 Wantedly, Inc. Example test convention For describing test case

    with method name Page Title Page Subtitle
  91. ©2020 Wantedly, Inc. Testing conventions Describes what is being tested

    Not too long Test one aspect of your app at a time Use words like given, when, then, etc Example
  92. ©2020 Wantedly, Inc. fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() { … } Testing conventions

    Example Initial state Action Outcome
  93. ©2020 Wantedly, Inc. CI/CD Pipeline Automate test further

  94. ©2020 Wantedly, Inc. Run tests against the latest code Automatically

    detect breaking changes
  95. ©2020 Wantedly, Inc. Ci/CD Pipeline Automate test further PR Trigger

    Run
  96. ©2020 Wantedly, Inc. Other considerations When writing tests

  97. ©2020 Wantedly, Inc. Time⏳ Page Title Page Subtitle Writing and

    execution time
  98. ©2020 Wantedly, Inc. Slow test can decrease productivity Long CI/CD

    pipeline execution time Can have financial impact Don’t try aim for 100% coverage Focus on testing the most important parts Other considerations Time
  99. ©2020 Wantedly, Inc. Do your manual test! Manual testing is

    still important Page Title Page Subtitle
  100. ©2020 Wantedly, Inc. Good for user interactions and animations Find

    edge cases, incorrect test doubles, etc Other considerations Manual test is important!
  101. ©2020 Wantedly, Inc. Making legacy code testable What we did

    and what we've learnt
  102. ©2020 Wantedly, Inc. Understanding the problems

  103. ©2020 Wantedly, Inc. Difficult to maintain Too complex, each part

    of the app required different test setup A lot of “magic” to inject test doubles Tightly coupled Hard to test in isolation Rewrite is not possible Making legacy code testable Understanding the problems
  104. ©2020 Wantedly, Inc. Create a plan Determine the type of

    test and decide on a convention Page Title Page Subtitle
  105. ©2020 Wantedly, Inc. Making legacy code testable Create a plan

    Firebase Robo Test Fragments and ViewModels ViewModels, Repositories, Helper classes
  106. ©2020 Wantedly, Inc. Descriptive test method name Develop branch is

    always shippable Every Pull Request needs to have added/removed test code Test libraries for easier setup AndroidX Test, Robolectric, MockK, Custom JUnit rules, etc. Making legacy code testable Create a plan
  107. ©2020 Wantedly, Inc. Migration process Page Title Page Subtitle

  108. ©2020 Wantedly, Inc. MFHBDZ BQQ Making legacy code testable Legacy

    module
  109. ©2020 Wantedly, Inc. BQQ MFHBDZ Making legacy code testable Legacy

    module
  110. ©2020 Wantedly, Inc. TFBSDI POCPBSEJOH BQQ MFHBDZ Making legacy code

    testable Legacy module
  111. ©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code

    testable Legacy module
  112. ©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code

    testable Legacy module
  113. ©2020 Wantedly, Inc. CI/CD infrastructure setup Page Title Page Subtitle

  114. ©2020 Wantedly, Inc. + Making legacy code testable CI/CD pipeline

    setup Fastlane
  115. ©2020 Wantedly, Inc. Making legacy code testable CI/CD pipeline setup

    + Fastlane Local Unit Tests JVM Instrumented Tests Firebase Test Lab Firebase Robo Test JaCoCo + Codecov Pull Request Nightly
  116. ©2020 Wantedly, Inc. ~60% Code coverage1 Within a year Consistent

    releases New features and updates Making legacy code testable CI/CD pipeline setup -65% Monthly fatal crashes 1: New and migrated code
  117. ©2020 Wantedly, Inc. Summary

  118. ©2020 Wantedly, Inc. Automated tests make development faster Makes team

    and project scalable Create test strategy and test conventions Everybody follows the same strategy and conventions Focus on writing maintainable tests Plan testable architecture early or create a migration plan. Projects will always get more complicated! Summary
  119. ©2020 Wantedly, Inc. Thank you! Page Title Page Subtitle