Testing as a Culture

Automated testing is a very important aspect of building any kind of software, including Android apps. There are many resources on the Internet on how to test our app and everybody has their own opinion on what is the most effective method. In Android, automated testing does not only mean unit tests, as there are different ways to automate the testing of our application. When building an app, setting up automated testing should be a culture, instead of an afterthought. However, there are still some teams who have very minimal tests and some don't even have any tests at all for their projects.
In this talk, I will discuss the mindset for testing an Android application. A mental model that we can apply when writing and improving our automated testing. I will also explore the types of tests that are available in an Android project and then present some techniques and tools that can help us structure our tests, such as the testing pyramid, Project Nitrogen, and Firebase Robo Test. In addition, I will talk about testing conventions and how it can help us write cleaner, more readable, and more maintainable test code. Lastly, I will share my experience of migrating a legacy codebase to a new architecture that is more scalable and testable.

Transcript

1. ©2020 Wantedly, Inc. Photo by Rami Al-zayat on Unsplash
   Testing as a Culture
   DroidKaigi 2020
   Malvin Sutanto | @malvinsutanto
   

   View Slide

2. ©2020 Wantedly, Inc.
   Introduction
   Malvin Sutanto
   Software Engineer @Wantedly
   Android, Kotlin.
   Twitter/Medium: @malvinsutanto
   

   View Slide

3. ©2020 Wantedly, Inc.
   Android application
   

   View Slide

4. ©2020 Wantedly, Inc.
   Android application over the years
   !
   

   View Slide

5. ©2020 Wantedly, Inc.
   !
   Android application over the years
   

   View Slide

6. ©2020 Wantedly, Inc.
   !"
   Android application over the years
   

   View Slide

7. ©2020 Wantedly, Inc.
   "#
   Android application over the years
   

   View Slide

8. ©2020 Wantedly, Inc.
   !$%&
   Android application over the years
   

   View Slide

9. ©2020 Wantedly, Inc.
   Can I release this?
   What if it crashes?
   Will there be any bugs?
   Will there be any problems?
   Android application over the years
   

   View Slide

10. ©2020 Wantedly, Inc.
    Who should test your app?
    

    View Slide

11. ©2020 Wantedly, Inc.
    Manual QA team
    Page Title Page Subtitle
    
    

    View Slide

12. ©2020 Wantedly, Inc.
    Follow scenarios and validate the outcomes
    Time consuming
    Not very scalable
    Hard to test different devices and features
    Hard to simulate edge cases
    How to simulate network errors?
    Who should test your app? Manual QA team
    

    View Slide

13. ©2020 Wantedly, Inc.
    Engineers
    )*
    Page Title Page Subtitle
    

    View Slide

14. ©2020 Wantedly, Inc.
    Know the code best
    Manual tests are necessary
    Is it productive?
    Still difficult to simulate edge cases
    Need to build and ship product to the users
    Who should test your app? Engineers
    

    View Slide

15. ©2020 Wantedly, Inc.
    End users
    +
    Page Title Page Subtitle
    
    

    View Slide

16. ©2020 Wantedly, Inc.
    +
    Page Title Page Subtitle
    
    

    View Slide

17. ©2020 Wantedly, Inc.
    Automated test
    for an Android app
    

    View Slide

18. ©2020 Wantedly, Inc.
    Difficult to set up correctly
    Especially in an existing project
    Rewriting test code for any changes
    Slowing you down
    Automated test for an Android app
    
    

    View Slide

19. ©2020 Wantedly, Inc.
    Why automated tests
    

    View Slide

20. ©2020 Wantedly, Inc.
    Scalable
    Page Title Page Subtitle
    

    View Slide

21. ©2020 Wantedly, Inc.
    Write once, run forever
    However many times you want at any time
    Automate process!
    Benefits of automated tests Scalable
    

    View Slide

22. ©2020 Wantedly, Inc.
    Consider this screen
    EditText for a message and a “Send” Button
    Type into the EditText and when the “Send” Button is
    pressed, send the message inside the EditText.
    If EditText is empty, the Button will be disabled.
    Benefits of automated tests Scalable
    

    View Slide

23. ©2020 Wantedly, Inc.
    @Test
    fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...}
    @Test
    fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...}
    @Test
    fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...}
    Benefits of automated tests Scalable
    

    View Slide

24. ©2020 Wantedly, Inc.
    10 0
    Mins Mins
    Automated test Manual test
    Writing time Writing time
    Benefits of automated tests Scalable
    

    View Slide

25. ©2020 Wantedly, Inc.
    10 2
    Mins Mins
    Automated test Manual test
    Writing time Writing time + 1x execution time
    Benefits of automated tests Scalable
    

    View Slide

26. ©2020 Wantedly, Inc.
    11 2
    Mins Mins
    Automated test Manual test
    Writing time + 1x execution time Writing time + 1x execution time
    Benefits of automated tests Scalable
    

    View Slide

27. ©2020 Wantedly, Inc.
    12 2
    Mins Mins
    Automated test Manual test
    Writing time + 2x execution time Writing time + 1x execution time
    Benefits of automated tests Scalable
    

    View Slide

28. ©2020 Wantedly, Inc.
    12 4
    Mins Mins
    Automated test Manual test
    Writing time + 2x execution time Writing time + 2x execution time
    Benefits of automated tests Scalable
    

    View Slide

29. ©2020 Wantedly, Inc.
    14 8
    Mins Mins
    Automated test Manual test
    Writing time + 4x execution time Writing time + 4x execution time
    Benefits of automated tests Scalable
    

    View Slide

30. ©2020 Wantedly, Inc.
    20 20
    Mins Mins
    Automated test Manual test
    Writing time + 10x execution time Writing time + 10x execution time
    Benefits of automated tests Scalable
    

    View Slide

31. ©2020 Wantedly, Inc.
    110 200
    Mins Mins
    Automated test Manual test
    Writing time + 100x execution time Writing time + 100x execution time
    Benefits of automated tests Scalable
    

    View Slide

32. ©2020 Wantedly, Inc.
    Benefits of automated tests Scalable
    Time
    Number of executions
    Automated tests
    Manual tests
    

    View Slide

33. ©2020 Wantedly, Inc.
    Simulate edge cases
    Page Title Page Subtitle
    

    View Slide

34. ©2020 Wantedly, Inc.
    Reliably test network errors
    Especially with proper test doubles
    Simulate error responses
    Simulate response timing
    Benefits of automated tests Simulate edge cases
    

    View Slide

35. ©2020 Wantedly, Inc.
    Documentation
    and information sharing
    Page Title Page Subtitle
    

    View Slide

36. ©2020 Wantedly, Inc.
    Structured and descriptive test cases to
    describe use cases
    Benefits of automated tests Documentation and information sharing
    

    View Slide

37. ©2020 Wantedly, Inc.
    @Test
    fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...}
    @Test
    fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...}
    @Test
    fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...}
    Benefits of automated tests Documentation and information sharing
    

    View Slide

38. ©2020 Wantedly, Inc.
    Structured and descriptive test cases to
    describe use cases
    Benefits of automated tests Documentation and information sharing
    Prevent silos of information
    Aid in code reviews
    

    View Slide

39. ©2020 Wantedly, Inc.
    Write better software
    Page Title Page Subtitle
    

    View Slide

40. ©2020 Wantedly, Inc.
    Decouple based on usage
    Reinforce single responsibility pattern
    Shorten the feedback loop
    Increase development velocity
    Benefits of automated tests Write better software
    

    View Slide

41. ©2020 Wantedly, Inc.
    Release updates with confidence
    Page Title Page Subtitle
    

    View Slide

42. ©2020 Wantedly, Inc.
    Automated test for Android projects
    Tools and frameworks
    

    View Slide

43. ©2020 Wantedly, Inc.
    Local unit test
    

    View Slide

44. ©2020 Wantedly, Inc.
    Located in src/test/java
    Runs in local machine’s JVM
    Relatively fast
    No Android framework’s API
    Automated tests for Android projects Local unit test
    

    View Slide

45. ©2020 Wantedly, Inc.
    Instrumented test
    

    View Slide

46. ©2020 Wantedly, Inc.
    Located in src/androidTest/java
    Runs on a device or an emulator
    Combined with Espresso
    Tests and validates UI components
    Relatively slow
    Automated tests for Android projects Instrumented test
    

    View Slide

47. ©2020 Wantedly, Inc.
    Robolectric
    

    View Slide

48. ©2020 Wantedly, Inc.
    Uses Android framework’s API in local JVM
    environment
    4.0+ supports JetPack’s androidx.tests APIs
    Shadow implementation of Android Framework APIs
    See: https://github.com/robolectric/robolectric
    Automated tests for Android projects Robolectric
    

    View Slide

49. ©2020 Wantedly, Inc.
    Project Nitrogen
    Page Title Page Subtitle
    

    View Slide

50. ©2020 Wantedly, Inc.
    Write once, run everywhere
    Choose whether to run your test code
    As a local unit test
    As an instrumented test
    Automated tests for Android projects Project Nitrogen
    

    View Slide

51. ©2020 Wantedly, Inc.
    Project Nitrogen
    Setting up shared tests
    https://www.youtube.com/watch?v=VJi2vmaQe6w
    Automated tests for Android projects Project Nitrogen
    build.gradle
    android {
    sourceSets {
    String sharedTestDir = ‘src/sharedTest/java’
    test {
    java.srcDir sharedTestDir
    }
    androidTest {
    java.srcDir sharedTestDir
    }
    }
    }
    

    View Slide

52. ©2020 Wantedly, Inc.
    import androidx.test.ext.junit.runners.AndroidJUnit4
    !!...
    @RunWith(AndroidJUnit4!::class)
    class SampleFragmentTest {
    !!...
    }
    Automated tests for Android projects Project Nitrogen
    See: https://github.com/android/testing-samples/blob/master/ui/espresso/FragmentScenarioSample
    

    View Slide

53. ©2020 Wantedly, Inc.
    UIAutomator
    

    View Slide

54. ©2020 Wantedly, Inc.
    High level, black-box testing
    Set of instructions to interact with the device
    Access device’s state, click at views, press the back button, rotate screen, etc.
    Tests a device as a whole
    Access home screen and app drawers, or interact with other apps
    Automated tests for Android projects UIAutomator
    

    View Slide

55. ©2020 Wantedly, Inc.
    Firebase Robo Test
    

    View Slide

56. ©2020 Wantedly, Inc.
    Simulates real user interactions
    Tests “flows” of an application
    Guide the interactions
    with roboscript and/or robo directives
    Automated tests for Android projects Firebase Robo Test
    https://medium.com/wantedly-engineering/
    android-application-testing-with-firebase-robo-test-c674e1754298
    

    View Slide

57. ©2020 Wantedly, Inc.
    Structuring test suite
    For an Android project
    

    View Slide

58. ©2020 Wantedly, Inc.
    Testing Pyramid
    

    View Slide

59. ©2020 Wantedly, Inc.
    Number of components
    Fidelity
    Execution time
    Maintenance effort
    Testing Pyramid
    Structuring test suite
    

    View Slide

60. ©2020 Wantedly, Inc.
    Number of tests
    Testing Pyramid
    Structuring test suite
    

    View Slide

61. ©2020 Wantedly, Inc.
    Testing Pyramid
    Structuring test suite
    End to end tests
    Integration tests
    Unit tests
    

    View Slide

62. ©2020 Wantedly, Inc.
    Testing Pyramid
    Structuring test suite
    End to end tests
    Integration tests
    Repositories and utility classes
    

    View Slide

63. ©2020 Wantedly, Inc.
    Testing Pyramid
    Structuring test suite
    End to end tests
    Fragments and ViewModels
    Repositories and utility classes
    

    View Slide

64. ©2020 Wantedly, Inc.
    Testing Pyramid
    Structuring test suite
    Firebase Robo Test or UI Automator
    Fragments and ViewModels
    Repositories and utility classes
    

    View Slide

65. ©2020 Wantedly, Inc.
    How?
    

    View Slide

66. ©2020 Wantedly, Inc.
    Determine effort and type of tests
    Create a balanced test suite
    Increase developer’s productivity
    Find the correct amount of tests
    Less exhaustive tests at the top
    How?
    Structuring test suite
    

    View Slide

67. ©2020 Wantedly, Inc.
    Structuring code for testability
    

    View Slide

68. ©2020 Wantedly, Inc.
    Test doubles
    Fakes and mocks
    

    View Slide

69. ©2020 Wantedly, Inc.
    Replacement objects
    With controlled behavior
    Create a predictable condition
    Fake and mock classes
    Structuring code for testability Test doubles
    

    View Slide

70. ©2020 Wantedly, Inc.
    Fake classes
    Easiest to create manually
    Actual object with “shortcut”
    implementations to act like the real
    implementation.
    interface UserRepository {
    suspend fun getUser(userId: Long): User
    }
    class FakeUserRepository: UserRepository {
    override suspend fun getUser(userId: Long): User {
    return User(
    id = userId,
    firstName = "John",
    lastName = "Smith"
    )
    }
    }
    Structuring code for testability Test doubles
    

    View Slide

71. ©2020 Wantedly, Inc.
    Mock classes
    Answer to set calls
    Created using Mockito or MockK
    Respond to a set of method calls whose
    answers have been defined.
    Throw an exception if it encounters any
    unexpected calls that have not been set.
    Structuring code for testability Test doubles
    interface UserRepository {
    suspend fun getUser(userId: Long): User
    suspend fun deleteUser(userId: Long)
    }
    "// Create a mock UserRepository with MockK
    val mockRepo = mockk()
    coEvery { mockRepo.getUser(any()) } answers {
    User(
    id = firstArg(),
    firstName = "John",
    lastName = "Smith"
    )
    }
    

    View Slide

72. ©2020 Wantedly, Inc.
    Structuring code for testability Test doubles - Mock classes
    !//Usage
    launch {
    mockRepo.getUser(1L)
    "// User(1L, "John", "Smith")
    mockRepo.deleteUser(1L)
    "// Throws MockKException: no answer found for UserRepository.deleteUser
    }
    

    View Slide

73. ©2020 Wantedly, Inc.
    Independent and isolated classes
    

    View Slide

74. ©2020 Wantedly, Inc.
    Decouple classes
    Allow for replacement with test doubles
    Use constructor’s arguments to pass in other classes or interfaces
    Avoid static getInstance methods
    Use Dependency Injection Framework
    Dagger
    Structuring code for testability Independent and isolated classes
    

    View Slide

75. ©2020 Wantedly, Inc.
    Structuring code for testability Independent and isolated classes
    val navController = mockk()
    ""...
    fun launchFragment(): FragmentScenario {
    return launchFragmentInContainer(themeResId = R.style.AppTheme) {
    SampleFragment()
    }.onFragment {
    Navigation.setViewNavController(it.requireView(), navController)
    }
    }
    ""...
    verify { navController.navigate(R.id.toOtherFragment) }
    Test in isolation
    

    View Slide

76. ©2020 Wantedly, Inc.
    Stateless tests
    Page Title Page Subtitle
    

    View Slide

77. ©2020 Wantedly, Inc.
    Start and finish at the same state
    Clean up DB, SharedPreferences, value flags, etc.
    Prevent side-effects and flakiness
    Might affect other test’s outcomes
    Properly set up and tear down tests
    Structuring code for testability Stateless tests
    

    View Slide

78. ©2020 Wantedly, Inc.
    class SampleFragmentTest {
    @Before
    fun setUp() {""...}
    @After
    fun tearDown() {""...}
    @Test
    fun givenMessageIsEmpty_ThenButtonIsDisabled() {""...}
    @Test
    fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {""...}
    }
    Structuring code for testability Stateless tests
    }
    

    View Slide

79. ©2020 Wantedly, Inc.
    class SampleFragmentTest {
    @Rule
    @JvmField
    val testSchedulerRule = TestSchedulerRule()
    ""... () () ()
    }
    Structuring code for testability Stateless tests
    }
    

    View Slide

80. ©2020 Wantedly, Inc.
    class TestSchedulerRule : TestRule {
    val testScheduler = TestScheduler()
    override fun apply(base: Statement, description: Description): Statement {
    return object : Statement() {
    override fun evaluate() {
    RxJavaPlugins.setIoSchedulerHandler { testScheduler }
    !!...
    try {
    base.evaluate() "// Test code will be run here.
    } finally {
    RxJavaPlugins.reset()
    ""...
    }
    }
    }
    }
    }
    Structuring code for testability Stateless tests
    Set up
    Test
    Tear down
    

    View Slide

81. ©2020 Wantedly, Inc.
    Synchronous tests
    

    View Slide

82. ©2020 Wantedly, Inc.
    Avoid multi-threading
    Can cause side-effects and increase execution time significantly
    Testing with 3rd party libraries
    TestScheduler for RxJava
    InstantTaskExecutorRule for Architecture Components
    TestDispatcher and runBlockingTest for coroutines
    Structuring code for testability Synchronous tests
    

    View Slide

83. ©2020 Wantedly, Inc.
    Testing conventions
    

    View Slide

84. ©2020 Wantedly, Inc.
    A set of rules or patterns for writing
    tests for a project
    Page Title Page Subtitle
    

    View Slide

85. ©2020 Wantedly, Inc.
    Type of test
    Test method naming
    Usage of test doubles
    Rules to apply
    etc
    Testing conventions
    

    View Slide

86. ©2020 Wantedly, Inc.
    Can be different for each projects
    Page Title Page Subtitle
    

    View Slide

87. ©2020 Wantedly, Inc.
    But everybody agrees on the same
    conventions
    Page Title Page Subtitle
    

    View Slide

88. ©2020 Wantedly, Inc.
    Consistency
    Easier to understand
    Page Title Page Subtitle
    easier to identify breaking changes higher productivity
    

    View Slide

89. ©2020 Wantedly, Inc.
    Allows you to scale your team
    Page Title Page Subtitle
    

    View Slide

90. ©2020 Wantedly, Inc.
    Example test convention
    For describing test case with method name
    Page Title Page Subtitle
    

    View Slide

91. ©2020 Wantedly, Inc.
    Testing conventions
    Describes what is being tested
    Not too long
    Test one aspect of your app at a time
    Use words like given, when, then, etc
    Example
    

    View Slide

92. ©2020 Wantedly, Inc.
    fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() { … }
    Testing conventions Example
    Initial state Action Outcome
    

    View Slide

93. ©2020 Wantedly, Inc.
    CI/CD Pipeline
    Automate test further
    

    View Slide

94. ©2020 Wantedly, Inc.
    Run tests against the latest code
    Automatically detect breaking changes
    

    View Slide

95. ©2020 Wantedly, Inc.
    Ci/CD Pipeline Automate test further
    PR
    Trigger
    Run
    

    View Slide

96. ©2020 Wantedly, Inc.
    Other considerations
    When writing tests
    

    View Slide

97. ©2020 Wantedly, Inc.
    Time⏳
    Page Title Page Subtitle
    Writing and execution time
    

    View Slide

98. ©2020 Wantedly, Inc.
    Slow test can decrease productivity
    Long CI/CD pipeline execution time
    Can have financial impact
    Don’t try aim for 100% coverage
    Focus on testing the most important parts
    Other considerations Time
    

    View Slide

99. ©2020 Wantedly, Inc.
    Do your manual test!
    Manual testing is still important
    Page Title Page Subtitle
    

    View Slide

100. ©2020 Wantedly, Inc.
     Good for user interactions and animations
     Find edge cases, incorrect test doubles, etc
     Other considerations Manual test is important!
     

     View Slide

101. ©2020 Wantedly, Inc.
     Making legacy code testable
     What we did and what we've learnt
     

     View Slide

102. ©2020 Wantedly, Inc.
     Understanding the problems
     

     View Slide

103. ©2020 Wantedly, Inc.
     Difficult to maintain
     Too complex, each part of the app required different test setup
     A lot of “magic” to inject test doubles
     Tightly coupled
     Hard to test in isolation
     Rewrite is not possible
     Making legacy code testable Understanding the problems
     

     View Slide

104. ©2020 Wantedly, Inc.
     Create a plan
     Determine the type of test and decide on a convention
     Page Title Page Subtitle
     

     View Slide

105. ©2020 Wantedly, Inc.
     Making legacy code testable Create a plan
     Firebase Robo Test
     Fragments and ViewModels
     ViewModels, Repositories, Helper classes
     

     View Slide

106. ©2020 Wantedly, Inc.
     Descriptive test method name
     Develop branch is always shippable
     Every Pull Request needs to have added/removed test code
     Test libraries for easier setup
     AndroidX Test, Robolectric, MockK, Custom JUnit rules, etc.
     Making legacy code testable Create a plan
     

     View Slide

107. ©2020 Wantedly, Inc.
     Migration process
     Page Title Page Subtitle
     

     View Slide

108. ©2020 Wantedly, Inc.
     MFHBDZ
     BQQ
     Making legacy code testable Legacy module
     

     View Slide

109. ©2020 Wantedly, Inc.
     BQQ
     MFHBDZ
     Making legacy code testable Legacy module
     

     View Slide

110. ©2020 Wantedly, Inc.
     TFBSDI
     POCPBSEJOH
     BQQ
     MFHBDZ
     Making legacy code testable Legacy module
     

     View Slide

111. ©2020 Wantedly, Inc.
     TFBSDI
     MFHBDZ
     POCPBSEJOH
     BQQ
     Making legacy code testable Legacy module
     

     View Slide

112. ©2020 Wantedly, Inc.
     TFBSDI
     MFHBDZ
     POCPBSEJOH
     BQQ
     Making legacy code testable Legacy module
     

     View Slide

113. ©2020 Wantedly, Inc.
     CI/CD infrastructure setup
     Page Title Page Subtitle
     

     View Slide

114. ©2020 Wantedly, Inc.
     +
     Making legacy code testable CI/CD pipeline setup
     Fastlane
     

     View Slide

115. ©2020 Wantedly, Inc.
     Making legacy code testable CI/CD pipeline setup
     +
     Fastlane
     Local Unit Tests
     JVM
     Instrumented Tests
     Firebase Test Lab
     Firebase Robo Test
     JaCoCo + Codecov
     Pull Request
     Nightly
     

     View Slide

116. ©2020 Wantedly, Inc.
     ~60%
     Code coverage1
     Within a year
     Consistent releases
     New features and updates
     Making legacy code testable CI/CD pipeline setup
     -65%
     Monthly fatal crashes
     1: New and migrated code
     

     View Slide

117. ©2020 Wantedly, Inc.
     Summary
     

     View Slide

118. ©2020 Wantedly, Inc.
     Automated tests make development faster
     Makes team and project scalable
     Create test strategy and test conventions
     Everybody follows the same strategy and conventions
     Focus on writing maintainable tests
     Plan testable architecture early
     or create a migration plan. Projects will always get more complicated!
     Summary
     

     View Slide

119. ©2020 Wantedly, Inc.
     Thank you!
     
     Page Title Page Subtitle
     

     View Slide