Slide 1

Slide 1 text

©2020 Wantedly, Inc. Photo by Rami Al-zayat on Unsplash Testing as a Culture DroidKaigi 2020 Malvin Sutanto | @malvinsutanto

Slide 2

Slide 2 text

©2020 Wantedly, Inc. Introduction Malvin Sutanto Software Engineer @Wantedly Android, Kotlin. Twitter/Medium: @malvinsutanto

Slide 3

Slide 3 text

©2020 Wantedly, Inc. Android application

Slide 4

Slide 4 text

©2020 Wantedly, Inc. Android application over the years !

Slide 5

Slide 5 text

©2020 Wantedly, Inc. ! Android application over the years

Slide 6

Slide 6 text

©2020 Wantedly, Inc. !" Android application over the years

Slide 7

Slide 7 text

©2020 Wantedly, Inc. "# Android application over the years

Slide 8

Slide 8 text

©2020 Wantedly, Inc. !$%& Android application over the years

Slide 9

Slide 9 text

©2020 Wantedly, Inc. Can I release this? What if it crashes? Will there be any bugs? Will there be any problems? Android application over the years

Slide 10

Slide 10 text

©2020 Wantedly, Inc. Who should test your app?

Slide 11

Slide 11 text

©2020 Wantedly, Inc. Manual QA team Page Title Page Subtitle

Slide 12

Slide 12 text

©2020 Wantedly, Inc. Follow scenarios and validate the outcomes Time consuming Not very scalable Hard to test different devices and features Hard to simulate edge cases How to simulate network errors? Who should test your app? Manual QA team

Slide 13

Slide 13 text

©2020 Wantedly, Inc. Engineers )* Page Title Page Subtitle

Slide 14

Slide 14 text

©2020 Wantedly, Inc. Know the code best Manual tests are necessary Is it productive? Still difficult to simulate edge cases Need to build and ship product to the users Who should test your app? Engineers

Slide 15

Slide 15 text

©2020 Wantedly, Inc. End users + Page Title Page Subtitle

Slide 16

Slide 16 text

©2020 Wantedly, Inc. + Page Title Page Subtitle

Slide 17

Slide 17 text

©2020 Wantedly, Inc. Automated test for an Android app

Slide 18

Slide 18 text

©2020 Wantedly, Inc. Difficult to set up correctly Especially in an existing project Rewriting test code for any changes Slowing you down Automated test for an Android app

Slide 19

Slide 19 text

©2020 Wantedly, Inc. Why automated tests

Slide 20

Slide 20 text

©2020 Wantedly, Inc. Scalable Page Title Page Subtitle

Slide 21

Slide 21 text

©2020 Wantedly, Inc. Write once, run forever However many times you want at any time Automate process! Benefits of automated tests Scalable

Slide 22

Slide 22 text

©2020 Wantedly, Inc. Consider this screen EditText for a message and a “Send” Button Type into the EditText and when the “Send” Button is pressed, send the message inside the EditText. If EditText is empty, the Button will be disabled. Benefits of automated tests Scalable

Slide 23

Slide 23 text

©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Scalable

Slide 24

Slide 24 text

©2020 Wantedly, Inc. 10 0 Mins Mins Automated test Manual test Writing time Writing time Benefits of automated tests Scalable

Slide 25

Slide 25 text

©2020 Wantedly, Inc. 10 2 Mins Mins Automated test Manual test Writing time Writing time + 1x execution time Benefits of automated tests Scalable

Slide 26

Slide 26 text

©2020 Wantedly, Inc. 11 2 Mins Mins Automated test Manual test Writing time + 1x execution time Writing time + 1x execution time Benefits of automated tests Scalable

Slide 27

Slide 27 text

©2020 Wantedly, Inc. 12 2 Mins Mins Automated test Manual test Writing time + 2x execution time Writing time + 1x execution time Benefits of automated tests Scalable

Slide 28

Slide 28 text

©2020 Wantedly, Inc. 12 4 Mins Mins Automated test Manual test Writing time + 2x execution time Writing time + 2x execution time Benefits of automated tests Scalable

Slide 29

Slide 29 text

©2020 Wantedly, Inc. 14 8 Mins Mins Automated test Manual test Writing time + 4x execution time Writing time + 4x execution time Benefits of automated tests Scalable

Slide 30

Slide 30 text

©2020 Wantedly, Inc. 20 20 Mins Mins Automated test Manual test Writing time + 10x execution time Writing time + 10x execution time Benefits of automated tests Scalable

Slide 31

Slide 31 text

©2020 Wantedly, Inc. 110 200 Mins Mins Automated test Manual test Writing time + 100x execution time Writing time + 100x execution time Benefits of automated tests Scalable

Slide 32

Slide 32 text

©2020 Wantedly, Inc. Benefits of automated tests Scalable Time Number of executions Automated tests Manual tests

Slide 33

Slide 33 text

©2020 Wantedly, Inc. Simulate edge cases Page Title Page Subtitle

Slide 34

Slide 34 text

©2020 Wantedly, Inc. Reliably test network errors Especially with proper test doubles Simulate error responses Simulate response timing Benefits of automated tests Simulate edge cases

Slide 35

Slide 35 text

©2020 Wantedly, Inc. Documentation and information sharing Page Title Page Subtitle

Slide 36

Slide 36 text

©2020 Wantedly, Inc. Structured and descriptive test cases to describe use cases Benefits of automated tests Documentation and information sharing

Slide 37

Slide 37 text

©2020 Wantedly, Inc. @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {!!...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {!!...} @Test fun givenMessageHasText_WhenDeleteAllText_ThenButtonIsDisabled() {!!...} Benefits of automated tests Documentation and information sharing

Slide 38

Slide 38 text

©2020 Wantedly, Inc. Structured and descriptive test cases to describe use cases Benefits of automated tests Documentation and information sharing Prevent silos of information Aid in code reviews

Slide 39

Slide 39 text

©2020 Wantedly, Inc. Write better software Page Title Page Subtitle

Slide 40

Slide 40 text

©2020 Wantedly, Inc. Decouple based on usage Reinforce single responsibility pattern Shorten the feedback loop Increase development velocity Benefits of automated tests Write better software

Slide 41

Slide 41 text

©2020 Wantedly, Inc. Release updates with confidence Page Title Page Subtitle

Slide 42

Slide 42 text

©2020 Wantedly, Inc. Automated test for Android projects Tools and frameworks

Slide 43

Slide 43 text

©2020 Wantedly, Inc. Local unit test

Slide 44

Slide 44 text

©2020 Wantedly, Inc. Located in src/test/java Runs in local machine’s JVM Relatively fast No Android framework’s API Automated tests for Android projects Local unit test

Slide 45

Slide 45 text

©2020 Wantedly, Inc. Instrumented test

Slide 46

Slide 46 text

©2020 Wantedly, Inc. Located in src/androidTest/java Runs on a device or an emulator Combined with Espresso Tests and validates UI components Relatively slow Automated tests for Android projects Instrumented test

Slide 47

Slide 47 text

©2020 Wantedly, Inc. Robolectric

Slide 48

Slide 48 text

©2020 Wantedly, Inc. Uses Android framework’s API in local JVM environment 4.0+ supports JetPack’s androidx.tests APIs Shadow implementation of Android Framework APIs See: https://github.com/robolectric/robolectric Automated tests for Android projects Robolectric

Slide 49

Slide 49 text

©2020 Wantedly, Inc. Project Nitrogen Page Title Page Subtitle

Slide 50

Slide 50 text

©2020 Wantedly, Inc. Write once, run everywhere Choose whether to run your test code As a local unit test As an instrumented test Automated tests for Android projects Project Nitrogen

Slide 51

Slide 51 text

©2020 Wantedly, Inc. Project Nitrogen Setting up shared tests https://www.youtube.com/watch?v=VJi2vmaQe6w Automated tests for Android projects Project Nitrogen build.gradle android { sourceSets { String sharedTestDir = ‘src/sharedTest/java’ test { java.srcDir sharedTestDir } androidTest { java.srcDir sharedTestDir } } }

Slide 52

Slide 52 text

©2020 Wantedly, Inc. import androidx.test.ext.junit.runners.AndroidJUnit4 !!... @RunWith(AndroidJUnit4!::class) class SampleFragmentTest { !!... } Automated tests for Android projects Project Nitrogen See: https://github.com/android/testing-samples/blob/master/ui/espresso/FragmentScenarioSample

Slide 53

Slide 53 text

©2020 Wantedly, Inc. UIAutomator

Slide 54

Slide 54 text

©2020 Wantedly, Inc. High level, black-box testing Set of instructions to interact with the device Access device’s state, click at views, press the back button, rotate screen, etc. Tests a device as a whole Access home screen and app drawers, or interact with other apps Automated tests for Android projects UIAutomator

Slide 55

Slide 55 text

©2020 Wantedly, Inc. Firebase Robo Test

Slide 56

Slide 56 text

©2020 Wantedly, Inc. Simulates real user interactions Tests “flows” of an application Guide the interactions with roboscript and/or robo directives Automated tests for Android projects Firebase Robo Test https://medium.com/wantedly-engineering/ android-application-testing-with-firebase-robo-test-c674e1754298

Slide 57

Slide 57 text

©2020 Wantedly, Inc. Structuring test suite For an Android project

Slide 58

Slide 58 text

©2020 Wantedly, Inc. Testing Pyramid

Slide 59

Slide 59 text

©2020 Wantedly, Inc. Number of components Fidelity Execution time Maintenance effort Testing Pyramid Structuring test suite

Slide 60

Slide 60 text

©2020 Wantedly, Inc. Number of tests Testing Pyramid Structuring test suite

Slide 61

Slide 61 text

©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Integration tests Unit tests

Slide 62

Slide 62 text

©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Integration tests Repositories and utility classes

Slide 63

Slide 63 text

©2020 Wantedly, Inc. Testing Pyramid Structuring test suite End to end tests Fragments and ViewModels Repositories and utility classes

Slide 64

Slide 64 text

©2020 Wantedly, Inc. Testing Pyramid Structuring test suite Firebase Robo Test or UI Automator Fragments and ViewModels Repositories and utility classes

Slide 65

Slide 65 text

©2020 Wantedly, Inc. How?

Slide 66

Slide 66 text

©2020 Wantedly, Inc. Determine effort and type of tests Create a balanced test suite Increase developer’s productivity Find the correct amount of tests Less exhaustive tests at the top How? Structuring test suite

Slide 67

Slide 67 text

©2020 Wantedly, Inc. Structuring code for testability

Slide 68

Slide 68 text

©2020 Wantedly, Inc. Test doubles Fakes and mocks

Slide 69

Slide 69 text

©2020 Wantedly, Inc. Replacement objects With controlled behavior Create a predictable condition Fake and mock classes Structuring code for testability Test doubles

Slide 70

Slide 70 text

©2020 Wantedly, Inc. Fake classes Easiest to create manually Actual object with “shortcut” implementations to act like the real implementation. interface UserRepository { suspend fun getUser(userId: Long): User } class FakeUserRepository: UserRepository { override suspend fun getUser(userId: Long): User { return User( id = userId, firstName = "John", lastName = "Smith" ) } } Structuring code for testability Test doubles

Slide 71

Slide 71 text

©2020 Wantedly, Inc. Mock classes Answer to set calls Created using Mockito or MockK Respond to a set of method calls whose answers have been defined. Throw an exception if it encounters any unexpected calls that have not been set. Structuring code for testability Test doubles interface UserRepository { suspend fun getUser(userId: Long): User suspend fun deleteUser(userId: Long) } "// Create a mock UserRepository with MockK val mockRepo = mockk() coEvery { mockRepo.getUser(any()) } answers { User( id = firstArg(), firstName = "John", lastName = "Smith" ) }

Slide 72

Slide 72 text

©2020 Wantedly, Inc. Structuring code for testability Test doubles - Mock classes !//Usage launch { mockRepo.getUser(1L) "// User(1L, "John", "Smith") mockRepo.deleteUser(1L) "// Throws MockKException: no answer found for UserRepository.deleteUser }

Slide 73

Slide 73 text

©2020 Wantedly, Inc. Independent and isolated classes

Slide 74

Slide 74 text

©2020 Wantedly, Inc. Decouple classes Allow for replacement with test doubles Use constructor’s arguments to pass in other classes or interfaces Avoid static getInstance methods Use Dependency Injection Framework Dagger Structuring code for testability Independent and isolated classes

Slide 75

Slide 75 text

©2020 Wantedly, Inc. Structuring code for testability Independent and isolated classes val navController = mockk() ""... fun launchFragment(): FragmentScenario { return launchFragmentInContainer(themeResId = R.style.AppTheme) { SampleFragment() }.onFragment { Navigation.setViewNavController(it.requireView(), navController) } } ""... verify { navController.navigate(R.id.toOtherFragment) } Test in isolation

Slide 76

Slide 76 text

©2020 Wantedly, Inc. Stateless tests Page Title Page Subtitle

Slide 77

Slide 77 text

©2020 Wantedly, Inc. Start and finish at the same state Clean up DB, SharedPreferences, value flags, etc. Prevent side-effects and flakiness Might affect other test’s outcomes Properly set up and tear down tests Structuring code for testability Stateless tests

Slide 78

Slide 78 text

©2020 Wantedly, Inc. class SampleFragmentTest { @Before fun setUp() {""...} @After fun tearDown() {""...} @Test fun givenMessageIsEmpty_ThenButtonIsDisabled() {""...} @Test fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() {""...} } Structuring code for testability Stateless tests }

Slide 79

Slide 79 text

©2020 Wantedly, Inc. class SampleFragmentTest { @Rule @JvmField val testSchedulerRule = TestSchedulerRule() ""... () () () } Structuring code for testability Stateless tests }

Slide 80

Slide 80 text

©2020 Wantedly, Inc. class TestSchedulerRule : TestRule { val testScheduler = TestScheduler() override fun apply(base: Statement, description: Description): Statement { return object : Statement() { override fun evaluate() { RxJavaPlugins.setIoSchedulerHandler { testScheduler } !!... try { base.evaluate() "// Test code will be run here. } finally { RxJavaPlugins.reset() ""... } } } } } Structuring code for testability Stateless tests Set up Test Tear down

Slide 81

Slide 81 text

©2020 Wantedly, Inc. Synchronous tests

Slide 82

Slide 82 text

©2020 Wantedly, Inc. Avoid multi-threading Can cause side-effects and increase execution time significantly Testing with 3rd party libraries TestScheduler for RxJava InstantTaskExecutorRule for Architecture Components TestDispatcher and runBlockingTest for coroutines Structuring code for testability Synchronous tests

Slide 83

Slide 83 text

©2020 Wantedly, Inc. Testing conventions

Slide 84

Slide 84 text

©2020 Wantedly, Inc. A set of rules or patterns for writing tests for a project Page Title Page Subtitle

Slide 85

Slide 85 text

©2020 Wantedly, Inc. Type of test Test method naming Usage of test doubles Rules to apply etc Testing conventions

Slide 86

Slide 86 text

©2020 Wantedly, Inc. Can be different for each projects Page Title Page Subtitle

Slide 87

Slide 87 text

©2020 Wantedly, Inc. But everybody agrees on the same conventions Page Title Page Subtitle

Slide 88

Slide 88 text

©2020 Wantedly, Inc. Consistency Easier to understand Page Title Page Subtitle easier to identify breaking changes higher productivity

Slide 89

Slide 89 text

©2020 Wantedly, Inc. Allows you to scale your team Page Title Page Subtitle

Slide 90

Slide 90 text

©2020 Wantedly, Inc. Example test convention For describing test case with method name Page Title Page Subtitle

Slide 91

Slide 91 text

©2020 Wantedly, Inc. Testing conventions Describes what is being tested Not too long Test one aspect of your app at a time Use words like given, when, then, etc Example

Slide 92

Slide 92 text

©2020 Wantedly, Inc. fun givenMessageHasText_WhenClickSend_ThenSendTextToAPI() { … } Testing conventions Example Initial state Action Outcome

Slide 93

Slide 93 text

©2020 Wantedly, Inc. CI/CD Pipeline Automate test further

Slide 94

Slide 94 text

©2020 Wantedly, Inc. Run tests against the latest code Automatically detect breaking changes

Slide 95

Slide 95 text

©2020 Wantedly, Inc. Ci/CD Pipeline Automate test further PR Trigger Run

Slide 96

Slide 96 text

©2020 Wantedly, Inc. Other considerations When writing tests

Slide 97

Slide 97 text

©2020 Wantedly, Inc. Time⏳ Page Title Page Subtitle Writing and execution time

Slide 98

Slide 98 text

©2020 Wantedly, Inc. Slow test can decrease productivity Long CI/CD pipeline execution time Can have financial impact Don’t try aim for 100% coverage Focus on testing the most important parts Other considerations Time

Slide 99

Slide 99 text

©2020 Wantedly, Inc. Do your manual test! Manual testing is still important Page Title Page Subtitle

Slide 100

Slide 100 text

©2020 Wantedly, Inc. Good for user interactions and animations Find edge cases, incorrect test doubles, etc Other considerations Manual test is important!

Slide 101

Slide 101 text

©2020 Wantedly, Inc. Making legacy code testable What we did and what we've learnt

Slide 102

Slide 102 text

©2020 Wantedly, Inc. Understanding the problems

Slide 103

Slide 103 text

©2020 Wantedly, Inc. Difficult to maintain Too complex, each part of the app required different test setup A lot of “magic” to inject test doubles Tightly coupled Hard to test in isolation Rewrite is not possible Making legacy code testable Understanding the problems

Slide 104

Slide 104 text

©2020 Wantedly, Inc. Create a plan Determine the type of test and decide on a convention Page Title Page Subtitle

Slide 105

Slide 105 text

©2020 Wantedly, Inc. Making legacy code testable Create a plan Firebase Robo Test Fragments and ViewModels ViewModels, Repositories, Helper classes

Slide 106

Slide 106 text

©2020 Wantedly, Inc. Descriptive test method name Develop branch is always shippable Every Pull Request needs to have added/removed test code Test libraries for easier setup AndroidX Test, Robolectric, MockK, Custom JUnit rules, etc. Making legacy code testable Create a plan

Slide 107

Slide 107 text

©2020 Wantedly, Inc. Migration process Page Title Page Subtitle

Slide 108

Slide 108 text

©2020 Wantedly, Inc. MFHBDZ BQQ Making legacy code testable Legacy module

Slide 109

Slide 109 text

©2020 Wantedly, Inc. BQQ MFHBDZ Making legacy code testable Legacy module

Slide 110

Slide 110 text

©2020 Wantedly, Inc. TFBSDI POCPBSEJOH BQQ MFHBDZ Making legacy code testable Legacy module

Slide 111

Slide 111 text

©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code testable Legacy module

Slide 112

Slide 112 text

©2020 Wantedly, Inc. TFBSDI MFHBDZ POCPBSEJOH BQQ Making legacy code testable Legacy module

Slide 113

Slide 113 text

©2020 Wantedly, Inc. CI/CD infrastructure setup Page Title Page Subtitle

Slide 114

Slide 114 text

©2020 Wantedly, Inc. + Making legacy code testable CI/CD pipeline setup Fastlane

Slide 115

Slide 115 text

©2020 Wantedly, Inc. Making legacy code testable CI/CD pipeline setup + Fastlane Local Unit Tests JVM Instrumented Tests Firebase Test Lab Firebase Robo Test JaCoCo + Codecov Pull Request Nightly

Slide 116

Slide 116 text

©2020 Wantedly, Inc. ~60% Code coverage1 Within a year Consistent releases New features and updates Making legacy code testable CI/CD pipeline setup -65% Monthly fatal crashes 1: New and migrated code

Slide 117

Slide 117 text

©2020 Wantedly, Inc. Summary

Slide 118

Slide 118 text

©2020 Wantedly, Inc. Automated tests make development faster Makes team and project scalable Create test strategy and test conventions Everybody follows the same strategy and conventions Focus on writing maintainable tests Plan testable architecture early or create a migration plan. Projects will always get more complicated! Summary

Slide 119

Slide 119 text

©2020 Wantedly, Inc. Thank you! Page Title Page Subtitle