Testing for Success in the Real World

Does that code you just wrote actually work? How do you know? How do your teammates know? You did write tests for that, right? What kind did you write though? Unit? Integration? System? End-to-End? What about mocking and stubbing? I know, you only changed two lines of a legacy piece of the app… but still, how do you know this didn’t break anything?

Ugh. Let’s face it testing your application is difficult and tedious. Where can you get the most bang for your buck? What’s the 20% of work that gets you 80% of the return? In this session you’ll learn where you can focus your attention to gain the most traction in your testing endeavors. From mocking api calls, to juxtaposing the benefits of unit testing vs end-to-end testing, we’ll cover it all.

Transcript

1. TESTING FOR SUCCESS IN THE REAL WORLD
2. assertThat(name).isEqualTo(“Donn Felker”) @donnfelker 2 Fragmented Podcast donnfelker.com
3. MY HISTORY WITH TESTING 3
4. 4 1999-2003: Wild West Cowboy Coder ▪ What’s a test? ▪ “It works on my box.” ▪ Yeah, I tested it. Just hit “refresh” and you’ll see that it still works. ▪ Didn’t really know what testing was/how to do it.
5. Debugging and Manual Verification Testing ▪ alert(“in here”) ▪ console.log(“in here”) ▪ Response.Write(“in here”) ▪ echo “in here” ▪ Log.d(“MainActivity”, “in here”) 5
6. 2002: Start Working in the Music Industry 6
7. 7 HEADER SIDEBAR MAIN MENU HEADER MENU FEATURE SIDEBAR CONTENT FEATURE IMAGE LINKS
8. 8 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); // ... } catch (Exception $e) { // do nothing }
9. 9 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); // ... } catch (Exception $e) { echo "<h1>Oh shit.</h1>"; }
10. 10 WEBSITE LAUNCHED
11. 11 FAST FORWARD 9 MONTHS
12. 12 “YO, MY WEBSITE IS COVERED IN SHIT! IT’S NOT A GOOD LOOK, SON! WASSUP!?” - A HIP HOP PRODUCER
15. OHHHHHHH SHIT. 15
16. 16 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); // ... } catch (Exception $e) { echo "<h1>Oh shit.</h1>"; } I FORGOT TO REMOVE THIS. THAT IS A BUG. A BAD ONE.
19. 19 DONE.
21. 2003-Current: Test. Test. Test. If it doesn’t have a test, it’s legacy code. Write tests for existing features, new features, etc. Testing creates confidence. 21
22. 2003-2009: Testing with .NET and Rails ▪ .NET - nUnit ▫ RhinoMocks for mocks ▫ Selenium for UI Automation ▪ Ruby on Rails- Test::Unit ▫ Dynamic Language Tricks for Mocks ▫ Selenium / Capybara for UI Automation 22
23. 2009-2011: Built over 30 Android Apps ▪ Over 10MM+ Installs ▪ All a Hodge Podge Spaghetti Mess ▪ ALL HAD ZERO TESTS 23
24. 2011-Groupon No tests when I started. No Espresso. Tons of manual QA. 24
26. How do we solve all these problems? 26
27. AUTOMATED TESTING Proper test coverage increases your confidence when you make a change to your code base and it is verified via a continuous integration server. 27
28. DIFFERENT TYPES OF TESTING INTEGRATION E2E UNIT 28
29. The Testing Pyramid 29
30. 30 The Testing Pyramid Explained Integration E2E
31. TYPES OF TESTING INTEGRATION E2E UNIT 31
32. 32 Unit Testing ▪ Usually best for lower level components that don’t require other dependencies to do their job. ▪ Example: ▫ A Calculator ▫ Simple Objects (POJO/POKO) ▫ Mapper/Adapter Pattern Implementations
33. 33 @Test fun `calculator can add`() { val calc = Calculator() assertThat(calc.add(2, 2)).isEqualTo(4) }
34. 34 class Calculator { fun add(first: Int, second: Int) = first + second // other functions } No outside integrations or systems to rely on. It’s a unit of work that can be tested.
35. 35 data class Customer( val firstName: String, val lastName: String, val age: Int, val level: String, val favoriteColor: String ) data class CustomerViewModel( val fullName: String, val age: Int )
36. interface Mapper<InputType, ReturnType> { fun map(input: InputType): ReturnType } class CustomerViewModelMapper : Mapper<Customer, CustomerViewModel> { override fun map(c: Customer): CustomerViewModel { return CustomerViewModel("${c.firstName} ${c.lastName}", c.age) } }
37. 37 @Test fun `should be able to map a customer view model from a customer object`() { val customer = Customer("Donn", "Felker", 66, "platinum", "Blue") val mapper: Mapper<Customer, CustomerViewModel> = CustomerViewModelMapper() val vm: CustomerViewModel = mapper.map(customer) assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}") assertThat(vm.age).isEqualTo(customer.age) }
38. 38 ARRANGE.ACT.ASSERT. @Test fun `should be able to map a customer view model from a customer object`() { // Arrange val customer = Customer("Donn", "Felker", 66, "platinum", "Blue") val mapper: Mapper<Customer, CustomerViewModel> = CustomerViewModelMapper() // Act val vm: CustomerViewModel = mapper.map(customer) // Assert assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}") assertThat(vm.age).isEqualTo(customer.age) }
39. INTEGRATION TESTING INTEGRATION E2E UNIT 39
40. 40 Class 1 Class 2
41. 41 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) { fun ordersFor(id: Int): List<Order> { val legacyOrders = legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
42. 42 Integration Test: Verifying Values @Test fun `when both apis are called both the orders are combined and returned`() { val legacyApi: LegacyApi = mock() val api: Api = mock() val legacyOrders = listOf(Order(1), Order(2)) whenever(legacyApi.getOrders(anyInt())).thenReturn(legacyOrders) val newOrders = listOf(Order(8), Order(9)) whenever(api.getOrders(anyInt())).thenReturn(newOrders) val aggregator = OrderAggregator(legacyApi, api) val aggregatedOrders = aggregator.ordersFor(42) val expected = newOrders.union(legacyOrders).toList() assertEquals(expected, aggregatedOrders) }
43. 43 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) { fun ordersFor(id: Int): List<Order> { val legacyOrders: List<Order> = listOf() //legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
44. 44 Integration Test: Verifying Behavior @Test fun `both apis should be called when retrieving orders`() { val legacyApi: LegacyApi = mock() val api: Api = mock() val aggregator = OrderAggregator(legacyApi, api) aggregator.ordersFor(42) verify(legacyApi).getOrders(42) verify(api).getOrders(42) }
45. 45 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) { fun ordersFor(id: Int): List<Order> { val legacyOrders = legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
47. E2E TESTING (End to End Testing) INTEGRATION E2E UNIT 47
49. Business Logic/etc TEST THE WHOLE STACK (END TO END) Remote APIs/Services UI 49
50. Basic Espresso Test 50 @Test fun verifyThatTheUserCanLogIn(){ onView(withId(R.id.email)).perform(typeText("donn@donnfelker.com")) onView(withId(R.id.password)).perform(typeText("android123")) onView(withId(R.id.login_button)).perform(click()) onView(withId(R.id.login_message)).check(matches(withText("Success!"))) }
51. 70% Unit Tests, 20% Integration Tests, 10% End-to-End Tests 70/20/10 51 Google Testing Pyramid Advice
52. 70% Unit Tests, 20% Integration Tests, 10% End-to-End Tests 70/20/10 52 Google Testing Pyramid Advice
53. YOUR USERS DON’T CARE ABOUT UNIT TESTS THEY CARE IF THE APP WORKS OR NOT. 53
55. 55 Declaring Package Bankruptcy (When your app has so many 1-star reviews that you cannot recover from it so you have to abandon the package name and upload a new one.)
56. 60% End-to-End Tests 40% Unit Tests/Integration 60:40 56
57. END-TO-END TESTS ARE THE MOST VALUABLE TESTS YOU HAVE 57
58. END-TO-END TESTS EMULATE REAL USERS 58
59. MOST END-TO-END TEST SUITES 59
60. Business Logic/etc TEST THE WHOLE STACK (END TO END) Remote APIs/Services UI 60 Bluetooth Camera APIs Remote APIs Networking etc
61. Your tests should have reproducible results. HERMETIC 61
63. HOW TO MAKE A TEST HERMETIC ▪ Remove the variance & external influence ▫ Remove the network ▫ Remove shared files ▪ Sanitize the Testing Environment ▪ Create Repeatable Environments 63
64. DECOUPLE VIA ARCHITECTURE 64
65. 65 DEPENDENCY INJECTION IS KEY
66. Remote APIS Bluetooth APIS Camera APIS Contact Picker Content Resolvers COMMON DIFFICULT AREAS OF TESTING 66
67. Bluetooth APIS → BlueTooth Delegate Camera APIS → Camera Delegate Contact Picker → Contact Picker Delegate Content Resolver → … yup ... a delegate DELEGATE PATTERN 67
68. 68 public void onActivityResult(int reqCode, int resultCode, Intent data) { super.onActivityResult(reqCode, resultCode, data); switch (reqCode) { case (PICK_CONTACT): if (resultCode == Activity.RESULT_OK) { Uri contactData = data.getData(); Cursor c = managedQuery(contactData, null, null, null, null); if (c.moveToFirst()) { String id = c.getString(c.getColumnIndexOrThrow(ContactsContract.Contacts._ID)); String hasPhone = c.getString(c.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER)); try { if (hasPhone.equalsIgnoreCase("1")) { Cursor phones = getContentResolver().query( ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null, ContactsContract.CommonDataKinds.Phone.CONTACT_ID + " = " + id, null, null); phones.moveToFirst(); String cNumber = phones.getString(phones.getColumnIndex("data1")); System.out.println("number is:" + cNumber); txtphno.setText("Phone Number is: "+cNumber); } String name = c.getString(c.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME)); txtname.setText("Name is: "+name); } catch (Exception ex) { st.getMessage(); } } } break; } }
69. 69 data class Contact(val name: String, val number: String) interface ContactRepository { fun getContactFor(uri: Uri): Contact } class ContentResolverDelegate(context: Context) : ContactRepository { override fun getContactFor(uri: Uri): Contact { val contactData = uri.data val c = context.contentResolver... if (c.moveToFirst()) { // blah blah blah } } }
70. 70 class FakeContactRepository : ContactRepository { lateinit var contact: Contact fun loadContact(contact: Contact) { this.contact = contact } override fun getContactFor(uri: Uri): Contact { return contact } }
71. 71 override protected fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent) { // inject the contactRepository via DI val contact = contactRepository.getContactFor(data.data) // do stuff with the contact }
73. 73 class ContactTest { @Inject var repository: FakeContactRepository @get:Rule var activityRule: ActivityTestRule<MainActivity> = ActivityTestRule(MainActivity::class.java) @Before fun setup() { // do injection with your DI of choice so you can get a handle on the fake repository } @Test fun verifyThatUserCanSendMessageToFriendThatWasPickedFromContactPicker() { val stubResult = Intent() stubResult.data = Uri.EMPTY val activityResult = Instrumentation.ActivityResult(RESULT_OK, stubResult) Intents.Intending(IntentMatchers.hasAction(Intent.ACTION_PICK)).respondWith(activityResult) val expectedContact = Contact("Donn Felker", "888-867-5309") repository.loadContact(expectedContact) activityRule.launchActivity(intent) // Starts contact picker, the intents api kicks in and takes over // returning our expected intent onView(withId(R.id.choose_friend)).perform(click()) onView(withId(R.id.friend_header)).check(matches(withText("Send to: Donn Felker"))) onView(withId(R.id.friend_number)).check(matches(withText("Number: 888-867-5309"))) } }
74. 74 Other Option - More work - More E2E ▪ Write some test code to manually insert (and remove on tear down) contacts into the android content resolver. ▪ Very difficult, if not impossible, for things like bluetooth, wireless, etc ▪ Removing more of the “system” does make it hermetic, but at a cost.
75. In E2E Tests Remove/Replace only what you absolutely have to. If it a component makes a test(s) flakey it might be a good candidate. 75
76. REMOVING THE NETWORK 76 Creating repeatable network calls
77. #1 Problem: The network is not reliable 77
78. Tools to Remove the Network ▪ MockWebServer ▪ WireMock ▪ Hoverfly ▪ Custom Mocks/Fakes/etc 78
79. Mocking the Network Services API UI 79 API Server(s) MockWebServer WireMock Custom/etc
80. Things you can do with these tools ▪ Return known responses ▪ Return HTTP errors ▪ Slow down the network ▪ Simulate very difficult HTTP Scenarios ▪ Get a reliable/measurable speed in your E2E tests that is not relate to network ▪ Make your API calls hermetic in test 80
81. WireMock: https://github.com/handstandsam/AndroidHttpMockingExamples MockWebServer: https://caster.io/courses/mockwebserver Resources 81
82. E2E HEURISTICS 82 Get the most out of Espresso by using these tools, tips and frameworks to speed up and stabilize your tests.
83. 83 MOCK OUT DIFFICULT TESTING AREAS
84. USE A CI SERVER 84
85. USE ANDROID TEST ORCHESTRATOR 85 Minimal shared state. Each test runs in its own Instrumentation instance. Therefore, if your tests share app state, most of that shared state is removed from your device's CPU or memory after each test. Crashes are isolated. Even if one test crashes, it takes down only its own instance of Instrumentation, so the other tests in your suite still run. https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
86. SHARD YOUR TESTS Parallelize your tests. Specify how many shards and how many tests to run. Run on various devices 86 https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/sharding
87. 87 ANDROID_SERIAL=emulator-5554 ./gradlew connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.numShards=3 -Pandroid.testInstrumentationRunnerArguments.shardIndex=0 // Runs first half of the tests adb -s DEVICE_1_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 0 > device_1_results // Runs second half of the tests adb -s DEVICE_2_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 1 > device_2_results VIA ADB VIA GRADLE
88. USE FLANK (ONLY IF YOU USE Firebase Test Lab) 88 Massively parallel Android and iOS test runner for Firebase Test Lab Built in Sharding Built in retry for flakey tests Set max run time per shard FTL Does cost $$$ https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
89. FINDING SUCCESS Applying the pareto principle to testing. 89
90. 90 My app doesn’t have any tests. Where do I start?
91. 91 Start with End-to-End Tests
92. Each test is an island of safety in an ocean of bugs. 92
93. 93 WHEN DO I WRITE UNIT & INTEGRATION TESTS?
94. 94 When you’re Test Driving a Feature. Why? - It forces good design.
95. Where does TDD fit into this? 95
96. 96 SOMETIMES YOU DON’T KNOW WHAT YOU DON’T KNOW. Do it when you can, if you can’t … write tests afterwards that cover your changes.
97. 60% End-to-End Tests 40% Unit Tests/Integration 60:40 97 GUIDING TARGET (but NOT required)
98. 98 WHERE TO GO FROM HERE...
99. 99 ADOPT AN E2E MINDSET. E2E TESTS MIMIC REAL USAGE. REAL USAGE = REAL USERS
100. 100 REMEMBER USERS DON’T CARE THAT YOUR UNIT TESTS PASS. THEY CARE THAT YOUR APP WORKS. SO ... WRITE SOME E2E TESTS.
102. Thank You You can find me at: @donnfelker (Twitter and IG) donnfelker.com 102