Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing for Success in the Real World

Testing for Success in the Real World

Does that code you just wrote actually work? How do you know? How do your teammates know? You did write tests for that, right? What kind did you write though? Unit? Integration? System? End-to-End? What about mocking and stubbing? I know, you only changed two lines of a legacy piece of the app… but still, how do you know this didn’t break anything?

Ugh. Let’s face it testing your application is difficult and tedious. Where can you get the most bang for your buck? What’s the 20% of work that gets you 80% of the return? In this session you’ll learn where you can focus your attention to gain the most traction in your testing endeavors. From mocking api calls, to juxtaposing the benefits of unit testing vs end-to-end testing, we’ll cover it all.

Daf1617c9a4ff129239e922e8c56af1b?s=128

Donn Felker

August 15, 2019
Tweet

Transcript

  1. TESTING FOR SUCCESS IN THE REAL WORLD

  2. assertThat(name).isEqualTo(“Donn Felker”) @donnfelker 2 Fragmented Podcast donnfelker.com

  3. MY HISTORY WITH TESTING 3

  4. 4 1999-2003: Wild West Cowboy Coder ▪ What’s a test?

    ▪ “It works on my box.” ▪ Yeah, I tested it. Just hit “refresh” and you’ll see that it still works. ▪ Didn’t really know what testing was/how to do it.
  5. Debugging and Manual Verification Testing ▪ alert(“in here”) ▪ console.log(“in

    here”) ▪ Response.Write(“in here”) ▪ echo “in here” ▪ Log.d(“MainActivity”, “in here”) 5
  6. 2002: Start Working in the Music Industry 6

  7. 7 HEADER SIDEBAR MAIN MENU HEADER MENU FEATURE SIDEBAR CONTENT

    FEATURE IMAGE LINKS
  8. 8 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); //

    ... } catch (Exception $e) { // do nothing }
  9. 9 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); //

    ... } catch (Exception $e) { echo "<h1>Oh shit.</h1>"; }
  10. 10 WEBSITE LAUNCHED

  11. 11 FAST FORWARD 9 MONTHS

  12. 12 “YO, MY WEBSITE IS COVERED IN SHIT! IT’S NOT

    A GOOD LOOK, SON! WASSUP!?” - A HIP HOP PRODUCER
  13. 13

  14. 14

  15. OHHHHHHH SHIT. 15

  16. 16 try { mysql_connect("localhost", "someuser", "somepass"); $selected = mysql_select_db("mydb"); //

    ... } catch (Exception $e) { echo "<h1>Oh shit.</h1>"; } I FORGOT TO REMOVE THIS. THAT IS A BUG. A BAD ONE.
  17. 17

  18. 18

  19. 19 DONE.

  20. 20

  21. 2003-Current: Test. Test. Test. If it doesn’t have a test,

    it’s legacy code. Write tests for existing features, new features, etc. Testing creates confidence. 21
  22. 2003-2009: Testing with .NET and Rails ▪ .NET - nUnit

    ▫ RhinoMocks for mocks ▫ Selenium for UI Automation ▪ Ruby on Rails- Test::Unit ▫ Dynamic Language Tricks for Mocks ▫ Selenium / Capybara for UI Automation 22
  23. 2009-2011: Built over 30 Android Apps ▪ Over 10MM+ Installs

    ▪ All a Hodge Podge Spaghetti Mess ▪ ALL HAD ZERO TESTS 23
  24. 2011-Groupon No tests when I started. No Espresso. Tons of

    manual QA. 24
  25. 25

  26. How do we solve all these problems? 26

  27. AUTOMATED TESTING Proper test coverage increases your confidence when you

    make a change to your code base and it is verified via a continuous integration server. 27
  28. DIFFERENT TYPES OF TESTING INTEGRATION E2E UNIT 28

  29. The Testing Pyramid 29

  30. 30 The Testing Pyramid Explained Integration E2E

  31. TYPES OF TESTING INTEGRATION E2E UNIT 31

  32. 32 Unit Testing ▪ Usually best for lower level components

    that don’t require other dependencies to do their job. ▪ Example: ▫ A Calculator ▫ Simple Objects (POJO/POKO) ▫ Mapper/Adapter Pattern Implementations
  33. 33 @Test fun `calculator can add`() { val calc =

    Calculator() assertThat(calc.add(2, 2)).isEqualTo(4) }
  34. 34 class Calculator { fun add(first: Int, second: Int) =

    first + second // other functions } No outside integrations or systems to rely on. It’s a unit of work that can be tested.
  35. 35 data class Customer( val firstName: String, val lastName: String,

    val age: Int, val level: String, val favoriteColor: String ) data class CustomerViewModel( val fullName: String, val age: Int )
  36. interface Mapper<InputType, ReturnType> { fun map(input: InputType): ReturnType } class

    CustomerViewModelMapper : Mapper<Customer, CustomerViewModel> { override fun map(c: Customer): CustomerViewModel { return CustomerViewModel("${c.firstName} ${c.lastName}", c.age) } }
  37. 37 @Test fun `should be able to map a customer

    view model from a customer object`() { val customer = Customer("Donn", "Felker", 66, "platinum", "Blue") val mapper: Mapper<Customer, CustomerViewModel> = CustomerViewModelMapper() val vm: CustomerViewModel = mapper.map(customer) assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}") assertThat(vm.age).isEqualTo(customer.age) }
  38. 38 ARRANGE.ACT.ASSERT. @Test fun `should be able to map a

    customer view model from a customer object`() { // Arrange val customer = Customer("Donn", "Felker", 66, "platinum", "Blue") val mapper: Mapper<Customer, CustomerViewModel> = CustomerViewModelMapper() // Act val vm: CustomerViewModel = mapper.map(customer) // Assert assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}") assertThat(vm.age).isEqualTo(customer.age) }
  39. INTEGRATION TESTING INTEGRATION E2E UNIT 39

  40. 40 Class 1 Class 2

  41. 41 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api)

    { fun ordersFor(id: Int): List<Order> { val legacyOrders = legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
  42. 42 Integration Test: Verifying Values @Test fun `when both apis

    are called both the orders are combined and returned`() { val legacyApi: LegacyApi = mock() val api: Api = mock() val legacyOrders = listOf(Order(1), Order(2)) whenever(legacyApi.getOrders(anyInt())).thenReturn(legacyOrders) val newOrders = listOf(Order(8), Order(9)) whenever(api.getOrders(anyInt())).thenReturn(newOrders) val aggregator = OrderAggregator(legacyApi, api) val aggregatedOrders = aggregator.ordersFor(42) val expected = newOrders.union(legacyOrders).toList() assertEquals(expected, aggregatedOrders) }
  43. 43 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api)

    { fun ordersFor(id: Int): List<Order> { val legacyOrders: List<Order> = listOf() //legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
  44. 44 Integration Test: Verifying Behavior @Test fun `both apis should

    be called when retrieving orders`() { val legacyApi: LegacyApi = mock() val api: Api = mock() val aggregator = OrderAggregator(legacyApi, api) aggregator.ordersFor(42) verify(legacyApi).getOrders(42) verify(api).getOrders(42) }
  45. 45 class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api)

    { fun ordersFor(id: Int): List<Order> { val legacyOrders = legacyApi.getOrders(id) val newOrders = api.getOrders(id) return newOrders.union(legacyOrders).toList() } }
  46. 46

  47. E2E TESTING (End to End Testing) INTEGRATION E2E UNIT 47

  48. 48

  49. Business Logic/etc TEST THE WHOLE STACK (END TO END) Remote

    APIs/Services UI 49
  50. Basic Espresso Test 50 @Test fun verifyThatTheUserCanLogIn(){ onView(withId(R.id.email)).perform(typeText("donn@donnfelker.com")) onView(withId(R.id.password)).perform(typeText("android123")) onView(withId(R.id.login_button)).perform(click())

    onView(withId(R.id.login_message)).check(matches(withText("Success!"))) }
  51. 70% Unit Tests, 20% Integration Tests, 10% End-to-End Tests 70/20/10

    51 Google Testing Pyramid Advice
  52. 70% Unit Tests, 20% Integration Tests, 10% End-to-End Tests 70/20/10

    52 Google Testing Pyramid Advice
  53. YOUR USERS DON’T CARE ABOUT UNIT TESTS THEY CARE IF

    THE APP WORKS OR NOT. 53
  54. 54

  55. 55 Declaring Package Bankruptcy (When your app has so many

    1-star reviews that you cannot recover from it so you have to abandon the package name and upload a new one.)
  56. 60% End-to-End Tests 40% Unit Tests/Integration 60:40 56

  57. END-TO-END TESTS ARE THE MOST VALUABLE TESTS YOU HAVE 57

  58. END-TO-END TESTS EMULATE REAL USERS 58

  59. MOST END-TO-END TEST SUITES 59

  60. Business Logic/etc TEST THE WHOLE STACK (END TO END) Remote

    APIs/Services UI 60 Bluetooth Camera APIs Remote APIs Networking etc
  61. Your tests should have reproducible results. HERMETIC 61

  62. 62

  63. HOW TO MAKE A TEST HERMETIC ▪ Remove the variance

    & external influence ▫ Remove the network ▫ Remove shared files ▪ Sanitize the Testing Environment ▪ Create Repeatable Environments 63
  64. DECOUPLE VIA ARCHITECTURE 64

  65. 65 DEPENDENCY INJECTION IS KEY

  66. Remote APIS Bluetooth APIS Camera APIS Contact Picker Content Resolvers

    COMMON DIFFICULT AREAS OF TESTING 66
  67. Bluetooth APIS → BlueTooth Delegate Camera APIS → Camera Delegate

    Contact Picker → Contact Picker Delegate Content Resolver → … yup ... a delegate DELEGATE PATTERN 67
  68. 68 public void onActivityResult(int reqCode, int resultCode, Intent data) {

    super.onActivityResult(reqCode, resultCode, data); switch (reqCode) { case (PICK_CONTACT): if (resultCode == Activity.RESULT_OK) { Uri contactData = data.getData(); Cursor c = managedQuery(contactData, null, null, null, null); if (c.moveToFirst()) { String id = c.getString(c.getColumnIndexOrThrow(ContactsContract.Contacts._ID)); String hasPhone = c.getString(c.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER)); try { if (hasPhone.equalsIgnoreCase("1")) { Cursor phones = getContentResolver().query( ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null, ContactsContract.CommonDataKinds.Phone.CONTACT_ID + " = " + id, null, null); phones.moveToFirst(); String cNumber = phones.getString(phones.getColumnIndex("data1")); System.out.println("number is:" + cNumber); txtphno.setText("Phone Number is: "+cNumber); } String name = c.getString(c.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME)); txtname.setText("Name is: "+name); } catch (Exception ex) { st.getMessage(); } } } break; } }
  69. 69 data class Contact(val name: String, val number: String) interface

    ContactRepository { fun getContactFor(uri: Uri): Contact } class ContentResolverDelegate(context: Context) : ContactRepository { override fun getContactFor(uri: Uri): Contact { val contactData = uri.data val c = context.contentResolver... if (c.moveToFirst()) { // blah blah blah } } }
  70. 70 class FakeContactRepository : ContactRepository { lateinit var contact: Contact

    fun loadContact(contact: Contact) { this.contact = contact } override fun getContactFor(uri: Uri): Contact { return contact } }
  71. 71 override protected fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent)

    { // inject the contactRepository via DI val contact = contactRepository.getContactFor(data.data) // do stuff with the contact }
  72. 72

  73. 73 class ContactTest { @Inject var repository: FakeContactRepository @get:Rule var

    activityRule: ActivityTestRule<MainActivity> = ActivityTestRule(MainActivity::class.java) @Before fun setup() { // do injection with your DI of choice so you can get a handle on the fake repository } @Test fun verifyThatUserCanSendMessageToFriendThatWasPickedFromContactPicker() { val stubResult = Intent() stubResult.data = Uri.EMPTY val activityResult = Instrumentation.ActivityResult(RESULT_OK, stubResult) Intents.Intending(IntentMatchers.hasAction(Intent.ACTION_PICK)).respondWith(activityResult) val expectedContact = Contact("Donn Felker", "888-867-5309") repository.loadContact(expectedContact) activityRule.launchActivity(intent) // Starts contact picker, the intents api kicks in and takes over // returning our expected intent onView(withId(R.id.choose_friend)).perform(click()) onView(withId(R.id.friend_header)).check(matches(withText("Send to: Donn Felker"))) onView(withId(R.id.friend_number)).check(matches(withText("Number: 888-867-5309"))) } }
  74. 74 Other Option - More work - More E2E ▪

    Write some test code to manually insert (and remove on tear down) contacts into the android content resolver. ▪ Very difficult, if not impossible, for things like bluetooth, wireless, etc ▪ Removing more of the “system” does make it hermetic, but at a cost.
  75. In E2E Tests Remove/Replace only what you absolutely have to.

    If it a component makes a test(s) flakey it might be a good candidate. 75
  76. REMOVING THE NETWORK 76 Creating repeatable network calls

  77. #1 Problem: The network is not reliable 77

  78. Tools to Remove the Network ▪ MockWebServer ▪ WireMock ▪

    Hoverfly ▪ Custom Mocks/Fakes/etc 78
  79. Mocking the Network Services API UI 79 API Server(s) MockWebServer

    WireMock Custom/etc
  80. Things you can do with these tools ▪ Return known

    responses ▪ Return HTTP errors ▪ Slow down the network ▪ Simulate very difficult HTTP Scenarios ▪ Get a reliable/measurable speed in your E2E tests that is not relate to network ▪ Make your API calls hermetic in test 80
  81. WireMock: https://github.com/handstandsam/AndroidHttpMockingExamples MockWebServer: https://caster.io/courses/mockwebserver Resources 81

  82. E2E HEURISTICS 82 Get the most out of Espresso by

    using these tools, tips and frameworks to speed up and stabilize your tests.
  83. 83 MOCK OUT DIFFICULT TESTING AREAS

  84. USE A CI SERVER 84

  85. USE ANDROID TEST ORCHESTRATOR 85 Minimal shared state. Each test

    runs in its own Instrumentation instance. Therefore, if your tests share app state, most of that shared state is removed from your device's CPU or memory after each test. Crashes are isolated. Even if one test crashes, it takes down only its own instance of Instrumentation, so the other tests in your suite still run. https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
  86. SHARD YOUR TESTS Parallelize your tests. Specify how many shards

    and how many tests to run. Run on various devices 86 https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/sharding
  87. 87 ANDROID_SERIAL=emulator-5554 ./gradlew connectedAndroidTest -Pandroid.testInstrumentationRunnerArguments.numShards=3 -Pandroid.testInstrumentationRunnerArguments.shardIndex=0 // Runs first half

    of the tests adb -s DEVICE_1_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 0 > device_1_results // Runs second half of the tests adb -s DEVICE_2_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 1 > device_2_results VIA ADB VIA GRADLE
  88. USE FLANK (ONLY IF YOU USE Firebase Test Lab) 88

    Massively parallel Android and iOS test runner for Firebase Test Lab Built in Sharding Built in retry for flakey tests Set max run time per shard FTL Does cost $$$ https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
  89. FINDING SUCCESS Applying the pareto principle to testing. 89

  90. 90 My app doesn’t have any tests. Where do I

    start?
  91. 91 Start with End-to-End Tests

  92. Each test is an island of safety in an ocean

    of bugs. 92
  93. 93 WHEN DO I WRITE UNIT & INTEGRATION TESTS?

  94. 94 When you’re Test Driving a Feature. Why? - It

    forces good design.
  95. Where does TDD fit into this? 95

  96. 96 SOMETIMES YOU DON’T KNOW WHAT YOU DON’T KNOW. Do

    it when you can, if you can’t … write tests afterwards that cover your changes.
  97. 60% End-to-End Tests 40% Unit Tests/Integration 60:40 97 GUIDING TARGET

    (but NOT required)
  98. 98 WHERE TO GO FROM HERE...

  99. 99 ADOPT AN E2E MINDSET. E2E TESTS MIMIC REAL USAGE.

    REAL USAGE = REAL USERS
  100. 100 REMEMBER USERS DON’T CARE THAT YOUR UNIT TESTS PASS.

    THEY CARE THAT YOUR APP WORKS. SO ... WRITE SOME E2E TESTS.
  101. 101

  102. Thank You You can find me at: @donnfelker (Twitter and

    IG) donnfelker.com 102