Testing for Success in the Real World

Does that code you just wrote actually work? How do you know? How do your teammates know? You did write tests for that, right? What kind did you write though? Unit? Integration? System? End-to-End? What about mocking and stubbing? I know, you only changed two lines of a legacy piece of the app… but still, how do you know this didn’t break anything?

Ugh. Let’s face it testing your application is difficult and tedious. Where can you get the most bang for your buck? What’s the 20% of work that gets you 80% of the return? In this session you’ll learn where you can focus your attention to gain the most traction in your testing endeavors. From mocking api calls, to juxtaposing the benefits of unit testing vs end-to-end testing, we’ll cover it all.

Transcript

1. TESTING FOR SUCCESS
   IN THE REAL WORLD
   

   View Slide

2. assertThat(name).isEqualTo(“Donn Felker”)
   @donnfelker
   2
   Fragmented Podcast
   donnfelker.com
   

   View Slide

3. MY HISTORY WITH TESTING
   3
   

   View Slide

4. 4
   1999-2003: Wild West Cowboy Coder
   ▪ What’s a test?
   ▪ “It works on my box.”
   ▪ Yeah, I tested it. Just
   hit “refresh” and you’ll
   see that it still works.
   
   ▪ Didn’t really know what
   testing was/how to do
   it.
   

   View Slide

5. Debugging and Manual Verification Testing
   ▪ alert(“in here”)
   ▪ console.log(“in here”)
   ▪ Response.Write(“in here”)
   ▪ echo “in here”
   ▪ Log.d(“MainActivity”, “in here”)
   5
   

   View Slide

6. 2002: Start Working in the Music Industry
   6
   

   View Slide

7. 7
   HEADER
   SIDEBAR
   MAIN MENU
   HEADER MENU
   FEATURE
   SIDEBAR
   CONTENT
   FEATURE
   IMAGE
   LINKS
   

   View Slide

8. 8
   try {
   mysql_connect("localhost", "someuser", "somepass");
   $selected = mysql_select_db("mydb");
   // ...
   } catch (Exception $e) {
   // do nothing
   }
   

   View Slide

9. 9
   try {
   mysql_connect("localhost", "someuser", "somepass");
   $selected = mysql_select_db("mydb");
   // ...
   } catch (Exception $e) {
   echo "Oh shit.";
   }
   

   View Slide

10. 10
    WEBSITE LAUNCHED
    
    

    View Slide

11. 11
    FAST FORWARD 9 MONTHS
    

    View Slide

12. 12
    
    “YO, MY WEBSITE IS COVERED IN SHIT!
    IT’S NOT A GOOD LOOK, SON! WASSUP!?”
    - A HIP HOP PRODUCER
    

    View Slide

13. View Slide

14. View Slide

15. OHHHHHHH SHIT.
    
    15
    

    View Slide

16. 16
    try {
    mysql_connect("localhost", "someuser", "somepass");
    $selected = mysql_select_db("mydb");
    // ...
    } catch (Exception $e) {
    echo "Oh shit.";
    }
    I FORGOT TO REMOVE THIS.
    THAT IS A BUG. A BAD ONE.
    

    View Slide

17. View Slide

18. View Slide

19. 19
    DONE.
    

    View Slide

20. View Slide

21. 2003-Current: Test. Test. Test.
    If it doesn’t have a test,
    it’s legacy code.
    Write tests for existing
    features, new features,
    etc.
    Testing creates confidence.
    21
    

    View Slide

22. 2003-2009: Testing with .NET and Rails
    ▪ .NET - nUnit
    ▫ RhinoMocks for mocks
    ▫ Selenium for UI Automation
    ▪ Ruby on Rails- Test::Unit
    ▫ Dynamic Language Tricks for Mocks
    ▫ Selenium / Capybara for UI Automation
    22
    

    View Slide

23. 2009-2011: Built over 30 Android Apps
    ▪ Over 10MM+ Installs
    ▪ All a Hodge Podge Spaghetti Mess
    ▪ ALL HAD ZERO TESTS
    23
    

    View Slide

24. 2011-Groupon
    No tests when I started.
    No Espresso.
    Tons of manual QA.
    24
    

    View Slide

25. View Slide

26. How do we solve all these
    problems?
    26
    

    View Slide

27. AUTOMATED TESTING
    Proper test coverage increases your confidence when you
    make a change to your code base and it is verified via
    a continuous integration server.
    27
    

    View Slide

28. DIFFERENT TYPES OF TESTING
    INTEGRATION
    E2E
    UNIT
    28
    

    View Slide

29. The Testing Pyramid
    29
    

    View Slide

30. 30
    The Testing Pyramid Explained
    Integration
    E2E
    

    View Slide

31. TYPES OF TESTING
    INTEGRATION
    E2E
    UNIT
    31
    

    View Slide

32. 32
    Unit Testing
    ▪ Usually best for lower level components
    that don’t require other dependencies to
    do their job.
    ▪ Example:
    ▫ A Calculator
    ▫ Simple Objects (POJO/POKO)
    ▫ Mapper/Adapter Pattern Implementations
    

    View Slide

33. 33
    @Test
    fun `calculator can add`() {
    val calc = Calculator()
    assertThat(calc.add(2, 2)).isEqualTo(4)
    }
    

    View Slide

34. 34
    class Calculator {
    fun add(first: Int, second: Int) = first + second
    // other functions
    }
    No outside integrations or systems to
    rely on. It’s a unit of work that can be
    tested.
    

    View Slide

35. 35
    data class Customer(
    val firstName: String,
    val lastName: String,
    val age: Int,
    val level: String,
    val favoriteColor: String
    )
    data class CustomerViewModel(
    val fullName: String,
    val age: Int
    )
    

    View Slide

36. interface Mapper {
    fun map(input: InputType): ReturnType
    }
    class CustomerViewModelMapper : Mapper {
    override fun map(c: Customer): CustomerViewModel {
    return CustomerViewModel("${c.firstName} ${c.lastName}", c.age)
    }
    }
    

    View Slide

37. 37
    @Test
    fun `should be able to map a customer view model from a customer object`() {
    val customer = Customer("Donn", "Felker", 66, "platinum", "Blue")
    val mapper: Mapper = CustomerViewModelMapper()
    val vm: CustomerViewModel = mapper.map(customer)
    assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}")
    assertThat(vm.age).isEqualTo(customer.age)
    }
    

    View Slide

38. 38
    ARRANGE.ACT.ASSERT.
    @Test
    fun `should be able to map a customer view model from a customer object`() {
    // Arrange
    val customer = Customer("Donn", "Felker", 66, "platinum", "Blue")
    val mapper: Mapper = CustomerViewModelMapper()
    // Act
    val vm: CustomerViewModel = mapper.map(customer)
    // Assert
    assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}")
    assertThat(vm.age).isEqualTo(customer.age)
    }
    

    View Slide

39. INTEGRATION TESTING
    INTEGRATION
    E2E
    UNIT
    39
    

    View Slide

40. 40
    Class 1 Class 2
    

    View Slide

41. 41
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders = legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }
    

    View Slide

42. 42
    Integration Test: Verifying Values
    @Test
    fun `when both apis are called both the orders are combined and returned`() {
    val legacyApi: LegacyApi = mock()
    val api: Api = mock()
    val legacyOrders = listOf(Order(1), Order(2))
    whenever(legacyApi.getOrders(anyInt())).thenReturn(legacyOrders)
    val newOrders = listOf(Order(8), Order(9))
    whenever(api.getOrders(anyInt())).thenReturn(newOrders)
    val aggregator = OrderAggregator(legacyApi, api)
    val aggregatedOrders = aggregator.ordersFor(42)
    val expected = newOrders.union(legacyOrders).toList()
    assertEquals(expected, aggregatedOrders)
    }
    

    View Slide

43. 43
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders: List = listOf() //legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }
    

    View Slide

44. 44
    Integration Test: Verifying Behavior
    @Test
    fun `both apis should be called when retrieving orders`() {
    val legacyApi: LegacyApi = mock()
    val api: Api = mock()
    val aggregator = OrderAggregator(legacyApi, api)
    aggregator.ordersFor(42)
    verify(legacyApi).getOrders(42)
    verify(api).getOrders(42)
    }
    

    View Slide

45. 45
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders = legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }
    

    View Slide

46. View Slide

47. E2E TESTING (End to End Testing)
    INTEGRATION
    E2E
    UNIT
    47
    

    View Slide

48. View Slide

49. Business Logic/etc
    TEST THE WHOLE STACK (END TO END)
    Remote APIs/Services
    UI
    49
    

    View Slide

50. Basic Espresso Test
    50
    @Test
    fun verifyThatTheUserCanLogIn(){
    onView(withId(R.id.email)).perform(typeText("donn@donnfelker.com"))
    onView(withId(R.id.password)).perform(typeText("android123"))
    onView(withId(R.id.login_button)).perform(click())
    onView(withId(R.id.login_message)).check(matches(withText("Success!")))
    }
    

    View Slide

51. 70% Unit Tests, 20% Integration Tests, 10%
    End-to-End Tests
    70/20/10
    51
    Google Testing Pyramid Advice
    

    View Slide

52. 70% Unit Tests, 20% Integration Tests, 10%
    End-to-End Tests
    70/20/10
    52
    Google Testing Pyramid Advice
    
    

    View Slide

53. YOUR USERS DON’T CARE
    ABOUT UNIT TESTS
    THEY CARE IF THE APP
    WORKS OR NOT.
    53
    

    View Slide

54. View Slide

55. 55
    Declaring Package Bankruptcy
    (When your app has so many 1-star reviews that
    you cannot recover from it so you have to
    abandon the package name and upload a new
    one.)
    

    View Slide

56. 60% End-to-End Tests
    40% Unit Tests/Integration
    60:40
    56
    

    View Slide

57. END-TO-END TESTS
    ARE THE MOST VALUABLE
    TESTS YOU HAVE
    57
    

    View Slide

58. END-TO-END TESTS
    EMULATE REAL USERS
    58
    

    View Slide

59. MOST END-TO-END
    TEST SUITES
    59
    

    View Slide

60. Business Logic/etc
    TEST THE WHOLE STACK (END TO END)
    Remote APIs/Services
    UI
    60
    
    
    
    Bluetooth
    Camera APIs
    Remote APIs
    Networking
    etc
    

    View Slide

61. Your tests should have reproducible results.
    HERMETIC
    61
    

    View Slide

62. View Slide

63. HOW TO MAKE A TEST HERMETIC
    ▪ Remove the variance & external influence
    ▫ Remove the network
    ▫ Remove shared files
    ▪ Sanitize the Testing Environment
    ▪ Create Repeatable Environments
    63
    

    View Slide

64. DECOUPLE VIA
    ARCHITECTURE
    64
    

    View Slide

65. 65
    DEPENDENCY INJECTION
    IS KEY
    

    View Slide

66. Remote APIS
    Bluetooth APIS
    Camera APIS
    Contact Picker
    Content Resolvers
    COMMON DIFFICULT AREAS OF TESTING
    66
    

    View Slide

67. Bluetooth APIS → BlueTooth Delegate
    Camera APIS → Camera Delegate
    Contact Picker → Contact Picker Delegate
    Content Resolver → … yup ... a delegate
    DELEGATE PATTERN
    67
    

    View Slide

68. 68
    public void onActivityResult(int reqCode, int resultCode, Intent data) {
    super.onActivityResult(reqCode, resultCode, data);
    switch (reqCode) {
    case (PICK_CONTACT):
    if (resultCode == Activity.RESULT_OK) {
    Uri contactData = data.getData();
    Cursor c = managedQuery(contactData, null, null, null, null);
    if (c.moveToFirst()) {
    String id = c.getString(c.getColumnIndexOrThrow(ContactsContract.Contacts._ID));
    String hasPhone = c.getString(c.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER));
    try {
    if (hasPhone.equalsIgnoreCase("1")) {
    Cursor phones = getContentResolver().query(
    ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null,
    ContactsContract.CommonDataKinds.Phone.CONTACT_ID + " = " + id,
    null, null);
    phones.moveToFirst();
    String cNumber = phones.getString(phones.getColumnIndex("data1"));
    System.out.println("number is:" + cNumber);
    txtphno.setText("Phone Number is: "+cNumber);
    }
    String name = c.getString(c.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
    txtname.setText("Name is: "+name);
    }
    catch (Exception ex)
    {
    st.getMessage();
    }
    }
    }
    break;
    }
    }
    
    

    View Slide

69. 69
    data class Contact(val name: String, val number: String)
    interface ContactRepository {
    fun getContactFor(uri: Uri): Contact
    }
    class ContentResolverDelegate(context: Context) : ContactRepository {
    override fun getContactFor(uri: Uri): Contact {
    val contactData = uri.data
    val c = context.contentResolver...
    if (c.moveToFirst()) {
    // blah blah blah
    }
    }
    }
    

    View Slide

70. 70
    class FakeContactRepository : ContactRepository {
    lateinit var contact: Contact
    fun loadContact(contact: Contact) {
    this.contact = contact
    }
    override fun getContactFor(uri: Uri): Contact {
    return contact
    }
    }
    

    View Slide

71. 71
    override protected fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent)
    {
    // inject the contactRepository via DI
    val contact = contactRepository.getContactFor(data.data)
    // do stuff with the contact
    }
    

    View Slide

72. View Slide

73. 73
    class ContactTest {
    @Inject var repository: FakeContactRepository
    @get:Rule
    var activityRule: ActivityTestRule = ActivityTestRule(MainActivity::class.java)
    @Before
    fun setup() {
    // do injection with your DI of choice so you can get a handle on the fake repository
    }
    @Test
    fun verifyThatUserCanSendMessageToFriendThatWasPickedFromContactPicker() {
    val stubResult = Intent()
    stubResult.data = Uri.EMPTY
    val activityResult = Instrumentation.ActivityResult(RESULT_OK, stubResult)
    Intents.Intending(IntentMatchers.hasAction(Intent.ACTION_PICK)).respondWith(activityResult)
    val expectedContact = Contact("Donn Felker", "888-867-5309")
    repository.loadContact(expectedContact)
    activityRule.launchActivity(intent)
    // Starts contact picker, the intents api kicks in and takes over
    // returning our expected intent
    onView(withId(R.id.choose_friend)).perform(click())
    onView(withId(R.id.friend_header)).check(matches(withText("Send to: Donn Felker")))
    onView(withId(R.id.friend_number)).check(matches(withText("Number: 888-867-5309")))
    }
    }
    

    View Slide

74. 74
    Other Option - More work - More E2E
    ▪ Write some test code to manually insert
    (and remove on tear down) contacts into
    the android content resolver.
    ▪ Very difficult, if not impossible, for
    things like bluetooth, wireless, etc
    ▪ Removing more of the “system” does make
    it hermetic, but at a cost.
    

    View Slide

75. In E2E Tests Remove/Replace only
    what you absolutely have to.
    If it a component makes a test(s)
    flakey it might be a good
    candidate.
    75
    

    View Slide

76. REMOVING THE NETWORK
    76
    Creating repeatable
    network calls
    

    View Slide

77. #1 Problem:
    The network is
    not reliable
    77
    

    View Slide

78. Tools to Remove the Network
    ▪ MockWebServer
    ▪ WireMock
    ▪ Hoverfly
    ▪ Custom Mocks/Fakes/etc
    78
    

    View Slide

79. Mocking the Network
    Services
    API
    UI
    79
    API
    Server(s)
    MockWebServer
    WireMock
    Custom/etc
    

    View Slide

80. Things you can do with these tools
    ▪ Return known responses
    ▪ Return HTTP errors
    ▪ Slow down the network
    ▪ Simulate very difficult HTTP Scenarios
    ▪ Get a reliable/measurable speed in your
    E2E tests that is not relate to network
    ▪ Make your API calls hermetic in test
    80
    

    View Slide

81. WireMock: https://github.com/handstandsam/AndroidHttpMockingExamples
    MockWebServer: https://caster.io/courses/mockwebserver
    Resources
    81
    

    View Slide

82. E2E HEURISTICS
    82
    Get the most out of Espresso by using these tools, tips
    and frameworks to speed up and stabilize your tests.
    

    View Slide

83. 83
    MOCK OUT
    DIFFICULT
    TESTING AREAS
    

    View Slide

84. USE A CI SERVER
    84
    

    View Slide

85. USE ANDROID TEST ORCHESTRATOR
    85
    Minimal shared state. Each test
    runs in its own Instrumentation
    instance. Therefore, if your tests
    share app state, most of that
    shared state is removed from your
    device's CPU or memory after each
    test.
    Crashes are isolated. Even if one
    test crashes, it takes down only
    its own instance of
    Instrumentation, so the other tests
    in your suite still run.
    https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
    

    View Slide

86. SHARD YOUR TESTS
    Parallelize your tests.
    Specify how many shards and
    how many tests to run.
    Run on various devices
    86
    https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/sharding
    

    View Slide

87. 87
    ANDROID_SERIAL=emulator-5554 ./gradlew connectedAndroidTest
    -Pandroid.testInstrumentationRunnerArguments.numShards=3
    -Pandroid.testInstrumentationRunnerArguments.shardIndex=0
    // Runs first half of the tests
    adb -s DEVICE_1_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 0 > device_1_results
    // Runs second half of the tests
    adb -s DEVICE_2_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 1 > device_2_results
    VIA ADB
    VIA GRADLE
    

    View Slide

88. USE FLANK (ONLY IF YOU USE Firebase Test Lab)
    88
    Massively parallel Android and iOS
    test runner for Firebase Test Lab
    Built in Sharding
    Built in retry for flakey tests
    Set max run time per shard
    FTL Does cost $$$
    https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator
    

    View Slide

89. FINDING SUCCESS
    Applying the pareto principle to testing.
    89
    

    View Slide

90. 90
    My app doesn’t have any tests.
    Where do I start?
    

    View Slide

91. 91
    Start with
    End-to-End Tests
    

    View Slide

92. Each test is an island of safety
    in an ocean of bugs.
    92
    

    View Slide

93. 93
    WHEN DO I WRITE UNIT &
    INTEGRATION TESTS?
    

    View Slide

94. 94
    When you’re Test
    Driving a Feature.
    Why? - It forces good design.
    

    View Slide

95. Where does TDD fit
    into this?
    95
    

    View Slide

96. 96
    SOMETIMES YOU DON’T KNOW
    WHAT YOU DON’T KNOW.
    Do it when you can, if you can’t … write tests
    afterwards that cover your changes.
    

    View Slide

97. 60% End-to-End Tests
    40% Unit Tests/Integration
    60:40
    97
    GUIDING TARGET (but NOT required)
    

    View Slide

98. 98
    WHERE TO GO FROM HERE...
    

    View Slide

99. 99
    ADOPT AN E2E MINDSET.
    E2E TESTS MIMIC REAL USAGE.
    REAL USAGE = REAL USERS
    

    View Slide

100. 100
     REMEMBER
     USERS DON’T CARE THAT YOUR UNIT
     TESTS PASS.
     THEY CARE THAT YOUR APP WORKS.
     SO ... WRITE SOME E2E TESTS.
     

     View Slide

101. View Slide

102. Thank You
     You can find me at:
     @donnfelker (Twitter and IG)
     donnfelker.com
     102
     

     View Slide