Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing for Success in the Real World

Testing for Success in the Real World

Does that code you just wrote actually work? How do you know? How do your teammates know? You did write tests for that, right? What kind did you write though? Unit? Integration? System? End-to-End? What about mocking and stubbing? I know, you only changed two lines of a legacy piece of the app… but still, how do you know this didn’t break anything?

Ugh. Let’s face it testing your application is difficult and tedious. Where can you get the most bang for your buck? What’s the 20% of work that gets you 80% of the return? In this session you’ll learn where you can focus your attention to gain the most traction in your testing endeavors. From mocking api calls, to juxtaposing the benefits of unit testing vs end-to-end testing, we’ll cover it all.

Donn Felker

August 15, 2019
Tweet

More Decks by Donn Felker

Other Decks in Technology

Transcript

  1. TESTING FOR SUCCESS
    IN THE REAL WORLD

    View Slide

  2. assertThat(name).isEqualTo(“Donn Felker”)
    @donnfelker
    2
    Fragmented Podcast
    donnfelker.com

    View Slide

  3. MY HISTORY WITH TESTING
    3

    View Slide

  4. 4
    1999-2003: Wild West Cowboy Coder
    ▪ What’s a test?
    ▪ “It works on my box.”
    ▪ Yeah, I tested it. Just
    hit “refresh” and you’ll
    see that it still works.

    ▪ Didn’t really know what
    testing was/how to do
    it.

    View Slide

  5. Debugging and Manual Verification Testing
    ▪ alert(“in here”)
    ▪ console.log(“in here”)
    ▪ Response.Write(“in here”)
    ▪ echo “in here”
    ▪ Log.d(“MainActivity”, “in here”)
    5

    View Slide

  6. 2002: Start Working in the Music Industry
    6

    View Slide

  7. 7
    HEADER
    SIDEBAR
    MAIN MENU
    HEADER MENU
    FEATURE
    SIDEBAR
    CONTENT
    FEATURE
    IMAGE
    LINKS

    View Slide

  8. 8
    try {
    mysql_connect("localhost", "someuser", "somepass");
    $selected = mysql_select_db("mydb");
    // ...
    } catch (Exception $e) {
    // do nothing
    }

    View Slide

  9. 9
    try {
    mysql_connect("localhost", "someuser", "somepass");
    $selected = mysql_select_db("mydb");
    // ...
    } catch (Exception $e) {
    echo "Oh shit.";
    }

    View Slide

  10. 10
    WEBSITE LAUNCHED

    View Slide

  11. 11
    FAST FORWARD 9 MONTHS

    View Slide

  12. 12

    “YO, MY WEBSITE IS COVERED IN SHIT!
    IT’S NOT A GOOD LOOK, SON! WASSUP!?”
    - A HIP HOP PRODUCER

    View Slide


  13. 13

    View Slide

  14. 14

    View Slide

  15. OHHHHHHH SHIT.

    15

    View Slide

  16. 16
    try {
    mysql_connect("localhost", "someuser", "somepass");
    $selected = mysql_select_db("mydb");
    // ...
    } catch (Exception $e) {
    echo "Oh shit.";
    }
    I FORGOT TO REMOVE THIS.
    THAT IS A BUG. A BAD ONE.

    View Slide

  17. 17

    View Slide

  18. 18

    View Slide

  19. 19
    DONE.

    View Slide

  20. 20

    View Slide

  21. 2003-Current: Test. Test. Test.
    If it doesn’t have a test,
    it’s legacy code.
    Write tests for existing
    features, new features,
    etc.
    Testing creates confidence.
    21

    View Slide

  22. 2003-2009: Testing with .NET and Rails
    ▪ .NET - nUnit
    ▫ RhinoMocks for mocks
    ▫ Selenium for UI Automation
    ▪ Ruby on Rails- Test::Unit
    ▫ Dynamic Language Tricks for Mocks
    ▫ Selenium / Capybara for UI Automation
    22

    View Slide

  23. 2009-2011: Built over 30 Android Apps
    ▪ Over 10MM+ Installs
    ▪ All a Hodge Podge Spaghetti Mess
    ▪ ALL HAD ZERO TESTS
    23

    View Slide

  24. 2011-Groupon
    No tests when I started.
    No Espresso.
    Tons of manual QA.
    24

    View Slide

  25. 25

    View Slide

  26. How do we solve all these
    problems?
    26

    View Slide

  27. AUTOMATED TESTING
    Proper test coverage increases your confidence when you
    make a change to your code base and it is verified via
    a continuous integration server.
    27

    View Slide

  28. DIFFERENT TYPES OF TESTING
    INTEGRATION
    E2E
    UNIT
    28

    View Slide

  29. The Testing Pyramid
    29

    View Slide

  30. 30
    The Testing Pyramid Explained
    Integration
    E2E

    View Slide

  31. TYPES OF TESTING
    INTEGRATION
    E2E
    UNIT
    31

    View Slide

  32. 32
    Unit Testing
    ▪ Usually best for lower level components
    that don’t require other dependencies to
    do their job.
    ▪ Example:
    ▫ A Calculator
    ▫ Simple Objects (POJO/POKO)
    ▫ Mapper/Adapter Pattern Implementations

    View Slide

  33. 33
    @Test
    fun `calculator can add`() {
    val calc = Calculator()
    assertThat(calc.add(2, 2)).isEqualTo(4)
    }

    View Slide

  34. 34
    class Calculator {
    fun add(first: Int, second: Int) = first + second
    // other functions
    }
    No outside integrations or systems to
    rely on. It’s a unit of work that can be
    tested.

    View Slide

  35. 35
    data class Customer(
    val firstName: String,
    val lastName: String,
    val age: Int,
    val level: String,
    val favoriteColor: String
    )
    data class CustomerViewModel(
    val fullName: String,
    val age: Int
    )

    View Slide

  36. interface Mapper {
    fun map(input: InputType): ReturnType
    }
    class CustomerViewModelMapper : Mapper {
    override fun map(c: Customer): CustomerViewModel {
    return CustomerViewModel("${c.firstName} ${c.lastName}", c.age)
    }
    }

    View Slide

  37. 37
    @Test
    fun `should be able to map a customer view model from a customer object`() {
    val customer = Customer("Donn", "Felker", 66, "platinum", "Blue")
    val mapper: Mapper = CustomerViewModelMapper()
    val vm: CustomerViewModel = mapper.map(customer)
    assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}")
    assertThat(vm.age).isEqualTo(customer.age)
    }

    View Slide

  38. 38
    ARRANGE.ACT.ASSERT.
    @Test
    fun `should be able to map a customer view model from a customer object`() {
    // Arrange
    val customer = Customer("Donn", "Felker", 66, "platinum", "Blue")
    val mapper: Mapper = CustomerViewModelMapper()
    // Act
    val vm: CustomerViewModel = mapper.map(customer)
    // Assert
    assertThat(vm.fullName).isEqualTo("${customer.firstName} ${customer.lastName}")
    assertThat(vm.age).isEqualTo(customer.age)
    }

    View Slide

  39. INTEGRATION TESTING
    INTEGRATION
    E2E
    UNIT
    39

    View Slide

  40. 40
    Class 1 Class 2

    View Slide

  41. 41
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders = legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }

    View Slide

  42. 42
    Integration Test: Verifying Values
    @Test
    fun `when both apis are called both the orders are combined and returned`() {
    val legacyApi: LegacyApi = mock()
    val api: Api = mock()
    val legacyOrders = listOf(Order(1), Order(2))
    whenever(legacyApi.getOrders(anyInt())).thenReturn(legacyOrders)
    val newOrders = listOf(Order(8), Order(9))
    whenever(api.getOrders(anyInt())).thenReturn(newOrders)
    val aggregator = OrderAggregator(legacyApi, api)
    val aggregatedOrders = aggregator.ordersFor(42)
    val expected = newOrders.union(legacyOrders).toList()
    assertEquals(expected, aggregatedOrders)
    }

    View Slide

  43. 43
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders: List = listOf() //legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }

    View Slide

  44. 44
    Integration Test: Verifying Behavior
    @Test
    fun `both apis should be called when retrieving orders`() {
    val legacyApi: LegacyApi = mock()
    val api: Api = mock()
    val aggregator = OrderAggregator(legacyApi, api)
    aggregator.ordersFor(42)
    verify(legacyApi).getOrders(42)
    verify(api).getOrders(42)
    }

    View Slide

  45. 45
    class OrderAggregator(private val legacyApi: LegacyApi, private val api: Api) {
    fun ordersFor(id: Int): List {
    val legacyOrders = legacyApi.getOrders(id)
    val newOrders = api.getOrders(id)
    return newOrders.union(legacyOrders).toList()
    }
    }

    View Slide

  46. 46

    View Slide

  47. E2E TESTING (End to End Testing)
    INTEGRATION
    E2E
    UNIT
    47

    View Slide

  48. 48

    View Slide

  49. Business Logic/etc
    TEST THE WHOLE STACK (END TO END)
    Remote APIs/Services
    UI
    49

    View Slide

  50. Basic Espresso Test
    50
    @Test
    fun verifyThatTheUserCanLogIn(){
    onView(withId(R.id.email)).perform(typeText("[email protected]"))
    onView(withId(R.id.password)).perform(typeText("android123"))
    onView(withId(R.id.login_button)).perform(click())
    onView(withId(R.id.login_message)).check(matches(withText("Success!")))
    }

    View Slide

  51. 70% Unit Tests, 20% Integration Tests, 10%
    End-to-End Tests
    70/20/10
    51
    Google Testing Pyramid Advice

    View Slide

  52. 70% Unit Tests, 20% Integration Tests, 10%
    End-to-End Tests
    70/20/10
    52
    Google Testing Pyramid Advice

    View Slide

  53. YOUR USERS DON’T CARE
    ABOUT UNIT TESTS
    THEY CARE IF THE APP
    WORKS OR NOT.
    53

    View Slide

  54. 54

    View Slide

  55. 55
    Declaring Package Bankruptcy
    (When your app has so many 1-star reviews that
    you cannot recover from it so you have to
    abandon the package name and upload a new
    one.)

    View Slide

  56. 60% End-to-End Tests
    40% Unit Tests/Integration
    60:40
    56

    View Slide

  57. END-TO-END TESTS
    ARE THE MOST VALUABLE
    TESTS YOU HAVE
    57

    View Slide

  58. END-TO-END TESTS
    EMULATE REAL USERS
    58

    View Slide

  59. MOST END-TO-END
    TEST SUITES
    59

    View Slide

  60. Business Logic/etc
    TEST THE WHOLE STACK (END TO END)
    Remote APIs/Services
    UI
    60



    Bluetooth
    Camera APIs
    Remote APIs
    Networking
    etc

    View Slide

  61. Your tests should have reproducible results.
    HERMETIC
    61

    View Slide

  62. 62

    View Slide

  63. HOW TO MAKE A TEST HERMETIC
    ▪ Remove the variance & external influence
    ▫ Remove the network
    ▫ Remove shared files
    ▪ Sanitize the Testing Environment
    ▪ Create Repeatable Environments
    63

    View Slide

  64. DECOUPLE VIA
    ARCHITECTURE
    64

    View Slide

  65. 65
    DEPENDENCY INJECTION
    IS KEY

    View Slide

  66. Remote APIS
    Bluetooth APIS
    Camera APIS
    Contact Picker
    Content Resolvers
    COMMON DIFFICULT AREAS OF TESTING
    66

    View Slide

  67. Bluetooth APIS → BlueTooth Delegate
    Camera APIS → Camera Delegate
    Contact Picker → Contact Picker Delegate
    Content Resolver → … yup ... a delegate
    DELEGATE PATTERN
    67

    View Slide

  68. 68
    public void onActivityResult(int reqCode, int resultCode, Intent data) {
    super.onActivityResult(reqCode, resultCode, data);
    switch (reqCode) {
    case (PICK_CONTACT):
    if (resultCode == Activity.RESULT_OK) {
    Uri contactData = data.getData();
    Cursor c = managedQuery(contactData, null, null, null, null);
    if (c.moveToFirst()) {
    String id = c.getString(c.getColumnIndexOrThrow(ContactsContract.Contacts._ID));
    String hasPhone = c.getString(c.getColumnIndex(ContactsContract.Contacts.HAS_PHONE_NUMBER));
    try {
    if (hasPhone.equalsIgnoreCase("1")) {
    Cursor phones = getContentResolver().query(
    ContactsContract.CommonDataKinds.Phone.CONTENT_URI, null,
    ContactsContract.CommonDataKinds.Phone.CONTACT_ID + " = " + id,
    null, null);
    phones.moveToFirst();
    String cNumber = phones.getString(phones.getColumnIndex("data1"));
    System.out.println("number is:" + cNumber);
    txtphno.setText("Phone Number is: "+cNumber);
    }
    String name = c.getString(c.getColumnIndex(ContactsContract.Contacts.DISPLAY_NAME));
    txtname.setText("Name is: "+name);
    }
    catch (Exception ex)
    {
    st.getMessage();
    }
    }
    }
    break;
    }
    }

    View Slide

  69. 69
    data class Contact(val name: String, val number: String)
    interface ContactRepository {
    fun getContactFor(uri: Uri): Contact
    }
    class ContentResolverDelegate(context: Context) : ContactRepository {
    override fun getContactFor(uri: Uri): Contact {
    val contactData = uri.data
    val c = context.contentResolver...
    if (c.moveToFirst()) {
    // blah blah blah
    }
    }
    }

    View Slide

  70. 70
    class FakeContactRepository : ContactRepository {
    lateinit var contact: Contact
    fun loadContact(contact: Contact) {
    this.contact = contact
    }
    override fun getContactFor(uri: Uri): Contact {
    return contact
    }
    }

    View Slide

  71. 71
    override protected fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent)
    {
    // inject the contactRepository via DI
    val contact = contactRepository.getContactFor(data.data)
    // do stuff with the contact
    }

    View Slide

  72. 72

    View Slide

  73. 73
    class ContactTest {
    @Inject var repository: FakeContactRepository
    @get:Rule
    var activityRule: ActivityTestRule = ActivityTestRule(MainActivity::class.java)
    @Before
    fun setup() {
    // do injection with your DI of choice so you can get a handle on the fake repository
    }
    @Test
    fun verifyThatUserCanSendMessageToFriendThatWasPickedFromContactPicker() {
    val stubResult = Intent()
    stubResult.data = Uri.EMPTY
    val activityResult = Instrumentation.ActivityResult(RESULT_OK, stubResult)
    Intents.Intending(IntentMatchers.hasAction(Intent.ACTION_PICK)).respondWith(activityResult)
    val expectedContact = Contact("Donn Felker", "888-867-5309")
    repository.loadContact(expectedContact)
    activityRule.launchActivity(intent)
    // Starts contact picker, the intents api kicks in and takes over
    // returning our expected intent
    onView(withId(R.id.choose_friend)).perform(click())
    onView(withId(R.id.friend_header)).check(matches(withText("Send to: Donn Felker")))
    onView(withId(R.id.friend_number)).check(matches(withText("Number: 888-867-5309")))
    }
    }

    View Slide

  74. 74
    Other Option - More work - More E2E
    ▪ Write some test code to manually insert
    (and remove on tear down) contacts into
    the android content resolver.
    ▪ Very difficult, if not impossible, for
    things like bluetooth, wireless, etc
    ▪ Removing more of the “system” does make
    it hermetic, but at a cost.

    View Slide

  75. In E2E Tests Remove/Replace only
    what you absolutely have to.
    If it a component makes a test(s)
    flakey it might be a good
    candidate.
    75

    View Slide

  76. REMOVING THE NETWORK
    76
    Creating repeatable
    network calls

    View Slide

  77. #1 Problem:
    The network is
    not reliable
    77

    View Slide

  78. Tools to Remove the Network
    ▪ MockWebServer
    ▪ WireMock
    ▪ Hoverfly
    ▪ Custom Mocks/Fakes/etc
    78

    View Slide

  79. Mocking the Network
    Services
    API
    UI
    79
    API
    Server(s)
    MockWebServer
    WireMock
    Custom/etc

    View Slide

  80. Things you can do with these tools
    ▪ Return known responses
    ▪ Return HTTP errors
    ▪ Slow down the network
    ▪ Simulate very difficult HTTP Scenarios
    ▪ Get a reliable/measurable speed in your
    E2E tests that is not relate to network
    ▪ Make your API calls hermetic in test
    80

    View Slide

  81. WireMock: https://github.com/handstandsam/AndroidHttpMockingExamples
    MockWebServer: https://caster.io/courses/mockwebserver
    Resources
    81

    View Slide

  82. E2E HEURISTICS
    82
    Get the most out of Espresso by using these tools, tips
    and frameworks to speed up and stabilize your tests.

    View Slide

  83. 83
    MOCK OUT
    DIFFICULT
    TESTING AREAS

    View Slide

  84. USE A CI SERVER
    84

    View Slide

  85. USE ANDROID TEST ORCHESTRATOR
    85
    Minimal shared state. Each test
    runs in its own Instrumentation
    instance. Therefore, if your tests
    share app state, most of that
    shared state is removed from your
    device's CPU or memory after each
    test.
    Crashes are isolated. Even if one
    test crashes, it takes down only
    its own instance of
    Instrumentation, so the other tests
    in your suite still run.
    https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator

    View Slide

  86. SHARD YOUR TESTS
    Parallelize your tests.
    Specify how many shards and
    how many tests to run.
    Run on various devices
    86
    https://source.android.com/devices/tech/test_infra/tradefed/architecture/advanced/sharding

    View Slide

  87. 87
    ANDROID_SERIAL=emulator-5554 ./gradlew connectedAndroidTest
    -Pandroid.testInstrumentationRunnerArguments.numShards=3
    -Pandroid.testInstrumentationRunnerArguments.shardIndex=0
    // Runs first half of the tests
    adb -s DEVICE_1_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 0 > device_1_results
    // Runs second half of the tests
    adb -s DEVICE_2_SERIAL shell am instrument -w -e numShards 2 -e shardIndex 1 > device_2_results
    VIA ADB
    VIA GRADLE

    View Slide

  88. USE FLANK (ONLY IF YOU USE Firebase Test Lab)
    88
    Massively parallel Android and iOS
    test runner for Firebase Test Lab
    Built in Sharding
    Built in retry for flakey tests
    Set max run time per shard
    FTL Does cost $$$
    https://developer.android.com/training/testing/junit-runner#using-android-test-orchestrator

    View Slide

  89. FINDING SUCCESS
    Applying the pareto principle to testing.
    89

    View Slide

  90. 90
    My app doesn’t have any tests.
    Where do I start?

    View Slide

  91. 91
    Start with
    End-to-End Tests

    View Slide

  92. Each test is an island of safety
    in an ocean of bugs.
    92

    View Slide

  93. 93
    WHEN DO I WRITE UNIT &
    INTEGRATION TESTS?

    View Slide

  94. 94
    When you’re Test
    Driving a Feature.
    Why? - It forces good design.

    View Slide

  95. Where does TDD fit
    into this?
    95

    View Slide

  96. 96
    SOMETIMES YOU DON’T KNOW
    WHAT YOU DON’T KNOW.
    Do it when you can, if you can’t … write tests
    afterwards that cover your changes.

    View Slide

  97. 60% End-to-End Tests
    40% Unit Tests/Integration
    60:40
    97
    GUIDING TARGET (but NOT required)

    View Slide

  98. 98
    WHERE TO GO FROM HERE...

    View Slide

  99. 99
    ADOPT AN E2E MINDSET.
    E2E TESTS MIMIC REAL USAGE.
    REAL USAGE = REAL USERS

    View Slide

  100. 100
    REMEMBER
    USERS DON’T CARE THAT YOUR UNIT
    TESTS PASS.
    THEY CARE THAT YOUR APP WORKS.
    SO ... WRITE SOME E2E TESTS.

    View Slide

  101. 101

    View Slide

  102. Thank You
    You can find me at:
    @donnfelker (Twitter and IG)
    donnfelker.com
    102

    View Slide