Pro Yearly is on sale from $80 to $50! »

Tests that get under the skin

Tests that get under the skin

When writing tests for your Android app its easy to end up with tests that are brittle, make changing your app harder, and require a lot of maintenance.
This is a tale of our mistakes, our learnings and how we have devised our testing strategy to allow us to write more meaningful, fast, flexible tests, that are easy to read and change and don't respond to structural changes.
While the talk touches on the tooling we use, it is predominantly about the strategy and how we utilise unidirectional data flow and viewmodels to write subcutaneous tests for the majority of our system on one hand and fast running UI tests on the other.

74b04af30c304a823093ec2cc519dc21?s=128

Florian Sprenger

November 09, 2019
Tweet

Transcript

  1. Florian Sprenger Tests that get under the skin

  2. Tests that get under the skin Outline • What do

    we expect from “good” tests • The traditional approach and how we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
  3. Tests that get under the skin Outline • What do

    we expect from “good” tests • The traditional approach and how we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
  4. Tests that get under the skin Outline • What do

    we expect from “good” tests • The traditional approach how where we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
  5. Tests that get under the skin What Are “Good” Tests?

    https://www.flickr.com/people/8512982@N05 https://creativecommons.org/licenses/by-sa/2.0/deed.en
  6. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting.
  7. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably.
  8. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability.
  9. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes.
  10. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes.
  11. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write.
  12. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read.
  13. Tests that get under the skin “Good” Tests • Minimise

    programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change.
  14. Tests that get under the skin • Xero makes cloud

    accounting software Some Context
  15. Tests that get under the skin • Xero makes cloud

    accounting software • Android App in development since 2013 Some Context
  16. Tests that get under the skin • Xero makes cloud

    accounting software • Android App in development since 2013 • 15 developers in 3 pods (2 in Wellington, 1 in Melbourne) Some Context
  17. Tests that get under the skin • Xero makes cloud

    accounting software • Android App in development since 2013 • ~15 developers in 3 pods (2 in Wellington, 1 in Melbourne) • Developers responsible for quality Some Context
  18. Tests that get under the skin • Xero makes cloud

    accounting software • Android App in development since 2013 • 15 developers in 3 pods (2 in Wellington, 1 in Melbourne) • Developers responsible for quality • Business rules matter Some Context
  19. Tests that get under the skin Traditional Approach • Unit

    tests Test your units without dependencies • Integration tests Test interaction between parts of system • UI tests Test the system using the UI Testing pyramid https://pixabay.com/users/BookBabe-55100
  20. Tests that get under the skin Unit Tests Test your

    units without dependencies In a layered architecture units are typically interpreted as the implementation of the layer: • InvoiceRepositoryTest • InvoicePresenterTest But what about dependencies?
  21. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default Image by Clker-Free-Vector-Images from Pixabay
  22. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values Image by Clker-Free-Vector-Images from Pixabay
  23. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes Image by Clker-Free-Vector-Images from Pixabay
  24. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Image by Clker-Free-Vector-Images from Pixabay
  25. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about Image by Clker-Free-Vector-Images from Pixabay
  26. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes Image by Clker-Free-Vector-Images from Pixabay
  27. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes • Unpredictable Image by Clker-Free-Vector-Images from Pixabay
  28. Undesirable properties Unit Tests Mocking • Everything is null /

    returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes • Unpredictable • Expensive to change Image by Clker-Free-Vector-Images from Pixabay
  29. Tests that get under the skin Integration tests Test interaction

    between parts of system Integration tests don’t suffer from the dependency problem as much.
  30. Tests that get under the skin Integration tests Test interaction

    between parts of system Integration tests don’t suffer from the dependency problem as much. Possibly hit the backend.
  31. Tests that get under the skin Integration tests Test interaction

    between parts of system Integration tests don’t suffer from the dependency problem as much. Possibly hit the backend. But there are some problems at the UI edge if the data flow is not unidirectional. For us for example: Presenter calls into View, View calls Presenter back
  32. Undesirable properties Integration Tests Dataflow at edges • Suffer from

    bi-directional data flow especially when participants at edges are mocked
  33. Undesirable properties Integration Tests Dataflow at edges • Suffer from

    bi-directional data flow especially when participants at edges are mocked Effects • Expensive to change
  34. Undesirable properties Integration Tests Dataflow at edges • Suffer from

    bi-directional data flow especially when participants at edges are mocked • Rely on external system when testing against an API Effects • Expensive to change
  35. Undesirable properties Integration Tests Dataflow at edges • Suffer from

    bi-directional data flow especially when participants at edges are mocked • Rely on external system when testing against an API Effects • Expensive to change • Unreliable • Unpredictable
  36. Tests that get under the skin UI Tests Test the

    system using the UI Can be really slow to run CC0 1.0
  37. Tests that get under the skin UI Tests Test the

    system using the UI Can be really slow to run Hard to set up test cases to test permutations CC0 1.0
  38. Tests that get under the skin UI Tests Test the

    system using the UI Can be really slow to run Hard to set up test cases to test permutations Need a backend system offering repeatability when testing the full stack CC0 1.0
  39. Tests that get under the skin Undesirable properties UI Tests

    • Slow
  40. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable
  41. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write
  42. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write • Hard to change
  43. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh
  44. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting
  45. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting • Expensive to write
  46. Tests that get under the skin Undesirable properties UI Tests

    • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting • Expensive to write • Expensive to change
  47. Tests that get under the skin What seems to be

    the root of our problems • Mocking Conclusion
  48. Tests that get under the skin What seems to be

    the root of our problems • Mocking Conclusion Potential Solutions • Mock as little as possible
  49. Tests that get under the skin What seems to be

    the root of our problems • Mocking • Bi-directional data flow Conclusion Potential Solutions • Mock as little as possible
  50. Tests that get under the skin What seems to be

    the root of our problems • Mocking • Bi-directional data flow Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels
  51. Tests that get under the skin What seems to be

    the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels
  52. Tests that get under the skin What seems to be

    the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels • Invest into infrastructure that makes test setup easy and parameterisable
  53. Tests that get under the skin What seems to be

    the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup • UI tests have broad scope and are slow Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels • Invest into infrastructure that makes test setup easy and parameterisable
  54. Tests that get under the skin Back To The Drawing

    Board Thamizhpparithi Maari / https://creativecommons.org/licenses/by-sa/3.0/deed.en
  55. Tests that get under the skin Subcutaneous Testing A definition

    “Subcutaneous tests are important because we want to be able to test business logic with the entirety of the system in play, with the exception of external connection points such as the user interface and external services...” Jimmy Bogard, https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/
  56. Tests that get under the skin Subcutaneous Testing A definition

    “...While a unit test focuses on small-scale design, a subcutaneous test does not address design, but instead tests basic inputs and outputs of the system as a whole.” Jimmy Bogard, https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/
  57. Tests that get under the skin Subcutaneous Testing A definition

    unpacked Excludes external connection points such as the user interface and external services. • We don’t include our UI in the tests • We don’t interact with backend services
  58. Tests that get under the skin Subcutaneous Testing A definition

    unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input
  59. Tests that get under the skin Subcutaneous Testing A definition

    unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output
  60. Tests that get under the skin Subcutaneous Testing A definition

    unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output • Calls of an API is an output
  61. Tests that get under the skin Subcutaneous Testing A definition

    unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output • Calls of an API is an output • Response of an API is an input
  62. Tests that get under the skin Subcutaneous Testing On paper

  63. Tests that get under the skin Subcutaneous Testing In code

    @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { } Bootstrapping a test environment
  64. Tests that get under the skin Subcutaneous Testing In code

    @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) } Bootstrapping a test environment
  65. Tests that get under the skin Subcutaneous Testing In code

    @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) // configure authentication state of user (session id, user id etc...) setUserLoggedIn() } Bootstrapping a test environment
  66. Tests that get under the skin Subcutaneous Testing In code

    @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) // configure authentication state of user (session id, user id etc...) setUserLoggedIn() // set up network response when organisation is loaded setCurrentOrg( // use sensible defaults organisation = createOrganisationResponse( // but allow parameterisation name = "GDG Melbourne" ) ) } Bootstrapping a test environment
  67. Tests that get under the skin Subcutaneous Testing In code

    @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) } Using test environment in a test
  68. Tests that get under the skin Subcutaneous Testing In code

    @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) } Using test environment in a test
  69. Tests that get under the skin Subcutaneous Testing In code

    @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) // request a state change from viewModel <- input from user vm.onDescriptionEdited(0, "Frankfurters") } Using test environment in a test
  70. Tests that get under the skin Subcutaneous Testing In code

    @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) // request a state change from viewModel <- input from user vm.onDescriptionEdited(0, "Frankfurters") // assert state of viewModel assertEquals(null, vm.state.lineItems[0].lineItem.suggestedAccountId) // assert network call has been made environment.verifyApiRequest(times(1)).getAccountSuggestion( AccountSuggestionRequest( description = "Frankfurters" ) ) } Using test environment in a test
  71. Tests that get under the skin Subcutaneous Testing • Testing

    our whole business logic • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
  72. Tests that get under the skin Subcutaneous Testing • Testing

    our whole business logic • We don’t care about implementation details* • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
  73. Tests that get under the skin Subcutaneous Testing • Testing

    our whole business logic • We don’t care about implementation details* • Investment into reusable, parameterisable infrastructure (as code) • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
  74. Tests that get under the skin Subcutaneous Testing • Testing

    our whole business logic • We don’t care about implementation details* • Investment into reusable, parameterisable infrastructure (as code) • Exclude API and UI • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
  75. Tests that get under the skin Testing The Skin Having

    our cake and eating it too
  76. Tests that get under the skin Testing UI Same principles

    apply Since we drive UI from ViewModel’s state just need to verify input and output: Input • Verify that state is drawn correctly Output • Verify that Interaction triggers an action on ViewModel
  77. Tests that get under the skin Testing UI

  78. Tests that get under the skin Testing UI In code

    @Rule @JvmField val testRule = BaseActivityTestRule(AddPaymentActivity::class.java, true, false) @Before fun beforeEachTestCase() { robot = AddPaymentRobot(testRule, testName) } Testing inputs
  79. Tests that get under the skin Testing UI In code

    @Test fun whenPaymentAmountHasValue50_thenAmountIsDisplayedInView() { val paymentDisplayAmount = BigDecimal("50.00") // set ViewModel state robot.act { setPaymentAmount(paymentDisplayAmount) } } Testing inputs
  80. Tests that get under the skin Testing UI In code

    @Test fun whenPaymentAmountHasValue50_thenAmountIsDisplayedInView() { val paymentDisplayAmount = BigDecimal("50.00") // set ViewModel state robot.act { setPaymentAmount(paymentDisplayAmount) } // assert on UI that correct amount is displayed robot.assert { paymentAmountMatches(paymentDisplayAmount.toString()) } } Testing inputs
  81. Tests that get under the skin Testing UI In code

    @Test fun whenPullToRefresh_thenListIsRefreshed() { robot.act { pullToRefresh() } robot.assert { refreshIsCalled() } } Testing outputs
  82. Tests that get under the skin Testing UI In code

    @Test fun whenPullToRefresh_thenListIsRefreshed() { robot.act { pullToRefresh() } robot.assert { refreshIsCalled() } } fun pullToRefresh() { Espresso.onView(ViewMatchers.withId(R.id.swipeContainer)) .perform(ViewActions.swipeDown()) } fun refreshIsCalled() { verify(viewModel, times(1)).refresh() } Testing outputs
  83. Tests that get under the skin Testing UI • Testing

    UI in isolation • Testing display state separate from testing interactions • Minimize programmer waiting.* • Run reliably.* • Predict deployability.* • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write.* • Be cheap to read. • Be cheap to change. Validating our approach
  84. Tests that get under the skin Summary 1. Set up

    a definition of “Good” tests
  85. Tests that get under the skin Summary 1. Set up

    a definition of “Good” tests 2. Looked at the traditional approach and where it does not satisfy our definition
  86. Tests that get under the skin Summary 1. Set up

    a definition of “Good” tests 2. Looked at the traditional approach and where it does not satisfy our definition 3. Explored a different approach using subcutaneous testing and how to apply the same principles to UI testing
  87. Tests that get under the skin That’s All Folks @flowsprenger

    End of Story by Nick Youngson CC BY-SA 3.0 ImageCreator https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934 https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/