Tests that get under the skin

When writing tests for your Android app its easy to end up with tests that are brittle, make changing your app harder, and require a lot of maintenance.
This is a tale of our mistakes, our learnings and how we have devised our testing strategy to allow us to write more meaningful, fast, flexible tests, that are easy to read and change and don't respond to structural changes.
While the talk touches on the tooling we use, it is predominantly about the strategy and how we utilise unidirectional data flow and viewmodels to write subcutaneous tests for the majority of our system on one hand and fast running UI tests on the other.

Transcript

1. Florian Sprenger Tests that get under the skin
2. Tests that get under the skin Outline • What do we expect from “good” tests • The traditional approach and how we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
3. Tests that get under the skin Outline • What do we expect from “good” tests • The traditional approach and how we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
4. Tests that get under the skin Outline • What do we expect from “good” tests • The traditional approach how where we failed to write “good” tests • A different approach: Subcutaneous testing and fast running UI tests
5. Tests that get under the skin What Are “Good” Tests? https://www.flickr.com/people/8512982@N05 https://creativecommons.org/licenses/by-sa/2.0/deed.en
6. Tests that get under the skin “Good” Tests • Minimise programmer waiting.
7. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably.
8. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability.
9. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes.
10. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes.
11. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write.
12. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read.
13. Tests that get under the skin “Good” Tests • Minimise programmer waiting. • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change.
14. Tests that get under the skin • Xero makes cloud accounting software Some Context
15. Tests that get under the skin • Xero makes cloud accounting software • Android App in development since 2013 Some Context
16. Tests that get under the skin • Xero makes cloud accounting software • Android App in development since 2013 • 15 developers in 3 pods (2 in Wellington, 1 in Melbourne) Some Context
17. Tests that get under the skin • Xero makes cloud accounting software • Android App in development since 2013 • ~15 developers in 3 pods (2 in Wellington, 1 in Melbourne) • Developers responsible for quality Some Context
18. Tests that get under the skin • Xero makes cloud accounting software • Android App in development since 2013 • 15 developers in 3 pods (2 in Wellington, 1 in Melbourne) • Developers responsible for quality • Business rules matter Some Context
19. Tests that get under the skin Traditional Approach • Unit tests Test your units without dependencies • Integration tests Test interaction between parts of system • UI tests Test the system using the UI Testing pyramid https://pixabay.com/users/BookBabe-55100
20. Tests that get under the skin Unit Tests Test your units without dependencies In a layered architecture units are typically interpreted as the implementation of the layer: • InvoiceRepositoryTest • InvoicePresenterTest But what about dependencies?
21. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default Image by Clker-Free-Vector-Images from Pixabay
22. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values Image by Clker-Free-Vector-Images from Pixabay
23. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes Image by Clker-Free-Vector-Images from Pixabay
24. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Image by Clker-Free-Vector-Images from Pixabay
25. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about Image by Clker-Free-Vector-Images from Pixabay
26. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes Image by Clker-Free-Vector-Images from Pixabay
27. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes • Unpredictable Image by Clker-Free-Vector-Images from Pixabay
28. Undesirable properties Unit Tests Mocking • Everything is null / returns null by default • Doesn’t enforce meaningful values • Ignorant to changes • Undefined behaviour of dependencies Effects • Hard to reason about • Respond to structure changes • Unpredictable • Expensive to change Image by Clker-Free-Vector-Images from Pixabay
29. Tests that get under the skin Integration tests Test interaction between parts of system Integration tests don’t suffer from the dependency problem as much.
30. Tests that get under the skin Integration tests Test interaction between parts of system Integration tests don’t suffer from the dependency problem as much. Possibly hit the backend.
31. Tests that get under the skin Integration tests Test interaction between parts of system Integration tests don’t suffer from the dependency problem as much. Possibly hit the backend. But there are some problems at the UI edge if the data flow is not unidirectional. For us for example: Presenter calls into View, View calls Presenter back
32. Undesirable properties Integration Tests Dataflow at edges • Suffer from bi-directional data flow especially when participants at edges are mocked
33. Undesirable properties Integration Tests Dataflow at edges • Suffer from bi-directional data flow especially when participants at edges are mocked Effects • Expensive to change
34. Undesirable properties Integration Tests Dataflow at edges • Suffer from bi-directional data flow especially when participants at edges are mocked • Rely on external system when testing against an API Effects • Expensive to change
35. Undesirable properties Integration Tests Dataflow at edges • Suffer from bi-directional data flow especially when participants at edges are mocked • Rely on external system when testing against an API Effects • Expensive to change • Unreliable • Unpredictable
36. Tests that get under the skin UI Tests Test the system using the UI Can be really slow to run CC0 1.0
37. Tests that get under the skin UI Tests Test the system using the UI Can be really slow to run Hard to set up test cases to test permutations CC0 1.0
38. Tests that get under the skin UI Tests Test the system using the UI Can be really slow to run Hard to set up test cases to test permutations Need a backend system offering repeatability when testing the full stack CC0 1.0
39. Tests that get under the skin Undesirable properties UI Tests • Slow
40. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable
41. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write
42. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write • Hard to change
43. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh
44. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting
45. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting • Expensive to write
46. Tests that get under the skin Undesirable properties UI Tests • Slow • Unreliable • Tedious to write • Hard to change • Tooling is meh Effects • Maximise programmer waiting • Expensive to write • Expensive to change
47. Tests that get under the skin What seems to be the root of our problems • Mocking Conclusion
48. Tests that get under the skin What seems to be the root of our problems • Mocking Conclusion Potential Solutions • Mock as little as possible
49. Tests that get under the skin What seems to be the root of our problems • Mocking • Bi-directional data flow Conclusion Potential Solutions • Mock as little as possible
50. Tests that get under the skin What seems to be the root of our problems • Mocking • Bi-directional data flow Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels
51. Tests that get under the skin What seems to be the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels
52. Tests that get under the skin What seems to be the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels • Invest into infrastructure that makes test setup easy and parameterisable
53. Tests that get under the skin What seems to be the root of our problems • Mocking • Bi-directional data flow • Tests rely on intricate setup • UI tests have broad scope and are slow Conclusion Potential Solutions • Mock as little as possible • Unidirectional flow using ViewModels • Invest into infrastructure that makes test setup easy and parameterisable
54. Tests that get under the skin Back To The Drawing Board Thamizhpparithi Maari / https://creativecommons.org/licenses/by-sa/3.0/deed.en
55. Tests that get under the skin Subcutaneous Testing A definition “Subcutaneous tests are important because we want to be able to test business logic with the entirety of the system in play, with the exception of external connection points such as the user interface and external services...” Jimmy Bogard, https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/
56. Tests that get under the skin Subcutaneous Testing A definition “...While a unit test focuses on small-scale design, a subcutaneous test does not address design, but instead tests basic inputs and outputs of the system as a whole.” Jimmy Bogard, https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/
57. Tests that get under the skin Subcutaneous Testing A definition unpacked Excludes external connection points such as the user interface and external services. • We don’t include our UI in the tests • We don’t interact with backend services
58. Tests that get under the skin Subcutaneous Testing A definition unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input
59. Tests that get under the skin Subcutaneous Testing A definition unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output
60. Tests that get under the skin Subcutaneous Testing A definition unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output • Calls of an API is an output
61. Tests that get under the skin Subcutaneous Testing A definition unpacked Instead tests basic inputs and outputs (of those external connection points) • User interaction is an input • ViewModel state is an output • Calls of an API is an output • Response of an API is an input
62. Tests that get under the skin Subcutaneous Testing On paper
63. Tests that get under the skin Subcutaneous Testing In code @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { } Bootstrapping a test environment
64. Tests that get under the skin Subcutaneous Testing In code @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) } Bootstrapping a test environment
65. Tests that get under the skin Subcutaneous Testing In code @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) // configure authentication state of user (session id, user id etc...) setUserLoggedIn() } Bootstrapping a test environment
66. Tests that get under the skin Subcutaneous Testing In code @Rule @JvmField // sets up a dagger graph similar to the real app through test rule val environment = TestEnvironment { // configure a feature to be on setFeatureEnabled(Feature.A_FEATURE_IN_DEVELOPMENT) // configure authentication state of user (session id, user id etc...) setUserLoggedIn() // set up network response when organisation is loaded setCurrentOrg( // use sensible defaults organisation = createOrganisationResponse( // but allow parameterisation name = "GDG Melbourne" ) ) } Bootstrapping a test environment
67. Tests that get under the skin Subcutaneous Testing In code @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) } Using test environment in a test
68. Tests that get under the skin Subcutaneous Testing In code @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) } Using test environment in a test
69. Tests that get under the skin Subcutaneous Testing In code @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) // request a state change from viewModel <- input from user vm.onDescriptionEdited(0, "Frankfurters") } Using test environment in a test
70. Tests that get under the skin Subcutaneous Testing In code @Test fun onDescriptionEdit_whenSuggestionCallReturnsEmpty_dontSetSuggestedId() { // bootstrap the viewModel under test with specific config // this will load the organisation doing networking calls, some test will populate db val vm = environment.createViewModel<InvoiceLineItemViewModel>( createInvoiceLineItemsViewConfig( lineItems = listOf(createInvoiceLineItem()) ) ) // set up network response <- input from api environment.setAccountSuggestions(createEmptyAccountSuggestionResponse()) // request a state change from viewModel <- input from user vm.onDescriptionEdited(0, "Frankfurters") // assert state of viewModel assertEquals(null, vm.state.lineItems[0].lineItem.suggestedAccountId) // assert network call has been made environment.verifyApiRequest(times(1)).getAccountSuggestion( AccountSuggestionRequest( description = "Frankfurters" ) ) } Using test environment in a test
71. Tests that get under the skin Subcutaneous Testing • Testing our whole business logic • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
72. Tests that get under the skin Subcutaneous Testing • Testing our whole business logic • We don’t care about implementation details* • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
73. Tests that get under the skin Subcutaneous Testing • Testing our whole business logic • We don’t care about implementation details* • Investment into reusable, parameterisable infrastructure (as code) • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
74. Tests that get under the skin Subcutaneous Testing • Testing our whole business logic • We don’t care about implementation details* • Investment into reusable, parameterisable infrastructure (as code) • Exclude API and UI • Minimize programmer waiting.* • Run reliably. • Predict deployability. • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write. • Be cheap to read. • Be cheap to change. Validating our approach
75. Tests that get under the skin Testing The Skin Having our cake and eating it too
76. Tests that get under the skin Testing UI Same principles apply Since we drive UI from ViewModel’s state just need to verify input and output: Input • Verify that state is drawn correctly Output • Verify that Interaction triggers an action on ViewModel
77. Tests that get under the skin Testing UI
78. Tests that get under the skin Testing UI In code @Rule @JvmField val testRule = BaseActivityTestRule(AddPaymentActivity::class.java, true, false) @Before fun beforeEachTestCase() { robot = AddPaymentRobot(testRule, testName) } Testing inputs
79. Tests that get under the skin Testing UI In code @Test fun whenPaymentAmountHasValue50_thenAmountIsDisplayedInView() { val paymentDisplayAmount = BigDecimal("50.00") // set ViewModel state robot.act { setPaymentAmount(paymentDisplayAmount) } } Testing inputs
80. Tests that get under the skin Testing UI In code @Test fun whenPaymentAmountHasValue50_thenAmountIsDisplayedInView() { val paymentDisplayAmount = BigDecimal("50.00") // set ViewModel state robot.act { setPaymentAmount(paymentDisplayAmount) } // assert on UI that correct amount is displayed robot.assert { paymentAmountMatches(paymentDisplayAmount.toString()) } } Testing inputs
81. Tests that get under the skin Testing UI In code @Test fun whenPullToRefresh_thenListIsRefreshed() { robot.act { pullToRefresh() } robot.assert { refreshIsCalled() } } Testing outputs
82. Tests that get under the skin Testing UI In code @Test fun whenPullToRefresh_thenListIsRefreshed() { robot.act { pullToRefresh() } robot.assert { refreshIsCalled() } } fun pullToRefresh() { Espresso.onView(ViewMatchers.withId(R.id.swipeContainer)) .perform(ViewActions.swipeDown()) } fun refreshIsCalled() { verify(viewModel, times(1)).refresh() } Testing outputs
83. Tests that get under the skin Testing UI • Testing UI in isolation • Testing display state separate from testing interactions • Minimize programmer waiting.* • Run reliably.* • Predict deployability.* • Respond to behaviour changes. • Not respond to structure changes. • Be cheap to write.* • Be cheap to read. • Be cheap to change. Validating our approach
84. Tests that get under the skin Summary 1. Set up a definition of “Good” tests
85. Tests that get under the skin Summary 1. Set up a definition of “Good” tests 2. Looked at the traditional approach and where it does not satisfy our definition
86. Tests that get under the skin Summary 1. Set up a definition of “Good” tests 2. Looked at the traditional approach and where it does not satisfy our definition 3. Explored a different approach using subcutaneous testing and how to apply the same principles to UI testing
87. Tests that get under the skin That’s All Folks @flowsprenger End of Story by Nick Youngson CC BY-SA 3.0 ImageCreator https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934 https://lostechies.com/jimmybogard/2010/08/25/an-effective-testing-strategy/