Introducing UI Testing to Your Team

Transcript

1. Introducing UI Testing to Your Team Mustafa Mutlu
2. @bekobird Android Engineer at N26 Mustafa MoneyBeam, FPS and BACS Transactions for the UK
3. 2014
4. 2014 Number of Android devs 5
5. 2016 Number of Android devs 5
6. 2016 Number of Android devs 5
7. 2017 5 Number of Android devs
8. 5 2018 15 Number of Android devs
9. 32 15 Now Number of Android devs 2019 5
10. E2E Tests Integration Tests Unit Tests Testing Strategy
11. E2E Tests Integration Tests Unit Tests Testing Strategy
12. E2E Tests Integration Tests Unit Tests Testing Strategy
13. E2E Tests Integration Tests Unit Tests Testing Strategy
14. E2E Tests Integration Tests Unit Tests Testing Strategy Slower Faster More Integration More Isolation
15. E2E Tests Integration Tests Testing Strategy
16. A Typical Test Case • ID and name of the test • Preconditions • Steps with expected result
17. A Typical Test Case • Step: When I enter MoneyBeam and select a contact • Expected: Then I see the money amount entrance screen and the screen has these: • Top left: origin account's name and balance • Top right: destination account's name • Center: money input • Center: description text • Bottom: Continue button that is not in enabled/clickable state
18. • 3200+ test cases • Testing 500 to 850 cases, in each release • Releasing every once in 3 weeks • This is a lot of work!
19. • Automated end-to-end UI tests • Both for Android and iOS • Black-box testing • Uses Appium Photo by ActionVance on Unsplash Batman Project
20. • Slow test execution • Not included in our CI pipeline • Difficult to maintain • Flakiness caused by E2E Batman Project
21. Espresso • Focused on testing the UI (not E2E) • Maintained by the same feature developers
22. Robot Pattern https://jakewharton.com/testing-robots/ Recommended talk by Jake Wharton
23. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
24. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
25. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
26. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
27. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
28. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
29. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(), withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
30. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(), withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
31. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(), withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
32. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(), withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
33. Robot Pattern View ViewModel Domain Data What How How N26 Android App
34. Robot Pattern View ViewModel Domain Data What How How N26 Android App Test Test What & How
35. Robot Pattern View ViewModel Domain Data What How How N26 Android App Robot How Test What
36. Robot Pattern View ViewModel Domain Data What How How N26 Android App Robot Test How What
37. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(), withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
38. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface", "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
39. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface", "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
40. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface", "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
41. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface", "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
42. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
43. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
44. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
45. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
46. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
47. Challenges
48. Animations "To avoid flakiness, we highly recommend that you turn off system animations on the virtual or physical devices used for testing." - Espresso docs
49. Animations • System animations • Dynamic Animations from AndroidX Our solution: add a switch to turn off custom animations
50. Dealing Asynchronicity • Tests need to know when to wait and proceed • Causing flakiness Our solution: make use of Idling Resources • OkHttpIdlingResource • RxIdler
51. Mocking Data Sources
52. Mocking Data Sources Domain Data
53. Mocking Data Sources Domain Data Repository Remote Data Source
54. Mocking Data Sources Domain Data Repository Remote Data Source Framework Data Source
55. Mocking Data Sources Domain Data Repository Remote Data Source Fake Data Source
56. Dealing with Localization English (US) English (Canada) • Same account (United States) • Same currency (USD) • Same language (English) • Different device locale • Result: different money representation
57. Dealing with Localization • Using device locale for formatting monetary amounts • Test device's locale is unpredictable • Causing flaky tests Our solution: add a test rule to define the device locale
58. Dealing With Flakiness • Identify the cause and update the test • Add retry for failed tests (FlakyTestRule from the Barista library)
59. Integration with CI Pipeline
60. Integration with CI Pipeline 3 min
61. Integration with CI Pipeline 5 min
62. Integration with CI Pipeline 10 min
63. Integration with CI Pipeline 20 min
64. Integration with CI Pipeline Then what?
65. Selectively Running the Tests • Find changed files from the Git history • Find which features are changed • Run the tests of the changed features • If there is a change in a common module, trigger the full run • See: goo.gle/androidx-dependency-tracker
66. 20+ minutes UI Testing Pipeline
67. 3 min UI Testing Pipeline
68. android-project git:(my-branch) ./run-on-remote.sh -h | Running on Remote usage: -s | --skip-build Skip building tasks -n name | --test-name name Specify the name of the test. Default is branch name -d device | --device device Specify the name of the device on which the tests will be run. Default is random -p packages | --packages packages The list of packages to run the tests (comma separated). Default is all -l | --log Log the output to log.txt -h | --help Display help
69. Running on Remote Test Name Device Name Execution Time Duration my-branch-name my-branch-name my-branch-name
70. Reporting the Test Results • Link Espresso tests with test case IDs • Add a test rule to watch the test results • Post the test results to test management tool
71. Sample Test Run MoneyBeam Spaces Premium
72. Unit Tests ~500 UI Tests Functional UI Tests Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 ~9000 Unit tests Today
73. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today Functional UI Tests ~9000 Unit tests
74. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today Functional UI Tests Automated Regression 10% ~9000 Unit tests
75. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today Functional UI Tests Automated Regression 50% ~9000 Unit tests
76. • Faster test execution • Part of our CI pipeline • Maintained together with the production code • ☑ Mostly trusty Espresso Project
77. A Few Ideas
78. Feature 1 Feature N Base App
79. App Feature 1 Base
80. App Feature 1 App Feature 1 Base
81. App Feature 1 App Feature 1 Base
82. Feature 1 App Running Tests on Feature Apps Feature 1
83. Is amount input displayed? ✅ Passed Is amount input displayed? ✅ Passed
84. Snapshot Testing Verifies view attributes: • Position • Color • Size Is amount input displayed? ✅ Passed Is amount input displayed? ✅ Passed
85. Snapshot Testing Verifies view attributes: • Position • Color • Size Is amount input displayed? ✅ Passed Is amount input displayed? ❌ Failed
86. Thank You Questions?