Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introducing UI Testing to Your Team

Introducing UI Testing to Your Team

Da49bf507d07666987c1546e894dee8d?s=128

Mustafa Berkay Mutlu

November 25, 2019
Tweet

Transcript

  1. Introducing UI Testing to Your Team Mustafa Mutlu

  2. @bekobird Android Engineer at N26 Mustafa MoneyBeam, FPS and BACS

    Transactions for the UK
  3. 2014

  4. 2014 Number of Android devs 5

  5. 2016 Number of Android devs 5

  6. 2016 Number of Android devs 5

  7. 2017 5 Number of Android devs

  8. 5 2018 15 Number of Android devs

  9. 32 15 Now Number of Android devs 2019 5

  10. E2E Tests Integration Tests Unit Tests Testing Strategy

  11. E2E Tests Integration Tests Unit Tests Testing Strategy

  12. E2E Tests Integration Tests Unit Tests Testing Strategy

  13. E2E Tests Integration Tests Unit Tests Testing Strategy

  14. E2E Tests Integration Tests Unit Tests Testing Strategy Slower Faster

    More Integration More Isolation
  15. E2E Tests Integration Tests Testing Strategy

  16. A Typical Test Case • ID and name of the

    test • Preconditions • Steps with expected result
  17. A Typical Test Case • Step: When I enter MoneyBeam

    and select a contact • Expected: Then I see the money amount entrance screen and the screen has these: • Top left: origin account's name and balance • Top right: destination account's name • Center: money input • Center: description text • Bottom: Continue button that is not in enabled/clickable state
  18. • 3200+ test cases • Testing 500 to 850 cases,

    in each release • Releasing every once in 3 weeks • This is a lot of work!
  19. • Automated end-to-end UI tests • Both for Android and

    iOS • Black-box testing • Uses Appium Photo by ActionVance on Unsplash Batman Project
  20. • Slow test execution • Not included in our CI

    pipeline • Difficult to maintain • Flakiness caused by E2E Batman Project
  21. Espresso • Focused on testing the UI (not E2E) •

    Maintained by the same feature developers
  22. Robot Pattern https://jakewharton.com/testing-robots/ Recommended talk by Jake Wharton

  23. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  24. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  25. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  26. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  27. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  28. @Test fun moneyBeamSample() { apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify()) val intent = MoneyBeamTransferActivity.newIntent(targetContext,

    "Namey McNameface") setupUser { firstName("John") lastName("Doe") currency("GBP") balance(10.toBigDecimal()) } activityScenarioRule.launchActivity(intent) ...
  29. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(),

    withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
  30. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(),

    withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
  31. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(),

    withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
  32. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(),

    withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
  33. Robot Pattern View ViewModel Domain Data What How How N26

    Android App
  34. Robot Pattern View ViewModel Domain Data What How How N26

    Android App Test Test What & How
  35. Robot Pattern View ViewModel Domain Data What How How N26

    Android App Robot How Test What
  36. Robot Pattern View ViewModel Domain Data What How How N26

    Android App Robot Test How What
  37. ... assertClickableNavigateUpButton() onView(allOf(withParent(withId(originContainer)), withId(nameText))) .check(matches(allOf(isDisplayed(), withText("John Doe")))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))) .check(matches(allOf(isDisplayed(),

    withText("JD")))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))) .check(matches(allOf(isDisplayed(), withText("£10.00")))) assertDisplayed(descriptionText, "Send to Namey") assertNotDisplayed(errorText) isButtonEnabled(continueButton) writeTo(amountEditText, "1100") assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000") assertNotDisplayed(descriptionText) isButtonDisabled(continueButton) }
  38. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface",

    "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
  39. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface",

    "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
  40. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface",

    "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
  41. ... moneyBeamSampleRobot { assertClickableNavigateUpButton() isOriginDisplayed("John Doe", "JD", "£10.00") isDestinationDisplayed("Namey McNameface",

    "NM") isDescriptionDisplayed("Send to Namey") isContinueButtonDisabled() enterAmount("1100") isErrorDisplayed("You can only send MoneyBeams up to £1,000") isContinueButtonDisabled() enterAmount("10") isDescriptionDisplayed("Send to Namey") isContinueButtonEnabled() } }
  42. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount

    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
  43. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount

    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
  44. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount

    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
  45. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount

    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
  46. class MoneyBeamSampleRobot : BaseRobot() { private val amountEditText = R.id.editMoneyBeamTransferAmount

    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription private val errorText = R.id.textMoneyBeamTransferAmountError // other view IDs fun isDescriptionDisplayed(text: String) { assertDisplayed(descriptionText, text) assertNotDisplayed(errorText) } fun isErrorDisplayed(text: String) { ... } fun isOriginDisplayed(name: String, initials: String, balance: String) { onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name)))) onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials)))) onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance)))) } fun isDestinationDisplayed(name: String, initials: String) { ... } fun isContinueButtonEnabled() = isButtonEnabled(continueButton) fun isContinueButtonDisabled() = isButtonDisabled(continueButton) fun enterAmount(amount: String) = writeTo(amountEditText, amount) fun clickOnContinueButton() = clickOn(continueButton) }
  47. Challenges

  48. Animations "To avoid flakiness, we highly recommend that you turn

    off system animations on the virtual or physical devices used for testing." - Espresso docs
  49. Animations • System animations • Dynamic Animations from AndroidX Our

    solution: add a switch to turn off custom animations
  50. Dealing Asynchronicity • Tests need to know when to wait

    and proceed • Causing flakiness Our solution: make use of Idling Resources • OkHttpIdlingResource • RxIdler
  51. Mocking Data Sources

  52. Mocking Data Sources Domain Data

  53. Mocking Data Sources Domain Data Repository Remote Data Source

  54. Mocking Data Sources Domain Data Repository Remote Data Source Framework

    Data Source
  55. Mocking Data Sources Domain Data Repository Remote Data Source Fake

    Data Source
  56. Dealing with Localization English (US) English (Canada) • Same account

    (United States) • Same currency (USD) • Same language (English) • Different device locale • Result: different money representation
  57. Dealing with Localization • Using device locale for formatting monetary

    amounts • Test device's locale is unpredictable • Causing flaky tests Our solution: add a test rule to define the device locale
  58. Dealing With Flakiness • Identify the cause and update the

    test • Add retry for failed tests (FlakyTestRule from the Barista library)
  59. Integration with CI Pipeline

  60. Integration with CI Pipeline 3 min

  61. Integration with CI Pipeline 5 min

  62. Integration with CI Pipeline 10 min

  63. Integration with CI Pipeline 20 min

  64. Integration with CI Pipeline Then what?

  65. Selectively Running the Tests • Find changed files from the

    Git history • Find which features are changed • Run the tests of the changed features • If there is a change in a common module, trigger the full run • See: goo.gle/androidx-dependency-tracker
  66. 20+ minutes UI Testing Pipeline

  67. 3 min UI Testing Pipeline

  68. android-project git:(my-branch) ./run-on-remote.sh -h | Running on Remote usage: -s

    | --skip-build Skip building tasks -n name | --test-name name Specify the name of the test. Default is branch name -d device | --device device Specify the name of the device on which the tests will be run. Default is random -p packages | --packages packages The list of packages to run the tests (comma separated). Default is all -l | --log Log the output to log.txt -h | --help Display help
  69. Running on Remote Test Name Device Name Execution Time Duration

    my-branch-name my-branch-name my-branch-name
  70. Reporting the Test Results • Link Espresso tests with test

    case IDs • Add a test rule to watch the test results • Post the test results to test management tool
  71. Sample Test Run MoneyBeam Spaces Premium

  72. Unit Tests ~500 UI Tests Functional UI Tests Oct 14

    Jan 2016 Jan 2017 Jan 2018 Feb 2019 ~9000 Unit tests Today
  73. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan

    2017 Jan 2018 Feb 2019 Today Functional UI Tests ~9000 Unit tests
  74. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan

    2017 Jan 2018 Feb 2019 Today Functional UI Tests Automated Regression 10% ~9000 Unit tests
  75. Unit Tests ~500 UI Tests Oct 14 Jan 2016 Jan

    2017 Jan 2018 Feb 2019 Today Functional UI Tests Automated Regression 50% ~9000 Unit tests
  76. • Faster test execution • Part of our CI pipeline

    • Maintained together with the production code • ☑ Mostly trusty Espresso Project
  77. A Few Ideas

  78. Feature 1 Feature N Base App

  79. App Feature 1 Base

  80. App Feature 1 App Feature 1 Base

  81. App Feature 1 App Feature 1 Base

  82. Feature 1 App Running Tests on Feature Apps Feature 1

  83. Is amount input displayed? ✅ Passed Is amount input displayed?

    ✅ Passed
  84. Snapshot Testing Verifies view attributes: • Position • Color •

    Size Is amount input displayed? ✅ Passed Is amount input displayed? ✅ Passed
  85. Snapshot Testing Verifies view attributes: • Position • Color •

    Size Is amount input displayed? ✅ Passed Is amount input displayed? ❌ Failed
  86. Thank You Questions?