Introducing UI Testing to Your Team

Transcript

1. Introducing UI Testing
   to Your Team
   Mustafa Mutlu
   

   View Slide

2. @bekobird
   Android Engineer at N26
   Mustafa
   MoneyBeam, FPS and BACS Transactions for the UK
   

   View Slide

3. View Slide

4. 2014
   Number of Android devs
   5
   

   View Slide

5. 2016
   Number of Android devs
   5
   

   View Slide

6. 2016
   Number of Android devs
   5
   

   View Slide

7. 2017
   5
   Number of Android devs
   

   View Slide

8. 5
   2018
   15
   Number of Android devs
   

   View Slide

9. 32
   15
   Now
   Number of Android devs
   2019
   5
   

   View Slide

10. E2E
    Tests
    Integration
    Tests
    Unit Tests
    Testing Strategy
    

    View Slide

11. E2E
    Tests
    Integration
    Tests
    Unit Tests
    Testing Strategy
    

    View Slide

12. E2E
    Tests
    Integration
    Tests
    Unit Tests
    Testing Strategy
    

    View Slide

13. E2E
    Tests
    Integration
    Tests
    Unit Tests
    Testing Strategy
    

    View Slide

14. E2E
    Tests
    Integration
    Tests
    Unit Tests
    Testing Strategy
    Slower
    Faster
    More Integration
    More Isolation
    

    View Slide

15. E2E
    Tests
    Integration
    Tests
    Testing Strategy
    

    View Slide

16. A Typical Test Case
    • ID and name of the test
    • Preconditions
    • Steps with expected result
    

    View Slide

17. A Typical Test Case
    • Step: When I enter MoneyBeam and select a contact
    • Expected:
    Then I see the money amount entrance screen
    and the screen has these:
    • Top left: origin account's name and balance
    • Top right: destination account's name
    • Center: money input
    • Center: description text
    • Bottom: Continue button that is not in enabled/clickable state
    

    View Slide

18. •
    3200+ test cases
    •
    Testing 500 to 850 cases, in each release
    •
    Releasing every once in 3 weeks
    •
    This is a lot of work!
    

    View Slide

19. • Automated end-to-end UI tests
    • Both for Android and iOS
    • Black-box testing
    • Uses Appium
    Photo by ActionVance on Unsplash
    Batman Project
    

    View Slide

20. •
    Slow test execution
    •
    Not included in our CI pipeline
    •
    Difficult to maintain
    •
    Flakiness caused by E2E
    Batman Project
    

    View Slide

21. Espresso
    • Focused on testing the UI (not E2E)
    • Maintained by the same feature developers
    

    View Slide

22. Robot Pattern
    https://jakewharton.com/testing-robots/
    Recommended talk by
    Jake Wharton
    

    View Slide

23. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

24. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

25. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

26. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

27. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

28. @Test
    fun moneyBeamSample() {
    apiServer.registerJsonResponse(MoneyBeamApiRequests.MyApiRequest("a-param"), responseRaw().jsonify())
    val intent = MoneyBeamTransferActivity.newIntent(targetContext, "Namey McNameface")
    setupUser {
    firstName("John")
    lastName("Doe")
    currency("GBP")
    balance(10.toBigDecimal())
    }
    activityScenarioRule.launchActivity(intent)
    ...
    

    View Slide

29. ...
    assertClickableNavigateUpButton()
    onView(allOf(withParent(withId(originContainer)), withId(nameText)))
    .check(matches(allOf(isDisplayed(), withText("John Doe"))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText)))
    .check(matches(allOf(isDisplayed(), withText("JD"))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText)))
    .check(matches(allOf(isDisplayed(), withText("£10.00"))))
    assertDisplayed(descriptionText, "Send to Namey")
    assertNotDisplayed(errorText)
    isButtonEnabled(continueButton)
    writeTo(amountEditText, "1100")
    assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000")
    assertNotDisplayed(descriptionText)
    isButtonDisabled(continueButton)
    }
    

    View Slide

30. ...
    assertClickableNavigateUpButton()
    onView(allOf(withParent(withId(originContainer)), withId(nameText)))
    .check(matches(allOf(isDisplayed(), withText("John Doe"))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText)))
    .check(matches(allOf(isDisplayed(), withText("JD"))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText)))
    .check(matches(allOf(isDisplayed(), withText("£10.00"))))
    assertDisplayed(descriptionText, "Send to Namey")
    assertNotDisplayed(errorText)
    isButtonEnabled(continueButton)
    writeTo(amountEditText, "1100")
    assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000")
    assertNotDisplayed(descriptionText)
    isButtonDisabled(continueButton)
    }
    

    View Slide

31. ...
    assertClickableNavigateUpButton()
    onView(allOf(withParent(withId(originContainer)), withId(nameText)))
    .check(matches(allOf(isDisplayed(), withText("John Doe"))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText)))
    .check(matches(allOf(isDisplayed(), withText("JD"))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText)))
    .check(matches(allOf(isDisplayed(), withText("£10.00"))))
    assertDisplayed(descriptionText, "Send to Namey")
    assertNotDisplayed(errorText)
    isButtonEnabled(continueButton)
    writeTo(amountEditText, "1100")
    assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000")
    assertNotDisplayed(descriptionText)
    isButtonDisabled(continueButton)
    }
    

    View Slide

32. ...
    assertClickableNavigateUpButton()
    onView(allOf(withParent(withId(originContainer)), withId(nameText)))
    .check(matches(allOf(isDisplayed(), withText("John Doe"))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText)))
    .check(matches(allOf(isDisplayed(), withText("JD"))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText)))
    .check(matches(allOf(isDisplayed(), withText("£10.00"))))
    assertDisplayed(descriptionText, "Send to Namey")
    assertNotDisplayed(errorText)
    isButtonEnabled(continueButton)
    writeTo(amountEditText, "1100")
    assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000")
    assertNotDisplayed(descriptionText)
    isButtonDisabled(continueButton)
    }
    

    View Slide

33. Robot Pattern
    View ViewModel Domain Data
    What
    How
    How
    N26 Android App
    

    View Slide

34. Robot Pattern
    View ViewModel Domain Data
    What
    How
    How
    N26 Android App
    Test
    Test
    What & How
    

    View Slide

35. Robot Pattern
    View ViewModel Domain Data
    What
    How
    How
    N26 Android App
    Robot
    How
    Test
    What
    

    View Slide

36. Robot Pattern
    View ViewModel Domain Data
    What
    How
    How
    N26 Android App
    Robot
    Test
    How
    What
    

    View Slide

37. ...
    assertClickableNavigateUpButton()
    onView(allOf(withParent(withId(originContainer)), withId(nameText)))
    .check(matches(allOf(isDisplayed(), withText("John Doe"))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText)))
    .check(matches(allOf(isDisplayed(), withText("JD"))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText)))
    .check(matches(allOf(isDisplayed(), withText("£10.00"))))
    assertDisplayed(descriptionText, "Send to Namey")
    assertNotDisplayed(errorText)
    isButtonEnabled(continueButton)
    writeTo(amountEditText, "1100")
    assertDisplayed(errorText, "You can only send MoneyBeams up to £1,000")
    assertNotDisplayed(descriptionText)
    isButtonDisabled(continueButton)
    }
    

    View Slide

38. ...
    moneyBeamSampleRobot {
    assertClickableNavigateUpButton()
    isOriginDisplayed("John Doe", "JD", "£10.00")
    isDestinationDisplayed("Namey McNameface", "NM")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonDisabled()
    enterAmount("1100")
    isErrorDisplayed("You can only send MoneyBeams up to £1,000")
    isContinueButtonDisabled()
    enterAmount("10")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonEnabled()
    }
    }
    

    View Slide

39. ...
    moneyBeamSampleRobot {
    assertClickableNavigateUpButton()
    isOriginDisplayed("John Doe", "JD", "£10.00")
    isDestinationDisplayed("Namey McNameface", "NM")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonDisabled()
    enterAmount("1100")
    isErrorDisplayed("You can only send MoneyBeams up to £1,000")
    isContinueButtonDisabled()
    enterAmount("10")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonEnabled()
    }
    }
    

    View Slide

40. ...
    moneyBeamSampleRobot {
    assertClickableNavigateUpButton()
    isOriginDisplayed("John Doe", "JD", "£10.00")
    isDestinationDisplayed("Namey McNameface", "NM")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonDisabled()
    enterAmount("1100")
    isErrorDisplayed("You can only send MoneyBeams up to £1,000")
    isContinueButtonDisabled()
    enterAmount("10")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonEnabled()
    }
    }
    

    View Slide

41. ...
    moneyBeamSampleRobot {
    assertClickableNavigateUpButton()
    isOriginDisplayed("John Doe", "JD", "£10.00")
    isDestinationDisplayed("Namey McNameface", "NM")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonDisabled()
    enterAmount("1100")
    isErrorDisplayed("You can only send MoneyBeams up to £1,000")
    isContinueButtonDisabled()
    enterAmount("10")
    isDescriptionDisplayed("Send to Namey")
    isContinueButtonEnabled()
    }
    }
    

    View Slide

42. class MoneyBeamSampleRobot : BaseRobot() {
    private val amountEditText = R.id.editMoneyBeamTransferAmount
    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription
    private val errorText = R.id.textMoneyBeamTransferAmountError
    // other view IDs
    fun isDescriptionDisplayed(text: String) {
    assertDisplayed(descriptionText, text)
    assertNotDisplayed(errorText)
    }
    fun isErrorDisplayed(text: String) { ... }
    fun isOriginDisplayed(name: String, initials: String, balance: String) {
    onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance))))
    }
    fun isDestinationDisplayed(name: String, initials: String) { ... }
    fun isContinueButtonEnabled() = isButtonEnabled(continueButton)
    fun isContinueButtonDisabled() = isButtonDisabled(continueButton)
    fun enterAmount(amount: String) = writeTo(amountEditText, amount)
    fun clickOnContinueButton() = clickOn(continueButton)
    }
    

    View Slide

43. class MoneyBeamSampleRobot : BaseRobot() {
    private val amountEditText = R.id.editMoneyBeamTransferAmount
    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription
    private val errorText = R.id.textMoneyBeamTransferAmountError
    // other view IDs
    fun isDescriptionDisplayed(text: String) {
    assertDisplayed(descriptionText, text)
    assertNotDisplayed(errorText)
    }
    fun isErrorDisplayed(text: String) { ... }
    fun isOriginDisplayed(name: String, initials: String, balance: String) {
    onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance))))
    }
    fun isDestinationDisplayed(name: String, initials: String) { ... }
    fun isContinueButtonEnabled() = isButtonEnabled(continueButton)
    fun isContinueButtonDisabled() = isButtonDisabled(continueButton)
    fun enterAmount(amount: String) = writeTo(amountEditText, amount)
    fun clickOnContinueButton() = clickOn(continueButton)
    }
    

    View Slide

44. class MoneyBeamSampleRobot : BaseRobot() {
    private val amountEditText = R.id.editMoneyBeamTransferAmount
    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription
    private val errorText = R.id.textMoneyBeamTransferAmountError
    // other view IDs
    fun isDescriptionDisplayed(text: String) {
    assertDisplayed(descriptionText, text)
    assertNotDisplayed(errorText)
    }
    fun isErrorDisplayed(text: String) { ... }
    fun isOriginDisplayed(name: String, initials: String, balance: String) {
    onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance))))
    }
    fun isDestinationDisplayed(name: String, initials: String) { ... }
    fun isContinueButtonEnabled() = isButtonEnabled(continueButton)
    fun isContinueButtonDisabled() = isButtonDisabled(continueButton)
    fun enterAmount(amount: String) = writeTo(amountEditText, amount)
    fun clickOnContinueButton() = clickOn(continueButton)
    }
    

    View Slide

45. class MoneyBeamSampleRobot : BaseRobot() {
    private val amountEditText = R.id.editMoneyBeamTransferAmount
    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription
    private val errorText = R.id.textMoneyBeamTransferAmountError
    // other view IDs
    fun isDescriptionDisplayed(text: String) {
    assertDisplayed(descriptionText, text)
    assertNotDisplayed(errorText)
    }
    fun isErrorDisplayed(text: String) { ... }
    fun isOriginDisplayed(name: String, initials: String, balance: String) {
    onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance))))
    }
    fun isDestinationDisplayed(name: String, initials: String) { ... }
    fun isContinueButtonEnabled() = isButtonEnabled(continueButton)
    fun isContinueButtonDisabled() = isButtonDisabled(continueButton)
    fun enterAmount(amount: String) = writeTo(amountEditText, amount)
    fun clickOnContinueButton() = clickOn(continueButton)
    }
    

    View Slide

46. class MoneyBeamSampleRobot : BaseRobot() {
    private val amountEditText = R.id.editMoneyBeamTransferAmount
    private val descriptionText = R.id.textMoneyBeamTransferAmountDescription
    private val errorText = R.id.textMoneyBeamTransferAmountError
    // other view IDs
    fun isDescriptionDisplayed(text: String) {
    assertDisplayed(descriptionText, text)
    assertNotDisplayed(errorText)
    }
    fun isErrorDisplayed(text: String) { ... }
    fun isOriginDisplayed(name: String, initials: String, balance: String) {
    onView(allOf(withParent(withId(originContainer)), withId(nameText))).check(matches(allOf(isDisplayed(), withText(name))))
    onView(allOf(withParent(withId(originContainer)), withId(initialsText))).check(matches(allOf(isDisplayed(), withText(initials))))
    onView(allOf(withParent(withId(originContainer)), withId(balanceText))).check(matches(allOf(isDisplayed(), withText(balance))))
    }
    fun isDestinationDisplayed(name: String, initials: String) { ... }
    fun isContinueButtonEnabled() = isButtonEnabled(continueButton)
    fun isContinueButtonDisabled() = isButtonDisabled(continueButton)
    fun enterAmount(amount: String) = writeTo(amountEditText, amount)
    fun clickOnContinueButton() = clickOn(continueButton)
    }
    

    View Slide

47. Challenges
    

    View Slide

48. Animations
    "To avoid flakiness, we highly recommend that you turn off system animations on the
    virtual or physical devices used for testing." - Espresso docs
    

    View Slide

49. Animations
    • System animations
    • Dynamic Animations from AndroidX
    
    
    Our solution: add a switch to turn off custom animations
    

    View Slide

50. Dealing Asynchronicity
    • Tests need to know when to wait and proceed
    • Causing flakiness
    Our solution: make use of Idling Resources
    • OkHttpIdlingResource
    • RxIdler
    

    View Slide

51. Mocking Data Sources
    

    View Slide

52. Mocking Data Sources
    Domain
    Data
    

    View Slide

53. Mocking Data Sources
    Domain
    Data
    Repository
    Remote Data Source
    

    View Slide

54. Mocking Data Sources
    Domain
    Data
    Repository
    Remote Data Source Framework Data Source
    
    

    View Slide

55. Mocking Data Sources
    Domain
    Data
    Repository
    Remote Data Source Fake Data Source
    
    

    View Slide

56. Dealing with Localization
    English (US) English (Canada)
    • Same account (United States)
    • Same currency (USD)
    • Same language (English)
    • Different device locale
    • Result: different money representation
    

    View Slide

57. Dealing with Localization
    • Using device locale for formatting monetary amounts
    • Test device's locale is unpredictable
    • Causing flaky tests
    Our solution: add a test rule to define the device locale
    

    View Slide

58. Dealing With Flakiness
    • Identify the cause and update the test
    • Add retry for failed tests (FlakyTestRule from the Barista library)
    

    View Slide

59. Integration with CI Pipeline
    

    View Slide

60. Integration with CI Pipeline
    3 min
    

    View Slide

61. Integration with CI Pipeline
    5 min
    

    View Slide

62. Integration with CI Pipeline
    10 min
    

    View Slide

63. Integration with CI Pipeline
    20 min
    

    View Slide

64. Integration with CI Pipeline
    Then what?
    

    View Slide

65. Selectively Running the Tests
    • Find changed files from the Git history
    • Find which features are changed
    • Run the tests of the changed features
    • If there is a change in a common module, trigger the full run
    • See: goo.gle/androidx-dependency-tracker
    

    View Slide

66. 20+ minutes
    UI Testing Pipeline
    

    View Slide

67. 3 min
    UI Testing Pipeline
    
    

    View Slide

68. android-project git:(my-branch) ./run-on-remote.sh -h
    |
    Running on Remote
    usage:
    -s | --skip-build Skip building tasks
    -n name | --test-name name Specify the name of the test. Default is branch name
    -d device | --device device Specify the name of the device on which the tests will be run. Default is random
    -p packages | --packages packages The list of packages to run the tests (comma separated). Default is all
    -l | --log Log the output to log.txt
    -h | --help Display help
    

    View Slide

69. Running on Remote
    Test Name Device Name
    Execution Time
    Duration
    
    my-branch-name
    my-branch-name
    my-branch-name
    

    View Slide

70. Reporting the Test Results
    • Link Espresso tests with test case IDs
    • Add a test rule to watch the test results
    • Post the test results to test management tool
    

    View Slide

71. Sample Test Run
    MoneyBeam Spaces Premium
    

    View Slide

72. Unit Tests
    ~500 UI Tests
    Functional UI
    Tests
    Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019
    ~9000 Unit tests
    Today
    

    View Slide

73. Unit Tests
    ~500 UI Tests
    Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today
    Functional UI Tests
    ~9000 Unit tests
    

    View Slide

74. Unit Tests
    ~500 UI Tests
    Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today
    Functional UI Tests
    Automated Regression
    10%
    ~9000 Unit tests
    

    View Slide

75. Unit Tests
    ~500 UI Tests
    Oct 14 Jan 2016 Jan 2017 Jan 2018 Feb 2019 Today
    Functional UI Tests
    Automated Regression
    50%
    ~9000 Unit tests
    

    View Slide

76. •
    Faster test execution
    •
    Part of our CI pipeline
    •
    Maintained together with the production code
    •
    ☑ Mostly trusty
    Espresso Project
    

    View Slide

77. A Few Ideas
    

    View Slide

78. Feature 1 Feature N
    Base
    App
    

    View Slide

79. App
    Feature 1
    Base
    

    View Slide

80. App Feature 1 App
    Feature 1
    Base
    

    View Slide

81. App Feature 1 App
    Feature 1
    Base
    
    

    View Slide

82. Feature 1 App
    Running Tests on Feature Apps
    Feature 1
    
    

    View Slide

83. Is amount input displayed?
    ✅ Passed
    Is amount input displayed?
    ✅ Passed
    

    View Slide

84. Snapshot Testing
    Verifies view attributes:
    • Position
    • Color
    • Size
    Is amount input displayed?
    ✅ Passed
    Is amount input displayed?
    ✅ Passed
    

    View Slide

85. Snapshot Testing
    Verifies view attributes:
    • Position
    • Color
    • Size
    Is amount input displayed?
    ✅ Passed
    Is amount input displayed?
    ❌ Failed
    

    View Slide

86. Thank You
    Questions?
    

    View Slide