Maintainable Espresso Tests with Robots and Screenshots

Maintainable Espresso Tests with Robots and Screenshots

AnDevCon 2017 - Washington DC
VIDEO: https://www.youtube.com/watch?v=qvzlQB6aieY

5701f31a8433a22ae736282de8d08cd6?s=128

Sam Edwards

July 17, 2017
Tweet

Transcript

  1. MAINTAINABLE ESPRESSO TESTS WITH ROBOTS AND SCREENSHOTS SAM EDWARDS -

    @HANDSTANDSAM
  2. The Dream Espresso Test Suite Easy to write Low maintenance

    UI tests added during the development cycle Hermetic, deterministic, targeted Run on every PR in Continuous Integration Reliable test execution environment Automated screenshots
  3. Audience Poll: Testing? Espresso? Screenshots? Continuous Integration?

  4. Why Espresso? Created by Google Access to app internals Easy

    to write Blazing fast
  5. Using Espresso

  6. Using Espresso

  7. Using Espresso

  8. Using Espresso

  9. Espresso Example @Test public void testLogin() { onView(withText(“Shopping App”))
 .check(matches(isDisplayed()));

    onView(withId(R.id.username))
 .perform(typeText("sam")); onView(withText(“LOG IN"))
 .perform(click()); }
  10. Espresso Example @Test public void testLogin() { onView(withText(“Shopping App”))
 .check(matches(isDisplayed()));

    onView(withId(R.id.username))
 .perform(typeText("sam")); onView(withText(“LOG IN"))
 .perform(click()); }
  11. Espresso Example @Test public void testLogin() { onView(withText(“Shopping App”))
 .check(matches(isDisplayed()));

    onView(withId(R.id.username))
 .perform(typeText("sam")); onView(withText(“LOG IN"))
 .perform(click()); }
  12. Espresso Test Recorder (Android Studio 2.2+) PROS: Great for beginners

    Great for projects without existing tests CONS: Only works on simple tests Tool is slow Generated code
  13. Espresso Test Recorder (Android Studio 2.2+)

  14. Espresso Test Recorder (Android Studio 2.2+)

  15. Espresso Test Recorder (Android Studio 2.2+)

  16. Espresso Test Recorder (Android Studio 2.2+)

  17. Layout Inspector

  18. None
  19. Who should write Espresso tests? “Espresso is targeted at developers,

    who believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test.”
  20. When Should I write an Espresso test? Avoid unless necessary

    Only testing the UI Small and targeted
  21. How is Espresso Fast? Aware of Android’s Main Thread Aware

    of Android’s AsyncTask Thread Pool Idling Resources Never have to Thread.sleep(5000);
  22. None
  23. None
  24. None
  25. None
  26. Configuration for Success Disabled Animations Use Idling Resources for Background

    work Utilize AsyncTask Thread pool
  27. Where can I learn more?

  28. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST
  29. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST
  30. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST X X X X X
  31. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST ROBOT
  32. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST ROBOT X
  33. ROBOT TESTING PATTERN FOR ANDROID INSTRUMENTATION TESTS TEST TEST TEST

    TEST TEST ROBOT WHAT HOW
  34. LOGIN ROBOT new LoginRobot() .username("sam") .password("password") .toggleRememberMe() .login();

  35. new LoginRobot().username("sam").password("password").toggleRememberMe().login(); public class LoginRobot { public LoginRobot username(String username)

    { ViewInteraction usernameEditText = onView(allOf(withId(R.id.username), isDisplayed())); usernameEditText.perform(replaceText(username), closeSoftKeyboard()); return this; } public LoginRobot password(String password) { ViewInteraction passwordEditText = onView(allOf(withId(R.id.password), isDisplayed())); passwordEditText.perform(replaceText(password), closeSoftKeyboard()); return this; } public LoginRobot toggleRememberMe() { ViewInteraction rememberMeCheckbox = onView(allOf(withId(R.id.remember_me), isDisplayed())); rememberMeCheckbox.perform(scrollTo(), click()); return this; } public LoginRobot login() { ViewInteraction submitButton = onView(allOf(withId(R.id.submit), isDisplayed())); submitButton.perform(click()); return this; } }
  36. new LoginRobot().username("sam").password("password").toggleRememberMe().login(); public class LoginRobot { public LoginRobot username(String username)

    { ViewInteraction usernameEditText = onView(allOf(withId(R.id.username), isDisplayed())); usernameEditText.perform(replaceText(username), closeSoftKeyboard()); return this; } public LoginRobot password(String password) { ViewInteraction passwordEditText = onView(allOf(withId(R.id.password), isDisplayed())); passwordEditText.perform(replaceText(password), closeSoftKeyboard()); return this; } public LoginRobot toggleRememberMe() { ViewInteraction rememberMeCheckbox = onView(allOf(withId(R.id.remember_me), isDisplayed())); rememberMeCheckbox.perform(scrollTo(), click()); return this; } public LoginRobot login() { ViewInteraction submitButton = onView(allOf(withId(R.id.submit), isDisplayed())); submitButton.perform(click()); return this; } }
  37. new LoginRobot().username("sam").password("password").toggleRememberMe().login(); public class LoginRobot { public LoginRobot username(String username)

    { ViewInteraction usernameEditText = onView(allOf(withId(R.id.username), isDisplayed())); usernameEditText.perform(replaceText(username), closeSoftKeyboard()); return this; } public LoginRobot password(String password) { ViewInteraction passwordEditText = onView(allOf(withId(R.id.password), isDisplayed())); passwordEditText.perform(replaceText(password), closeSoftKeyboard()); return this; } public LoginRobot toggleRememberMe() { ViewInteraction rememberMeCheckbox = onView(allOf(withId(R.id.remember_me), isDisplayed())); rememberMeCheckbox.perform(scrollTo(), click()); return this; } public LoginRobot login() { ViewInteraction submitButton = onView(allOf(withId(R.id.submit), isDisplayed())); submitButton.perform(click()); return this; } }
  38. new LoginRobot().username("sam").password("password").toggleRememberMe().login(); public class LoginRobot { public LoginRobot username(String username)

    { ViewInteraction usernameEditText = onView(allOf(withId(R.id.username), isDisplayed())); usernameEditText.perform(replaceText(username), closeSoftKeyboard()); return this; } public LoginRobot password(String password) { ViewInteraction passwordEditText = onView(allOf(withId(R.id.password), isDisplayed())); passwordEditText.perform(replaceText(password), closeSoftKeyboard()); return this; } public LoginRobot toggleRememberMe() { ViewInteraction rememberMeCheckbox = onView(allOf(withId(R.id.remember_me), isDisplayed())); rememberMeCheckbox.perform(scrollTo(), click()); return this; } public LoginRobot login() { ViewInteraction submitButton = onView(allOf(withId(R.id.submit), isDisplayed())); submitButton.perform(click()); return this; } }
  39. HOME ROBOT ITEM
 ROBOT LOGIN ROBOT CATEGORY ROBOT new LoginRobot().username(“sam").password("password").toggleRememberMe().login();

    new HomeRobot().category(“Fruits”); new CategoryRobot().item(“Pineapple”); new ItemRobot().addToCart(); MULTI-SCREEN TESTS
  40. HOME ROBOT ITEM
 ROBOT LOGIN ROBOT CATEGORY ROBOT new LoginRobot().username(“sam").password("password").toggleRememberMe().login();

    new HomeRobot().category(“Fruits”); new CategoryRobot().item(“Pineapple”); new ItemRobot().addToCart(); MULTI-SCREEN TESTS
  41. HOME ROBOT ITEM
 ROBOT LOGIN ROBOT CATEGORY ROBOT new LoginRobot().username(“sam").password("password").toggleRememberMe().login();

    new HomeRobot().category(“Fruits”); new CategoryRobot().item(“Pineapple”); new ItemRobot().addToCart(); MULTI-SCREEN TESTS
  42. HOME ROBOT ITEM
 ROBOT LOGIN ROBOT CATEGORY ROBOT new LoginRobot().username(“sam").password("password").toggleRememberMe().login();

    new HomeRobot().category(“Fruits”); new CategoryRobot().item(“Pineapple”); new ItemRobot().addToCart(); MULTI-SCREEN TESTS
  43. HOME ROBOT ITEM
 ROBOT LOGIN ROBOT CATEGORY ROBOT new LoginRobot().username(“sam").password("password").toggleRememberMe().login();

    new HomeRobot().category(“Fruits”); new CategoryRobot().item(“Pineapple”); new ItemRobot().addToCart(); MULTI-SCREEN TESTS
  44. public class LoginRobot { public LoginRobot username(String username) { ViewInteraction

    usernameEditText = onView(allOf(withId(R.id.username), isDisplayed())); usernameEditText
 .perform(replaceText(username), closeSoftKeyboard()); takeScreenshot(“entered_username”); return this; } } SCREENSHOTS FOR EACH ROBOT METHOD
  45. public class LoginRobot { public LoginRobot login() { ViewInteraction submitButton

    = onView(allOf(withId(R.id.submit), isDisplayed())); takeScreenshot(“logging_in”); submitButton.perform(click()); return this; } } SCREENSHOTS FOR EACH ROBOT METHOD
  46. Screenshots - WHY? 1. See What’s Being Tested

  47. 1. See What’s Being Tested

  48. 1. See What’s Being Tested

  49. 1. See What’s Being Tested

  50. 1. See What’s Being Tested

  51. Screenshots - WHY? 1. See What’s Being Tested 2. Diagnose

    Failures
  52. 2. Diagnose Failures android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching:

    with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is- layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is- focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is- focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is- enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2} | +--->Toolbar{id=16909232, res-name=action_bar, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is- focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +---->TextView{id=-1, visibility=VISIBLE, width=206, height=54, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is- layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=29.0, text=Hello World, input-type=0, ime-target=false, has-links=false} | +---->ActionMenuView{id=-1, visibility=VISIBLE, width=80, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is- focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=640.0, y=0.0, child-count=1} | +----->OverflowMenuButton{id=-1, desc=More options, visibility=VISIBLE, width=80, height=96, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=true, is-enabled=true, is- focused=false, is-focusable=true, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=8.0}
  53. 2. Diagnose Failures android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching:

    with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has- window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout- requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child- count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is- focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input- connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is- focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input- connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is- focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2} | +--->Toolbar{id=16909232, res-name=action_bar, visibility=VISIBLE, width=720, height=112, has-focus=false, has- focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +---->TextView{id=-1, visibility=VISIBLE, width=206, height=54, has-focus=false, has-focusable=false, has-window- focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is- selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=29.0, text=Hello World, input-type=0, ime-target=false, has-links=false}
  54. Screenshots on Failures

  55. Screenshots on Failures Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
  56. Screenshots - WHY? 1. See What’s Being Tested 2. Diagnose

    Failures 3. Share Your Tests
  57. NO ONE Sees The Value of Your Tests… YET Your

    Bosses Product Owners Designers Manual Testers
  58. To them it’s just console output.

  59. Let EVERYONE see
 what’s being tested.

  60. User Interfaces are VISUAL A User Interface is something very

    hard to represent in a non-visual way. Imagine you need to tell someone who is visually impaired what the Mona Lisa painting looks like.
  61. Same Test, But Looks Different on Different Devices

  62. DEVICE SIZE OS VERSION DEVICE MODEL LANGUAGE

  63. Test Reports Spoon Screenshot Reports Isolated Logs for each test

    execution
  64. Spoon http://square.github.io/spoon/

  65. Spoon Test Runner Run instrumentation tests in parallel Shard tests

    across multiple devices Beautiful HTML reports java -jar spoon-runner-1.7.0-jar-with-dependencies.jar \
 --apk example-app.apk \
 --test-apk example-tests.apk
  66. Spoon HTML Reports

  67. None
  68. How Taking Screenshots Work Traverse View Hierarchy Generate Image Save

    to disk in
 a specific location adb pull screenshot report finds images
  69. Screenshot Libraries Spoon Falcon UiAutomator

  70. Other Screenshot Libraries Firebase Test Lab - Screen Shotter Fastlane

    - Screengrab
  71. Should I use Devices or Emulators? Emulator for Development and

    Continuous Integration Use devices for release regressions
  72. Use The Android Emulator Run real versions of Android Configurable

    Faster than real devices Available on demand and scalable Cheaper and lower maintenance For screenshots, run low density (mdpi - 384x640)
  73. None
  74. Combating Shared State Clearing preferences Clearing databases Clearing files Resetting

    singletons Resetting static variables Re-injecting Dependency Injections Killing any Background Work
  75. How AndroidJUnitRunner Works The more you know…

  76. None
  77. None
  78. None
  79. Isolated Test Execution Run each test completely separated adb shell

    pm clear com.mypackage (Total data wipe)
  80. None
  81. None
  82. To Achieve This Run Test Dex Parser to dump all

    methods https://github.com/linkedin/dex-test-parser Execute each test method individually Combine test results CONS: Need custom scripts Can’t use Spoon Won’t work on Firebase Test Lab
  83. Android Test Orchestrator Announced at Google I/O 2017. Available in

    the upcoming version of the Android Testing Support Library. Runs "pm clear” before each run. Service APK running in the background. Will be integrated with Android Studio.
  84. None
  85. None
  86. None
  87. None
  88. None
  89. Small, Targeted Tests Start in the activity you are testing

    with ActivityTestRule. If your login page breaks, none of your tests will pass. As with any test, isolating it and testing it with the expected inputs is the best way to understand if it’s passing.
  90. Skip to the important part Jump directly to what you

    are testing. Setup state without doing it through your UI.
  91. Hermetic and Deterministic Tests Mock everything external to what you

    are testing on the UI. Mock all API requests WireMock MockWebServer and RESTMock Mockito
  92. None
  93. None
  94. Intent Boundaries for Targeted Tests

  95. None
  96. None
  97. None
  98. Continuous Integration Tests must be run to stay up to

    date Run on every PR Start with Local machine - Have to write and maintain scripts End with cloud hosted - Firebase Test Lab, Amazon Device Farm, Circle CI, Travis CI, BuddyBuild, etc.
  99. Speed - Parallelization & Sharding Sharding Infinite availability with cloud

    based services Shard 1 Shard 2 Shard 3 Shard 4 Shard 5 Run Tests
  100. Words of Wisdom

  101. WITH ROBOTS AND SCREENSHOTS MAINTAINABLE ESPRESSO TESTS THANK YOU! SAM

    EDWARDS - @HANDSTANDSAM