DevFestFL 2017 - Espresso: A Screenshot is Worth 1,000 Words

5701f31a8433a22ae736282de8d08cd6?s=47 Sam Edwards
November 11, 2017

DevFestFL 2017 - Espresso: A Screenshot is Worth 1,000 Words

5701f31a8433a22ae736282de8d08cd6?s=128

Sam Edwards

November 11, 2017
Tweet

Transcript

  1. A Screenshot is Worth 1,000 Words Sam Edwards - @HandstandSam

    - Capital One
  2. Sam Edwards @HandstandSam • Android GDE • Architected and leading

    Espresso efforts for Capital One Wallet • Opinions in this talk are my own. http://handstandsam.com
  3. Audience POLL: • Testing? • Espresso? • Espresso with CI?

    • Screenshots?
  4. Outline • What is Espresso? • Screenshots - Why? How?

    When? The Cost? • How to get “Screenshots for Free” • Tips and Tricks
  5. None
  6. None
  7. What is Espresso? • A library provided by Google for

    Android UI testing • Built on Android Test Instrumentation • Aware of the idle state and therefore FAST • Easy to read and write
  8. Espresso Basics https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

  9. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

  10. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

  11. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

  12. Disable Animations

  13. NEVER Thread.sleep() • AsyncTask Threadpool ◦ Leverage for Networking and

    RxJava • Idling Resources - Use CountingIdlingResource ◦ increment(); and decrement(); https://developer.android.com/reference/android/support/test/espresso/contrib/CountingIdlingResource.html
  14. Screenshots - WHY?

  15. 1. See What is Being Tested

  16. Terminal Output Test Results

  17. Android Studio Test Results

  18. Spoon Test Report without Screenshots

  19. Spoon Test Report with Screenshots

  20. 1. See What is Being Tested 2. Diagnose Failures

  21. Oops, Your Test Failed android.support.test.espresso.NoMatchingViewException: No views in hierarchy found

    matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2}
  22. Ahh, I See Why android.support.test.espresso.NoMatchingViewException: No views in hierarchy found

    matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false,
  23. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
  24. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
  25. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
  26. Screenshot on FAILURE Example Report

  27. 1. See What is Being Tested 2. Diagnose Failures 3.

    Share Your Tests
  28. NO ONE Sees The Value of Your Tests… YET •

    Your Boss • Product Owners • Designers • Manual Testers
  29. That’s because they can’t tell what your tests are doing

    and don’t find any use for them. To them it’s just console output.
  30. Let EVERYONE see what’s being tested.

  31. User Interfaces are VISUAL • A User Interface is something

    very hard to represent in a non-visual way. • Exercise: ◦ Imagine you need to tell someone who is blind what the Mona Lisa painting looks like. What would you say? https://en.wikipedia.org/wiki/Mona_Lisa
  32. Same Test, But Looks Different on Different Devices

  33. None
  34. None
  35. Android Embraces Differences

  36. LANGUAGE DEVICE TYPE DEVICE SIZE ANDROID OS VERSION Fragmentation and

    Permutations
  37. Test Runners and HTML Reports • Run Instrumentation Commands •

    Collect Logs • Collect Screenshots • Generate HTML Reports
  38. Spoon - http://square.github.io/spoon/

  39. Composer - https://github.com/gojuno/composer

  40. Spoon Test Runner java -jar spoon-runner-1.7.0-jar-with-dependencies.jar \ --apk example-app.apk \

    --test-apk example-tests.apk • Run instrumentation tests in parallel • Shard tests across multiple devices • Beautiful HTML reports
  41. Screenshots - HOW?

  42. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon

  43. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon

  44. Falcon & (Falcon Spoon Compat) FalconSpoon.screenshot(activity, tag); FalconSpoon.screenshot(activity, tag, testClassName,

    testMethodName); https://github.com/jraska/Falcon/
  45. UiAutomator File outputFile = Spoon.screenshot(activity, tag, testClass, testMethod); uiDevice.takeScreenshot(outputFile); •

    Works inside and outside of your app • API 18+ (Jelly Bean MR2) https://developer.android.com/topic/libraries/testing-support-library/in dex.html#UIAutomator
  46. Other Alternatives • Firebase Test Lab’s “Screenshotter” ◦ https://firebase.google.com/docs/test-lab/test-screenshots •

    Fastlane’s “Screengrab” ◦ https://github.com/fastlane/fastlane/tree/master/screengrab
  47. Typical Espresso

  48. Typical Espresso Test @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public

    ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }
  49. Typical Espresso Test With Screenshots @RunWith(AndroidJUnit4.class) public class LoginTest {

    @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); takeScreenshot("entered_username"); takeScreenshot("clicking_login"); onView(withText("LOGIN")).perform(click()); takeScreenshot("homescreen_shown"); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }
  50. “Screenshots for Free” via Robots

  51. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test
  52. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test X X X X X X X View Changed
  53. Robot Testing Pattern View Presenter Model What How

  54. Robot Testing Pattern Test View Presenter Model What & How

    What How
  55. Robot Testing Pattern Robot View Presenter Model How What How

    Test What
  56. UI Test Architecture Robot Testing Pattern Robot View Presenter Model

    Test
  57. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test Robot
  58. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test Robot X View Changed
  59. Espresso Test With a Robot @RunWith(AndroidJUnit4.class) public class LoginTest {

    @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ new LoginRobot().username("sam").login().assertHomeScreenShown(); } }
  60. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  61. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  62. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  63. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  64. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT

    • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  65. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT

    • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
  66. Screenshots - At What Cost?

  67. Execution Time in Minutes

  68. Report Size in MB

  69. When Should I Take Screenshots? • Well, it depends… but

    ALWAYS on FAILURE • To generate reports for product and design teams • On small/low-res emulators with mdpi screen density for smaller screenshots.
  70. Use Screenshots Wisely • Programmatically configurable • Change config without

    recompiling ◦ Leverage Instrumentation Args ▪ --e screenshots=true ▪ --e screenshots=false
  71. “Easy Screenshots” Coming Soon...

  72. Easy Screenshots • Screenshots on Failure for FREE • Settings:

    ALL, FAILURE_ONLY or NONE ◦ Can be changed: ▪ Programmatically ▪ Via instrumentation args • Configurable ◦ Reporting Framework ◦ Screenshot Implementation
  73. 1,000 Words Sam Edwards @HandstandSam

  74. None
  75. Visualize What Your Tests are Doing

  76. Tips and Tricks

  77. “Can’t write to sdcard” error • Use @GrantPermissionRule from latest

    Android Test Support Library • Declare in app’s Manifest: ◦ READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE
  78. Leverage Continuous Integration • Jenkins • Circle CI • BuddyBuild

    • etc
  79. Capital One Wallet

  80. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }
  81. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }
  82. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }
  83. Target Audience for Espresso “Espresso is targeted at developers, who

    believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test.”
  84. Learn More About Espresso

  85. Screenshot Libraries - Dialog Comparison UiAutomator Falcon Spoon

  86. Execution Time Overhead in Minutes 384x640 mdpi Nexus 5 Nexus

    6P Falcon +16.53% +62.31% +114.39% UiAutomator +28.93% +116.15% +168.87%
  87. Report Size Overhead in MB 384x640 mdpi Nexus 5 Nexus

    6P Falcon +17.35% +66.41% +90.13% UiAutomator +18.41% +64.77% +89.19%
  88. Get Currently RESUMED Activity private static Activity getCurrentActivityInstance() { final

    Activity[] activity = new Activity[1]; getInstrumentation().runOnMainSync(new Runnable() { public void run() { Collection resumedActivities = ActivityLifecycleMonitorRegistry.getInstance() .getActivitiesInStage(RESUMED); if (resumedActivities.iterator().hasNext()) { Activity currentActivity = (Activity) resumedActivities.iterator().next(); activity[0] = currentActivity; logger.trace("Activity obtained of type: " + currentActivity.getClass().getName()); } } }); return activity[0]; }
  89. Get Test Method & Class from Current Stack Trace public

    static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }
  90. Get Test Method & Class from Current Stack Trace public

    static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }
  91. Resulting Screenshot Code screenshot(activity, “username_entered”, testClass, testMethod); takeScreenshot(“username_entered”);

  92. Pixel by Pixel Visual Regression Testing

  93. Deterministic View Screenshot Comparison

  94. Screenshot-tests-for-android by Facebook public class MyTests { @Test public void

    doScreenshot() { View view = mLayoutInflater.inflate(R.layout.my_layout, null, false); ViewHelpers.setupView(view) .setExactWidthDp(300) .layout(); Screenshot.snap(view) .record(); } }
  95. Robot Testing Pattern View Presenter Model What How

  96. Application Architecture Robot Testing Pattern View Presenter Model

  97. Application Architecture Robot Testing Pattern View Presenter Model Test

  98. Android SDK Hierarchy Viewer

  99. How do I get started?

  100. Espresso Test Recorder in Android Studio 2.2+ https://developer.android.com/studio/test/espresso-test-recorder.html

  101. Espresso Test Recorder in Android Studio 2.2+

  102. Robot Testing Pattern • A Test Architecture Pattern for Maintainability

    • Learned about it from Jake Wharton’s talk: ◦ VIDEO: https://realm.io/news/kau-jake-wharton-testing-robots/ ◦ SLIDES: https://speakerdeck.com/jakewharton/testing-robots-kotlin-night-may-2016
  103. Create a Custom Test Runner • Control test runner behavior

    without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false
  104. Define Your “LOW BAR” Device • Android Emulator Virtual Device

    • 384x640 • mdpi - 160 dots per inch • Able to run in continuous integration
  105. Android Studio - Layout Inspector

  106. Android Studio - Layout Inspector

  107. Test Butler by LinkedIn “Reliable Android testing, at your service.”

    • Stabilizes the Android emulator ◦ Disables animations ◦ Disables crash & ANR dialogs ◦ Locks the keyguard, WiFi radio, and CPU to ensure they don't go to sleep unexpectedly while tests are running. • Handles changing global emulator settings and holds relevant permissions so your app doesn't have to. ◦ Enable/disable WiFi ◦ Change device orientation ◦ Set location services mode ◦ Set application locale https://github.com/linkedin/test-butler
  108. Create a Custom Test Runner • Control test runner behavior

    without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false
  109. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner

    { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }
  110. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner

    { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }
  111. GIFs • Generated by default • Processing overhead ◦ 11

    Seconds for 21 Screenshots -> • Disable using “--no-animations”
  112. To Add Android Test Orchestrator Composer Reporting @GrantPermissionRule