DevFestFL 2017 - Espresso: A Screenshot is Worth 1,000 Words

Transcript

1. A Screenshot is Worth 1,000 Words Sam Edwards - @HandstandSam

   - Capital One

2. Sam Edwards @HandstandSam • Android GDE • Architected and leading

   Espresso efforts for Capital One Wallet • Opinions in this talk are my own. http://handstandsam.com

3. Audience POLL: • Testing? • Espresso? • Espresso with CI?

   • Screenshots?

4. Outline • What is Espresso? • Screenshots - Why? How?

   When? The Cost? • How to get “Screenshots for Free” • Tips and Tricks
5. None
6. None

7. What is Espresso? • A library provided by Google for

   Android UI testing • Built on Android Test Instrumentation • Aware of the idle state and therefore FAST • Easy to read and write

8. Espresso Basics https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

9. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

10. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

11. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/

12. Disable Animations

13. NEVER Thread.sleep() • AsyncTask Threadpool ◦ Leverage for Networking and

    RxJava • Idling Resources - Use CountingIdlingResource ◦ increment(); and decrement(); https://developer.android.com/reference/android/support/test/espresso/contrib/CountingIdlingResource.html

14. Screenshots - WHY?

15. 1. See What is Being Tested

16. Terminal Output Test Results

17. Android Studio Test Results

18. Spoon Test Report without Screenshots

19. Spoon Test Report with Screenshots

20. 1. See What is Being Tested 2. Diagnose Failures

21. Oops, Your Test Failed android.support.test.espresso.NoMatchingViewException: No views in hierarchy found

    matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2}

22. Ahh, I See Why android.support.test.espresso.NoMatchingViewException: No views in hierarchy found

    matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false,

23. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });

24. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });

25. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable

    throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });

26. Screenshot on FAILURE Example Report

27. 1. See What is Being Tested 2. Diagnose Failures 3.

    Share Your Tests

28. NO ONE Sees The Value of Your Tests… YET •

    Your Boss • Product Owners • Designers • Manual Testers

29. That’s because they can’t tell what your tests are doing

    and don’t find any use for them. To them it’s just console output.

30. Let EVERYONE see what’s being tested.

31. User Interfaces are VISUAL • A User Interface is something

    very hard to represent in a non-visual way. • Exercise: ◦ Imagine you need to tell someone who is blind what the Mona Lisa painting looks like. What would you say? https://en.wikipedia.org/wiki/Mona_Lisa

32. Same Test, But Looks Different on Different Devices

33. None
34. None

35. Android Embraces Differences

36. LANGUAGE DEVICE TYPE DEVICE SIZE ANDROID OS VERSION Fragmentation and

    Permutations

37. Test Runners and HTML Reports • Run Instrumentation Commands •

    Collect Logs • Collect Screenshots • Generate HTML Reports

38. Spoon - http://square.github.io/spoon/

39. Composer - https://github.com/gojuno/composer

40. Spoon Test Runner java -jar spoon-runner-1.7.0-jar-with-dependencies.jar \ --apk example-app.apk \

    --test-apk example-tests.apk • Run instrumentation tests in parallel • Shard tests across multiple devices • Beautiful HTML reports

41. Screenshots - HOW?

42. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon

43. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon

44. Falcon & (Falcon Spoon Compat) FalconSpoon.screenshot(activity, tag); FalconSpoon.screenshot(activity, tag, testClassName,

    testMethodName); https://github.com/jraska/Falcon/

45. UiAutomator File outputFile = Spoon.screenshot(activity, tag, testClass, testMethod); uiDevice.takeScreenshot(outputFile); •

    Works inside and outside of your app • API 18+ (Jelly Bean MR2) https://developer.android.com/topic/libraries/testing-support-library/in dex.html#UIAutomator

46. Other Alternatives • Firebase Test Lab’s “Screenshotter” ◦ https://firebase.google.com/docs/test-lab/test-screenshots •

    Fastlane’s “Screengrab” ◦ https://github.com/fastlane/fastlane/tree/master/screengrab

47. Typical Espresso

48. Typical Espresso Test @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public

    ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }

49. Typical Espresso Test With Screenshots @RunWith(AndroidJUnit4.class) public class LoginTest {

    @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); takeScreenshot("entered_username"); takeScreenshot("clicking_login"); onView(withText("LOGIN")).perform(click()); takeScreenshot("homescreen_shown"); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }

50. “Screenshots for Free” via Robots

51. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test

52. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test X X X X X X X View Changed

53. Robot Testing Pattern View Presenter Model What How

54. Robot Testing Pattern Test View Presenter Model What & How

    What How

55. Robot Testing Pattern Robot View Presenter Model How What How

    Test What

56. UI Test Architecture Robot Testing Pattern Robot View Presenter Model

    Test

57. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test Robot

58. Robot Testing Pattern View Presenter Model Test What Test Test

    Test Test Test Test Robot X View Changed

59. Espresso Test With a Robot @RunWith(AndroidJUnit4.class) public class LoginTest {

    @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ new LoginRobot().username("sam").login().assertHomeScreenShown(); } }

60. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

61. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

62. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

63. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final

    Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

64. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT

    • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

65. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT

    • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }

66. Screenshots - At What Cost?

67. Execution Time in Minutes

68. Report Size in MB

69. When Should I Take Screenshots? • Well, it depends… but

    ALWAYS on FAILURE • To generate reports for product and design teams • On small/low-res emulators with mdpi screen density for smaller screenshots.

70. Use Screenshots Wisely • Programmatically configurable • Change config without

    recompiling ◦ Leverage Instrumentation Args ▪ --e screenshots=true ▪ --e screenshots=false

71. “Easy Screenshots” Coming Soon...

72. Easy Screenshots • Screenshots on Failure for FREE • Settings:

    ALL, FAILURE_ONLY or NONE ◦ Can be changed: ▪ Programmatically ▪ Via instrumentation args • Configurable ◦ Reporting Framework ◦ Screenshot Implementation

73. 1,000 Words Sam Edwards @HandstandSam

74. None

75. Visualize What Your Tests are Doing

76. Tips and Tricks

77. “Can’t write to sdcard” error • Use @GrantPermissionRule from latest

    Android Test Support Library • Declare in app’s Manifest: ◦ READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE

78. Leverage Continuous Integration • Jenkins • Circle CI • BuddyBuild

    • etc

79. Capital One Wallet

80. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }

81. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }

82. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click());

    }

83. Target Audience for Espresso “Espresso is targeted at developers, who

    believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test.”

84. Learn More About Espresso

85. Screenshot Libraries - Dialog Comparison UiAutomator Falcon Spoon

86. Execution Time Overhead in Minutes 384x640 mdpi Nexus 5 Nexus

    6P Falcon +16.53% +62.31% +114.39% UiAutomator +28.93% +116.15% +168.87%

87. Report Size Overhead in MB 384x640 mdpi Nexus 5 Nexus

    6P Falcon +17.35% +66.41% +90.13% UiAutomator +18.41% +64.77% +89.19%

88. Get Currently RESUMED Activity private static Activity getCurrentActivityInstance() { final

    Activity[] activity = new Activity[1]; getInstrumentation().runOnMainSync(new Runnable() { public void run() { Collection resumedActivities = ActivityLifecycleMonitorRegistry.getInstance() .getActivitiesInStage(RESUMED); if (resumedActivities.iterator().hasNext()) { Activity currentActivity = (Activity) resumedActivities.iterator().next(); activity[0] = currentActivity; logger.trace("Activity obtained of type: " + currentActivity.getClass().getName()); } } }); return activity[0]; }

89. Get Test Method & Class from Current Stack Trace public

    static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }

90. Get Test Method & Class from Current Stack Trace public

    static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }

91. Resulting Screenshot Code screenshot(activity, “username_entered”, testClass, testMethod); takeScreenshot(“username_entered”);

92. Pixel by Pixel Visual Regression Testing

93. Deterministic View Screenshot Comparison

94. Screenshot-tests-for-android by Facebook public class MyTests { @Test public void

    doScreenshot() { View view = mLayoutInflater.inflate(R.layout.my_layout, null, false); ViewHelpers.setupView(view) .setExactWidthDp(300) .layout(); Screenshot.snap(view) .record(); } }

95. Robot Testing Pattern View Presenter Model What How

96. Application Architecture Robot Testing Pattern View Presenter Model

97. Application Architecture Robot Testing Pattern View Presenter Model Test

98. Android SDK Hierarchy Viewer

99. How do I get started?

100. Espresso Test Recorder in Android Studio 2.2+ https://developer.android.com/studio/test/espresso-test-recorder.html

101. Espresso Test Recorder in Android Studio 2.2+

102. Robot Testing Pattern • A Test Architecture Pattern for Maintainability

     • Learned about it from Jake Wharton’s talk: ◦ VIDEO: https://realm.io/news/kau-jake-wharton-testing-robots/ ◦ SLIDES: https://speakerdeck.com/jakewharton/testing-robots-kotlin-night-may-2016

103. Create a Custom Test Runner • Control test runner behavior

     without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false

104. Define Your “LOW BAR” Device • Android Emulator Virtual Device

     • 384x640 • mdpi - 160 dots per inch • Able to run in continuous integration

105. Android Studio - Layout Inspector

106. Android Studio - Layout Inspector

107. Test Butler by LinkedIn “Reliable Android testing, at your service.”

     • Stabilizes the Android emulator ◦ Disables animations ◦ Disables crash & ANR dialogs ◦ Locks the keyguard, WiFi radio, and CPU to ensure they don't go to sleep unexpectedly while tests are running. • Handles changing global emulator settings and holds relevant permissions so your app doesn't have to. ◦ Enable/disable WiFi ◦ Change device orientation ◦ Set location services mode ◦ Set application locale https://github.com/linkedin/test-butler

108. Create a Custom Test Runner • Control test runner behavior

     without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false

109. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner

     { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }

110. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner

     { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }

111. GIFs • Generated by default • Processing overhead ◦ 11

     Seconds for 21 Screenshots -> • Disable using “--no-animations”

112. To Add Android Test Orchestrator Composer Reporting @GrantPermissionRule