DevFestFL 2017 - Espresso: A Screenshot is Worth 1,000 Words

Speaker: https://twitter.com/handstandsam
Event: https://devfestflorida.org/schedule/

Transcript

1. A Screenshot is Worth 1,000 Words Sam Edwards - @HandstandSam - Capital One
2. Sam Edwards @HandstandSam • Android GDE • Architected and leading Espresso efforts for Capital One Wallet • Opinions in this talk are my own. http://handstandsam.com
3. Audience POLL: • Testing? • Espresso? • Espresso with CI? • Screenshots?
4. Outline • What is Espresso? • Screenshots - Why? How? When? The Cost? • How to get “Screenshots for Free” • Tips and Tricks
7. What is Espresso? • A library provided by Google for Android UI testing • Built on Android Test Instrumentation • Aware of the idle state and therefore FAST • Easy to read and write
8. Espresso Basics https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/
9. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/
10. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/
11. Espresso Matchers, Actions and Assertions https://google.github.io/android-testing-support-library/docs/espresso/cheatsheet/
12. Disable Animations
13. NEVER Thread.sleep() • AsyncTask Threadpool ◦ Leverage for Networking and RxJava • Idling Resources - Use CountingIdlingResource ◦ increment(); and decrement(); https://developer.android.com/reference/android/support/test/espresso/contrib/CountingIdlingResource.html
14. Screenshots - WHY?
15. 1. See What is Being Tested
16. Terminal Output Test Results
17. Android Studio Test Results
18. Spoon Test Report without Screenshots
19. Spoon Test Report with Screenshots
20. 1. See What is Being Tested 2. Diagnose Failures
21. Oops, Your Test Failed android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2}
22. Ahh, I See Why android.support.test.espresso.NoMatchingViewException: No views in hierarchy found matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false,
23. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
24. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
25. Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } });
26. Screenshot on FAILURE Example Report
27. 1. See What is Being Tested 2. Diagnose Failures 3. Share Your Tests
28. NO ONE Sees The Value of Your Tests… YET • Your Boss • Product Owners • Designers • Manual Testers
29. That’s because they can’t tell what your tests are doing and don’t find any use for them. To them it’s just console output.
30. Let EVERYONE see what’s being tested.
31. User Interfaces are VISUAL • A User Interface is something very hard to represent in a non-visual way. • Exercise: ◦ Imagine you need to tell someone who is blind what the Mona Lisa painting looks like. What would you say? https://en.wikipedia.org/wiki/Mona_Lisa
32. Same Test, But Looks Different on Different Devices
35. Android Embraces Differences
36. LANGUAGE DEVICE TYPE DEVICE SIZE ANDROID OS VERSION Fragmentation and Permutations
37. Test Runners and HTML Reports • Run Instrumentation Commands • Collect Logs • Collect Screenshots • Generate HTML Reports
38. Spoon - http://square.github.io/spoon/
39. Composer - https://github.com/gojuno/composer
40. Spoon Test Runner java -jar spoon-runner-1.7.0-jar-with-dependencies.jar \ --apk example-app.apk \ --test-apk example-tests.apk • Run instrumentation tests in parallel • Shard tests across multiple devices • Beautiful HTML reports
41. Screenshots - HOW?
42. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon
43. Screenshot Libraries - Overflow Menus Comparison UiAutomator Falcon Spoon
44. Falcon & (Falcon Spoon Compat) FalconSpoon.screenshot(activity, tag); FalconSpoon.screenshot(activity, tag, testClassName, testMethodName); https://github.com/jraska/Falcon/
45. UiAutomator File outputFile = Spoon.screenshot(activity, tag, testClass, testMethod); uiDevice.takeScreenshot(outputFile); • Works inside and outside of your app • API 18+ (Jelly Bean MR2) https://developer.android.com/topic/libraries/testing-support-library/in dex.html#UIAutomator
46. Other Alternatives • Firebase Test Lab’s “Screenshotter” ◦ https://firebase.google.com/docs/test-lab/test-screenshots • Fastlane’s “Screengrab” ◦ https://github.com/fastlane/fastlane/tree/master/screengrab
47. Typical Espresso
48. Typical Espresso Test @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }
49. Typical Espresso Test With Screenshots @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); takeScreenshot("entered_username"); takeScreenshot("clicking_login"); onView(withText("LOGIN")).perform(click()); takeScreenshot("homescreen_shown"); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } }
50. “Screenshots for Free” via Robots
51. Robot Testing Pattern View Presenter Model Test What Test Test Test Test Test Test
52. Robot Testing Pattern View Presenter Model Test What Test Test Test Test Test Test X X X X X X X View Changed
53. Robot Testing Pattern View Presenter Model What How
54. Robot Testing Pattern Test View Presenter Model What & How What How
55. Robot Testing Pattern Robot View Presenter Model How What How Test What
56. UI Test Architecture Robot Testing Pattern Robot View Presenter Model Test
57. Robot Testing Pattern View Presenter Model Test What Test Test Test Test Test Test Robot
58. Robot Testing Pattern View Presenter Model Test What Test Test Test Test Test Test Robot X View Changed
59. Espresso Test With a Robot @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ new LoginRobot().username("sam").login().assertHomeScreenShown(); } }
60. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
61. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
62. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
63. Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
64. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
65. When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT • Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; }
66. Screenshots - At What Cost?
67. Execution Time in Minutes
68. Report Size in MB
69. When Should I Take Screenshots? • Well, it depends… but ALWAYS on FAILURE • To generate reports for product and design teams • On small/low-res emulators with mdpi screen density for smaller screenshots.
70. Use Screenshots Wisely • Programmatically configurable • Change config without recompiling ◦ Leverage Instrumentation Args ▪ --e screenshots=true ▪ --e screenshots=false
71. “Easy Screenshots” Coming Soon...
72. Easy Screenshots • Screenshots on Failure for FREE • Settings: ALL, FAILURE_ONLY or NONE ◦ Can be changed: ▪ Programmatically ▪ Via instrumentation args • Configurable ◦ Reporting Framework ◦ Screenshot Implementation
73. 1,000 Words Sam Edwards @HandstandSam
75. Visualize What Your Tests are Doing
76. Tips and Tricks
77. “Can’t write to sdcard” error • Use @GrantPermissionRule from latest Android Test Support Library • Declare in app’s Manifest: ◦ READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE
78. Leverage Continuous Integration • Jenkins • Circle CI • BuddyBuild • etc
79. Capital One Wallet
80. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); }
81. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); }
82. Espresso Example @Test public void testLogin() { onView(withText("LOGIN")).check(matches(not(isEnabled()))); onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); }
83. Target Audience for Espresso “Espresso is targeted at developers, who believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test.”
84. Learn More About Espresso
85. Screenshot Libraries - Dialog Comparison UiAutomator Falcon Spoon
86. Execution Time Overhead in Minutes 384x640 mdpi Nexus 5 Nexus 6P Falcon +16.53% +62.31% +114.39% UiAutomator +28.93% +116.15% +168.87%
87. Report Size Overhead in MB 384x640 mdpi Nexus 5 Nexus 6P Falcon +17.35% +66.41% +90.13% UiAutomator +18.41% +64.77% +89.19%
88. Get Currently RESUMED Activity private static Activity getCurrentActivityInstance() { final Activity[] activity = new Activity[1]; getInstrumentation().runOnMainSync(new Runnable() { public void run() { Collection resumedActivities = ActivityLifecycleMonitorRegistry.getInstance() .getActivitiesInStage(RESUMED); if (resumedActivities.iterator().hasNext()) { Activity currentActivity = (Activity) resumedActivities.iterator().next(); activity[0] = currentActivity; logger.trace("Activity obtained of type: " + currentActivity.getClass().getName()); } } }); return activity[0]; }
89. Get Test Method & Class from Current Stack Trace public static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }
90. Get Test Method & Class from Current Stack Trace public static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; }
91. Resulting Screenshot Code screenshot(activity, “username_entered”, testClass, testMethod); takeScreenshot(“username_entered”);
92. Pixel by Pixel Visual Regression Testing
93. Deterministic View Screenshot Comparison
94. Screenshot-tests-for-android by Facebook public class MyTests { @Test public void doScreenshot() { View view = mLayoutInflater.inflate(R.layout.my_layout, null, false); ViewHelpers.setupView(view) .setExactWidthDp(300) .layout(); Screenshot.snap(view) .record(); } }
95. Robot Testing Pattern View Presenter Model What How
96. Application Architecture Robot Testing Pattern View Presenter Model
97. Application Architecture Robot Testing Pattern View Presenter Model Test
98. Android SDK Hierarchy Viewer
99. How do I get started?
100. Espresso Test Recorder in Android Studio 2.2+ https://developer.android.com/studio/test/espresso-test-recorder.html
101. Espresso Test Recorder in Android Studio 2.2+
102. Robot Testing Pattern • A Test Architecture Pattern for Maintainability • Learned about it from Jake Wharton’s talk: ◦ VIDEO: https://realm.io/news/kau-jake-wharton-testing-robots/ ◦ SLIDES: https://speakerdeck.com/jakewharton/testing-robots-kotlin-night-may-2016
103. Create a Custom Test Runner • Control test runner behavior without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false
104. Define Your “LOW BAR” Device • Android Emulator Virtual Device • 384x640 • mdpi - 160 dots per inch • Able to run in continuous integration
105. Android Studio - Layout Inspector
106. Android Studio - Layout Inspector
107. Test Butler by LinkedIn “Reliable Android testing, at your service.” • Stabilizes the Android emulator ◦ Disables animations ◦ Disables crash & ANR dialogs ◦ Locks the keyguard, WiFi radio, and CPU to ensure they don't go to sleep unexpectedly while tests are running. • Handles changing global emulator settings and holds relevant permissions so your app doesn't have to. ◦ Enable/disable WiFi ◦ Change device orientation ◦ Set location services mode ◦ Set application locale https://github.com/linkedin/test-butler
108. Create a Custom Test Runner • Control test runner behavior without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false
109. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }
110. Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner { @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" }
111. GIFs • Generated by default • Processing overhead ◦ 11 Seconds for 21 Screenshots -> • Disable using “--no-animations”
112. To Add Android Test Orchestrator Composer Reporting @GrantPermissionRule