Zero to Hero in UI testing

In this talk we'll discuss the problems around UI testing - why is it hard to write such tests, how to get started, how to make them part of your CI system. We'll present a recipe for building a scalable framework that makes writing UI tests a pleasure. Most of the common "gotcha-s" will be solved along the way, as well as giving a few best practices. Once the foundations are laid out, a whole new set of scaling problems arises, so we'll talk about possible solutions for them.

Whether you still don't have any UI tests in your project, or you already have a good suite built, this talk will give you some useful tips & ideas.

=====
Presented on 21st Oct as part of Mobilization 2017

Transcript

1. Europe 2017 ZERO TO HERO IN UI TESTING VESSELIN ILIEV, ANDROID DEVELOPER @ ASOS (OR HOW TO CREATE A SCALABLE TEST SUITE)
2. @ilievVesselin Europe 2017 WHAT IS UI TESTING?
3. @ilievVesselin Europe 2017 WHY SHOULD YOU CARE ABOUT UI TESTS? source [1]
4. @ilievVesselin Europe 2017 WHY SHOULD YOU CARE ABOUT UI TESTS? source [2] Less meaningful
 for the user Very meaningful
 for the user
5. @ilievVesselin Europe 2017 WHY ARE WE NOT WRITING UI TESTS? • NO TIME source [3]
6. @ilievVesselin Europe 2017 WHY ARE WE NOT WRITING UI TESTS? • NO TIME Cost / Effort Time Manual Automated • TOO MUCH EFFORT TO GET STARTED
7. @ilievVesselin Europe 2017 WHY ARE WE NOT WRITING UI TESTS? • NO TIME • TOO MUCH EFFORT TO GET STARTED • THIS IS QAS JOB
8. @ilievVesselin Europe 2017 WHY ARE WE NOT WRITING UI TESTS? • NO TIME • TESTS ARE UNSTABLE (FLAKY) • TOO MUCH EFFORT TO GET STARTED • THIS IS QAS JOB
9. @ilievVesselin Europe 2017 WRITING YOUR FIRST UI TEST
10. @ilievVesselin Europe 2017 WRITING YOUR FIRST UI TEST dependencies { … androidTestImplementation 'com.android.support.test.espresso:espresso-core:x.x.x' } defaultConfig { testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner" } build.gradle
11. @ilievVesselin Europe 2017 WRITING YOUR FIRST UI TEST @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class);
 
 @Test
 public void willOpenMainAppScreen() throws Exception { onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed())); } } (for more info on Espresso see [4])
12. @ilievVesselin Europe 2017 WRITING YOUR FIRST UI TEST @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class);
 
 @Test
 public void willOpenMainAppScreen() throws Exception { onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed())); } } (for more info on Espresso see [4])
13. @ilievVesselin Europe 2017 • WHY MOCK? • TEST YOUR CODE, NOT YOUR DEPENDENCIES • MAKE TESTS PREDICTABLE • SAME PRINCIPLES AS UNIT TESTING APPLY
 • WHAT TO MOCK? • NETWORK LAYER • SYSTEM-PROVIDED RESOURCES (LOCATION, SENSOR DATA, CONNECTION INFO) • EVERYTHING YOU DON’T CONTROL MOCKING EXTERNAL DEPENDENCIES
14. @ilievVesselin Europe 2017 • STUB CLASSES MOCKING THE NETWORK LAYER public interface AppInitRestApiService {
 
 @GET("init")
 Single<ConfigModel> initializeApp();
 } public class FakeInitRestApiService implements AppInitRestApiService {
 
 @Override
 public Single<ConfigModel> initializeApp() {
 return Single.just(DUMMY_CONFIG_MODEL);
 }
 }
15. @ilievVesselin Europe 2017 • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION MOCKING THE NETWORK LAYER public class MockResponseInterceptor implements Interceptor {
 
 private final Map<String, Response> responsesMap = new HashMap<>();
 
 @Override
 public Response intercept(@NonNull Chain chain) throws IOException {
 …
 }
 
 public void defineResponse(String path, Response response) {
 responsesMap.put(path, response);
 }
 }
16. @ilievVesselin Europe 2017 • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION • OKREPLAY [5] MOCKING THE NETWORK LAYER @Test @OkReplay
 public void someTest() throws Exception {
 // some network operations ... }
17. @ilievVesselin Europe 2017 • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION • OKREPLAY [5] • PACKAGE A MOCK WEBSERVER IN THE APP • MOCKWEBSERVER [6] • RESTMOCK [7] • WIREMOCK [8] MOCKING THE NETWORK LAYER
18. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
19. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
20. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
21. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
22. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
23. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
24. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
25. @ilievVesselin Europe 2017 MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
26. @ilievVesselin Europe 2017 MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
27. @ilievVesselin Europe 2017 MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
28. @ilievVesselin Europe 2017 MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
29. @ilievVesselin Europe 2017 MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
30. @ilievVesselin Europe 2017 SO FAR @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false);
 
 private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); 
 @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher);
 server.start(PORT);
 }
 
 @Test
 public void willOpenMainAppScreen() throws Exception {
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse());
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 
 activityTestRule.launchActivity(null);
 
 onView(withId(R.id. home_screen_view_pager)).check(matches(isDisplayed()));
 }
 
 private MockResponse appInitResponse() {
 ...
 }
 
 private MockResponse appDataResponse() {
 ...
 }
 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
 }
31. @ilievVesselin Europe 2017 KEEPING THE TEST CODE CLEAN • WHAT? VS HOW?
32. @ilievVesselin Europe 2017 KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? @Test
 public void willOpenMainAppScreen() throws Exception { … } WHAT?
33. @ilievVesselin Europe 2017 KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? … private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); …
 
 @Test
 public void willOpenMainAppScreen() throws Exception {
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse());
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 
 activityTestRule.launchActivity(null);
 
 onView(withId(R.id. home_screen_view_pager)).check(matches(isDisplayed()));
 } … HOW?
34. @ilievVesselin Europe 2017 KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? • EXPRESS BEHAVIOR IN THE TESTS
35. @ilievVesselin Europe 2017 CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) @Test
 public void willOpenMainAppScreen() throws Exception {
 given.server.respondsWithSuccessWhenStartingTheApp();
 
 when.user.launchesTheApp();
 
 then.user.sees.homeScreen();
 }
36. @ilievVesselin Europe 2017 CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) class ServerArrangements { … public void respondsWithSuccessWhenStartingTheApp() {
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); } } class UserActions { … public void launchesTheApp() {
 activityRule.launchActivity(null);
 } } class Verifications { … public void homeScreen() {
 onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed())); } }
37. @ilievVesselin Europe 2017 CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) • SQUARE’S ROBOTS PATTERN [9] • PAGE OBJECT PATTERN IN DETAILS [10] • GWEN LIBRARY [11] • KAKAO LIBRARY [12]
38. @ilievVesselin Europe 2017 BUILD YOUR SUITE @Test
 public void willSeeAllAvailablePaymentMethodsExceptTheseAlreadyInWallet() {
 given.paymentsApiServer.returnsAllPaymentMethodsUnrestricted();
 given.walletApiServer.returnsCardAndKlarnaAsDefault();
 given.user.hasLaunchedCheckout();
 
 when.user.wantsToChangePaymentMethod();
 then.user.sees.walletScreenFromCheckout();
 
 when.user.isOnWalletScreen.andSelectsAddNewPaymentMethod();
 then.user.sees.addPaymentMethodScreen()
 .withCardCTA()
 .withPayPalCTA()
 .withIdealCTA()
 .withSofortCTA()
 .withoutKlarnaCTA();
 }
39. @ilievVesselin Europe 2017 BUILD YOUR SUITE
40. @ilievVesselin Europe 2017 RUN TESTS LOCALLY (CI) • ANDROIDJUNITRUNNER [13] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN
41. @ilievVesselin Europe 2017 RUN TESTS LOCALLY (CI) • ANDROIDJUNITRUNNER [13] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN • SPOON [14] • RUN TESTS ON MULTIPLE DEVICES
42. @ilievVesselin Europe 2017 RUN TESTS LOCALLY (CI) • ANDROIDJUNITRUNNER [13] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN • SPOON [14] • RUN TESTS ON MULTIPLE DEVICES • FORK [15] • TEST SHARDING BY DEFAULT
43. @ilievVesselin Europe 2017 • CLOUD SOLUTIONS • FIREBASE TEST LAB, AWS DEVICE FARM, XAMARIN TEST CLOUD, SAUCELABS TESTOBJECT, KOBITON, PERFECTO, ETC … • BIG RANGE OF AVAILABLE DEVICES • NO TEST SHARDING • GREAT FOR ”NIGHTLY” BUILDS • RUN “LOCALLY” WITH CLOUD DEVICES • GENYMOTION CLOUD [16] • CUSTOM RULE SHARDING • FLANK [17] RUN TESTS IN THE CLOUD (NIGHTLY)
44. @ilievVesselin Europe 2017 • ANIMATIONS (TESTBUTLER [18] ) COMMON SOURCES OF FLAKINESS
45. @ilievVesselin Europe 2017 • ANIMATIONS (TESTBUTLER [18] ) • DIFFERENT DEVICES, OS VERSIONS, MANUFACTURERS (CI ONLY) • SYSTEM POPUPS (UPDATES, WARNINGS, RANDOM SAMSUNG NOTIFICATIONS) • LOSS OF NETWORK (E.G. “NO INTERNET” POPUPS) • SYNCHRONIZING IO OPERATIONS • RXRULE [19] & RXIDLER [20] • VOLLEYRULE [21] • ASSERTING SHORT-LIVED NOTIFICATIONS (TOASTS, SNACKBARS) • HANDLERS & POSTDELAYED() • HAVING A LOADING PROGRESS BAR [22] COMMON SOURCES OF FLAKINESS
46. @ilievVesselin Europe 2017 • ANIMATIONS (TESTBUTLER [18] ) • DIFFERENT DEVICES, OS VERSIONS, MANUFACTURERS (CI ONLY) • SYSTEM POPUPS (UPDATES, WARNINGS, RANDOM SAMSUNG NOTIFICATIONS) • LOSS OF NETWORK (E.G. “NO INTERNET” POPUPS) • SYNCHRONIZING IO OPERATIONS • RXRULE [19] & RXIDLER [20] • VOLLEYRULE [21] • ASSERTING SHORT-LIVED NOTIFICATIONS (TOASTS, SNACKBARS) • HANDLERS & POSTDELAYED() • HAVING A LOADING PROGRESS BAR [22] • CLICK VS LONGCLICK COMMON SOURCES OF FLAKINESS
47. @ilievVesselin Europe 2017 • RUNNING TIME • DIFFERENTIATE BETWEEN CI & NIGHTLY • SMALL TEST SUITES ARE USEFUL • FLAKINESS • PRODUCE APK EVEN IF A TEST FAILS • FLAKINESS MONITOR SCALING ISSUES
48. @ilievVesselin Europe 2017 • RAPIDLY INCREASING COVERAGE • EASY ONBOARDING OF NEW TEAM MEMBERS • MAKE BIG CHANGES EASIER • ROI INCREASES SIGNIFICANTLY OVER TIME BENEFITS OF A GOOD TEST FRAMEWORK
49. @ilievVesselin Europe 2017 … THANKS! Happy UI testing!
50. @ilievVesselin Europe 2017 REFERENCES 1. https://twitter.com/codejobs/status/813056723271450624 2. Testing pyramid 3. https://twitter.com/qcoding/status/578395218078244864 4. Espresso documentation 5. OkReplay library 6. MockWebServer library 7. RESTMock library 8. WireMock library • Running WireMock on Android 9. Robot pattern explanation video 10. Page Object pattern in details 11. GWEN library 12. Kakao library 13. AndroidJUnitRunner 14. Spoon library 15. Fork library
51. @ilievVesselin Europe 2017 REFERENCES 16. Test sharding on Genymotion cloud devices 17. Flank - experimental 18. Open-sourcing TestButler 19. Synchronise RxJava and Espresso 20. RxIdler 21. Synchronise Volley and Espresso 22. Testing a ProgressBar with Espresso • Interesting discussion • A (dirty) solution • Another (dirty) solution