Zero to Hero in UI testing

In this talk we'll discuss the problems around UI testing - why is it hard to write such tests, how to get started, how to make them part of your CI system. We'll present a recipe for building a scalable framework that makes writing UI tests a pleasure. Most of the common "gotcha-s" will be solved along the way, as well as giving a few best practices. Once the foundations are laid out, a whole new set of scaling problems arises, so we'll talk about possible solutions for them.

Whether you still don't have any UI tests in your project, or you already have a good suite built, this talk will give you some useful tips & ideas.

=====
Presented on 26th Sept as part of Londroid @ ASOS

Transcript

1. #Londroid ZERO TO HERO IN UI TESTING
 (OR HOW TO CREATE A SCALABLE TEST SUITE) VESSELIN ILIEV, ANDROID DEVELOPER @ ASOS
2. @ilievVesselin #Londroid WHO WE ARE? 11M+ 4.8 AVERAGE RATING
 on 200,000+ Customer Reviews ACTIVE INSTALLS MONTHLY AVERAGE FREQUENCY 80
 mins 8x MONTHLY AVERAGE TIME SPENT
3. @ilievVesselin #Londroid WHAT IS UI TESTING?
4. @ilievVesselin #Londroid WHY SHOULD YOU CARE ABOUT UI TESTS? source [1]
5. @ilievVesselin #Londroid WHY SHOULD YOU CARE ABOUT UI TESTS? Less meaningful
 for the user Very meaningful
 for the user source [2]
6. @ilievVesselin #Londroid WHY ARE WE NOT WRITING UI TESTS? • NO TIME source [3]
7. @ilievVesselin #Londroid WHY ARE WE NOT WRITING UI TESTS? Cost / Effort Time Manual Automated • NO TIME • TOO MUCH EFFORT TO GET STARTED
8. @ilievVesselin #Londroid WHY ARE WE NOT WRITING UI TESTS? • NO TIME • TESTS ARE UNSTABLE (FLAKY) • TOO MUCH EFFORT TO GET STARTED • THIS IS QAS JOB
9. @ilievVesselin #Londroid WRITING YOUR FIRST UI TEST
10. @ilievVesselin #Londroid WRITING YOUR FIRST UI TEST dependencies { … androidTestCompile 'com.android.support.test.espresso:espresso-core:x.x.x' } defaultConfig { testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner" } build.gradle
11. @ilievVesselin #Londroid WRITING YOUR FIRST UI TEST @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class);
 
 @Test
 public void willOpenMainAppScreen() throws Exception { onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed())); } } (for more info on Espresso see [4])
12. @ilievVesselin #Londroid WRITING YOUR FIRST UI TEST
13. @ilievVesselin #Londroid MOCKING EXTERNAL DEPENDENCIES • WHY MOCK? • TEST YOUR CODE, NOT YOUR DEPENDENCIES • MAKE TESTS PREDICTABLE • SAME PRINCIPLES AS UNIT TESTING APPLY • WHAT TO MOCK? • NETWORK LAYER • SYSTEM-PROVIDED RESOURCES (LOCATION, SENSOR DATA, CONNECTION INFO) • EVERYTHING YOU DON’T ACTUALLY CONTROL
14. @ilievVesselin #Londroid MOCKING THE NETWORK LAYER • STUB CLASSES public interface AppInitRestApiService {
 
 @GET("init")
 Single<ConfigModel> initializeApp();
 } public class FakeInitRestApiService implements AppInitRestApiService {
 
 @Override
 public Single<ConfigModel> initializeApp() {
 return Single.just(DUMMY_CONFIG_MODEL);
 }
 }
15. @ilievVesselin #Londroid MOCKING THE NETWORK LAYER • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION public class MockResponseInterceptor implements Interceptor {
 
 private final Map<String, Response> responsesMap = new HashMap<>();
 
 @Override
 public Response intercept(@NonNull Chain chain) throws IOException {
 …
 }
 
 public void defineResponse(String path, Response response) {
 responsesMap.put(path, response);
 }
 }
16. @ilievVesselin #Londroid MOCKING THE NETWORK LAYER • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION • OKREPLAY [5] @Test @OkReplay
 public void someTest() throws Exception {
 // some network operations ... }
17. @ilievVesselin #Londroid MOCKING THE NETWORK LAYER • STUB CLASSES • OKHTTP INTERCEPTOR • CUSTOM SOLUTION • OKREPLAY [5] • PACKAGE A MOCK WEBSERVER IN THE APP • MOCKWEBSERVER [6] • RESTMOCK [7] • WIREMOCK [8]
18. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
19. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
20. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 } public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false); private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher); server.start(PORT);
 }
 
 ... 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
21. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
22. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
23. @ilievVesselin #Londroid MOCKWEBSERVER EXAMPLE ... 
 @Test
 public void willOpenMainAppScreen() throws Exception { 
 
 }
 
 ... dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); 
 private MockResponse appInitResponse() {
 MockResponse response = new MockResponse();
 response.setResponseCode(HttpURLConnection.HTTP_OK);
 response.setBody(TestServerResponses.APP_INIT_RESPONSE);
 return response;
 } activityTestRule.launchActivity(null); onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed()));
24. @ilievVesselin #Londroid MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
25. @ilievVesselin #Londroid MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
26. @ilievVesselin #Londroid MOCKDISPATCHER EXAMPLE private final Map<String, MockResponse> responsesMap = new HashMap<>(); public class MockDispatcher extends Dispatcher { } @Override
 public MockResponse dispatch(RecordedRequest recordedRequest) throws InterruptedException {
 MockResponse response = responsesMap.get(recordedRequest.getPath());
 if (response != null) {
 return response;
 }
 
 return new MockResponse()
 .setBody("No responses found for the request: " + recordedRequest)
 .setResponseCode(404);
 } 
 public void add(String requestPath, MockResponse response) {
 responsesMap.put(requestPath, response);
 }
27. @ilievVesselin #Londroid SO FAR @RunWith(AndroidJUnit4.class)
 public class AppStartTest {
 
 @Rule
 public ActivityTestRule<SplashActivity> activityTestRule = new ActivityTestRule<>(SplashActivity.class, true, false);
 
 private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); 
 @Before
 public void setUp() throws Exception {
 server.setDispatcher(dispatcher);
 server.start(PORT);
 }
 
 @Test
 public void willOpenMainAppScreen() throws Exception {
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse());
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 
 activityTestRule.launchActivity(null);
 
 onView(withId(R.id. home_screen_view_pager)).check(matches(isDisplayed()));
 }
 
 private MockResponse appInitResponse() {
 ...
 }
 
 private MockResponse appDataResponse() {
 ...
 }
 
 @After
 public void tearDown() throws Exception {
 server.shutdown();
 }
 }
28. @ilievVesselin #Londroid KEEPING THE TEST CODE CLEAN • WHAT? VS HOW?
29. @ilievVesselin #Londroid KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? @Test
 public void willOpenMainAppScreen() throws Exception { … } WHAT?
30. @ilievVesselin #Londroid KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? … private MockWebServer server = new MockWebServer(); private MockDispatcher dispatcher = new MockDispatcher(); …
 
 @Test
 public void willOpenMainAppScreen() throws Exception {
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse());
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 
 activityTestRule.launchActivity(null);
 
 onView(withId(R.id. home_screen_view_pager)).check(matches(isDisplayed()));
 } … HOW?
31. @ilievVesselin #Londroid KEEPING THE TEST CODE CLEAN • WHAT? VS HOW? • EXPRESS BEHAVIOR IN THE TESTS
32. @ilievVesselin #Londroid CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) @Test
 public void willOpenMainAppScreen() throws Exception {
 given.server.respondsWithSuccessWhenStartingTheApp();
 
 when.user.launchesTheApp();
 
 then.user.sees.homeScreen();
 }
33. @ilievVesselin #Londroid CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) class ServerArrangements { … public void respondsWithSuccessWhenStartingTheApp() {
 dispatcher.add(REQUEST_PATH_INIT_APP, appInitResponse());
 dispatcher.add(REQUEST_PATH_FETCH_APP_DATA, appDataResponse()); } } class UserActions { … public void launchesTheApp() {
 activityRule.launchActivity(null);
 } } class Verifications { … public void homeScreen() {
 onView(withId(R.id.home_screen_view_pager)).check(matches(isDisplayed())); } }
34. @ilievVesselin #Londroid CREATE YOUR OWN DSL • CREATE YOUR OWN SYNTAX (JUST A FEW CLASSES) • SQUARE’S ROBOTS PATTERN [9] • SHAZAM’S GWEN [10]
35. @ilievVesselin #Londroid BUILD YOUR SUITE @Test
 public void willSeeAllAvailablePaymentMethodsExceptTheseAlreadyInWallet() {
 given.paymentsApiServer.returnsAllPaymentMethodsUnrestricted();
 given.walletApiServer.returnsCardAndKlarnaAsDefault();
 given.user.hasLaunchedCheckout();
 
 when.user.wantsToChangePaymentMethod();
 then.user.sees.walletScreenFromCheckout();
 
 when.user.isOnWalletScreen.andSelectsAddNewPaymentMethod();
 then.user.sees.addPaymentMethodScreen()
 .withCardCTA()
 .withPayPalCTA()
 .withIdealCTA()
 .withSofortCTA()
 .withoutKlarnaCTA();
 }
36. @ilievVesselin #Londroid BUILD YOUR SUITE
37. @ilievVesselin #Londroid RUNNING YOUR TESTS LOCALLY • ANDROIDJUNITRUNNER [11] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN
38. @ilievVesselin #Londroid RUNNING YOUR TESTS LOCALLY • ANDROIDJUNITRUNNER [11] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN • SPOON [12] • RUN TESTS ON MULTIPLE DEVICES
39. @ilievVesselin #Londroid RUNNING YOUR TESTS LOCALLY • ANDROIDJUNITRUNNER [11] • RUN JUST A TEST, CLASS, PACKAGE, ANNOTATION, FILE DEFINITION, EXCLUSIONS • SHARD TEST RUN • SPOON [12] • RUN TESTS ON MULTIPLE DEVICES • FORK [13] • TEST SHARDING BY DEFAULT
40. @ilievVesselin #Londroid RUNNING YOUR TESTS IN THE CLOUD • CLOUD SOLUTIONS • FIREBASE TEST LAB, AWS DEVICE FARM, XAMARIN TEST CLOUD, SAUCELABS TESTOBJECT, KOBITON, PERFECTO, ETC … • BIG RANGE OF AVAILABLE DEVICES • NO TEST SHARDING • GREAT FOR ”NIGHTLY” BUILDS • RUN “LOCALLY” WITH CLOUD DEVICES • GENYMOTION CLOUD [14] • CUSTOM RULE SHARDING • FLANK [15]
41. @ilievVesselin #Londroid COMMON SOURCES OF FLAKINESS • ANIMATIONS (TESTBUTLER [16] )
42. @ilievVesselin #Londroid COMMON SOURCES OF FLAKINESS • ANIMATIONS (TESTBUTLER [16] ) • DIFFERENT DEVICES, OS VERSIONS, MANUFACTURERS (CI ONLY) • SYSTEM POPUPS (UPDATES, WARNINGS, RANDOM SAMSUNG NOTIFICATIONS) • LOSS OF NETWORK (E.G. “NO INTERNET” POPUPS) • SYNCHRONIZING IO OPERATIONS • RXRULE [17] & RXIDLER [18] • VOLLEYRULE [19] • ASSERTING SHORT-LIVED NOTIFICATIONS (TOASTS, SNACKBARS) • HANDLERS & POSTDELAYED() • HAVING A LOADING PROGRESS BAR [20] • CLICK VS LONGCLICK
43. @ilievVesselin #Londroid SCALING ISSUES • RUNNING TIME • DIFFERENTIATE BETWEEN CI & NIGHTLY • SMALL TEST SUITES ARE USEFUL • FLAKINESS • PRODUCE APK EVEN IF A TEST FAILS • FLAKINESS MONITOR
44. @ilievVesselin #Londroid BENEFITS OF A GOOD TEST FRAMEWORK • RAPIDLY INCREASING COVERAGE • EASY ONBOARDING OF NEW TEAM MEMBERS • MAKE BIG CHANGES EASIER • ROI INCREASES SIGNIFICANTLY OVER TIME
45. @ilievVesselin #Londroid Happy UI testing! … THANKS!
46. @ilievVesselin #Londroid REFERENCES 1. https://twitter.com/codejobs/status/813056723271450624 2. Testing pyramid 3. https://twitter.com/qcoding/status/578395218078244864 4. Espresso documentation 5. OkReplay library 6. MockWebServer library 7. RESTMock library 8. WireMock library • Running WireMock on Android 9. Robot pattern explanation video 10. GWEN library 11. AndroidJUnitRunner 12. Spoon library 13. Fork library 14. Test sharding on Genymotion cloud devices 15. Flank - experimental
47. @ilievVesselin #Londroid REFERENCES 16. Open-sourcing TestButler 17. Synchronise RxJava and Espresso 18. RxIdler 19. Synchronise Volley and Espresso 20. Testing a ProgressBar with Espresso • Interesting discussion • A (dirty) solution • Another (dirty) solution