DroidconSF 2017 - UI Testing

Video: https://www.youtube.com/watch?v=SkkO6x6LhCQ

How hard can it be to chain together a few Espresso commands and produce an Android UI test? There's even a test recorder built into the IDE for this purpose! It is easy... Unless, of course, one actually wants to run the test continuously and have it block merges to master. Now, we need not just a test, but a test that provides a reliable signal; a test that is not flaky. At Slack, we started by solving the problem of writing reliable UI tests, by making them targeted and hermetic. And we made good progress. However, as our UI test suite grew to hundreds of tests, we discovered a problem - developers hated the process. It was cumbersome and time-consuming. It turns out, the difficult part of writing the test was not about testing at all - it was about setting up application state. In this talk, we will present our approach of simplifying our test setup to the extent that makes it easy and, dare we say it - enjoyable - to develop reliable UI tests.

Transcript

1. UI Tests are Easy to Write said no Android dev ever Kevin Lai & Valera Zakharov
2. Why?
3. Aug 2016 Nov 2017
4. 1. No time 2. Too hard to develop 3. Not enough knowledge
10. Any test is good... … until you start running it in CI
13. API for Test Authors onView(withId(R.id.greeting)) .perform(click()); onView(withText("Hello Droidcon SF!")) .check(matches(isDisplayed()));
25. https://academy.realm.io/posts/writing-solid-ui-tests-on-android-mdevcamp-zakharov
26. UI tests can still be Targeted & Hermetic
29. Today
30. At Slack
31. Android Test Pyramid at Slack 100ish e2e 555 functional UI 3448 unit
32. Developer Feedback Workflow Ease of Testing App Quality Release Process “Writing quality UI tests is still a difficult thing to do. It would be great to have some time put into building up a base to work from” 2.5/4
33. http://jakewharton.com/testing-robots
34. What How How What Presenter / Components Model View Robot Test UI Test Architecture
35. I’m going on a UI test adventure
36. The Problem
37. What How How What Presenter / Components Model View Robot Test Test
38. Lots of Components Data Providers - fetch and cache Persistence - cache data Managers - app-wide process or state (e.g. message sending, channel syncing) Helpers - data massaging API Actions - REST and websocket requests Presenters? Great!
39. Repetitive Mocking - User User user = User.builder().../* setters */...build(); when(mockUsersDataProvider.getUser(USER_ID)) .thenReturn(Observable.just(user)); when(mockUserPresenceManager.isUserActive(USER_ID, true)) .thenReturn(true); when(mockDndInfoDataProvider.getDndInfo(userId)) .thenReturn(Observable.just(new DndInfo(/* params */)));
40. Repetitive Mocking - Channel Channel channel = Channel.builder().../* setters */...build(); when(mockChannelsDataProvider.getChannelObservable(CHANNEL_ID)) .thenReturn(Observable.just(channel)); when(mockChannelNameHelper.getDisplayNameObservable(eq(channel))) .thenReturn(Observable.just(CHANNEL_NAME));
41. @Before @Override public void setUp() throws Exception { super.setUp(); User user = User.builder().../* setters */...build(); when(mockUsersDataProvider.getUser(USER_ID)) .thenReturn(Observable.just(user)); when(mockUserPresenceManager.isUserActive(USER_ID, true)) .thenReturn(true); when(mockDndInfoDataProvider.getDndInfo(userId)) .thenReturn(Observable.just(new DndInfo(/* params */))); Channel channel = Channel.builder().../* setters */...build(); when(mockChannelsDataProvider.getMessagingChannelObservable(CHANNEL_ID)) .thenReturn(Observable.just(channel)); when(mockChannelNameHelper.getDisplayNameObservable(eq(channel))) .thenReturn(Observable.just(CHANNEL_NAME)); launchActivity(); }
42. The Solution
44. What How How What Presenter / Components Model View Robot Test Test
45. How How What Robot Test View Model Presenter / Components What
46. Power Pack
48. How Robot Test View How PowerPack What Presenter / Components Model How What
49. Set up Models not Components slackPowerPack.usersPack() .addUser(User.builder().../* setters */...build()) .addUser(User.builder().../* setters */...build()) .pack(); slackPowerPack.channelsPack() .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .pack();
50. Under the hood public UsersPack.Builder usersPack() { return UsersPack.builder() .dndInfoDataProvider(mockDndInfoDataProvider) .persistentStore(mockPersistentStore) .usersDataProvider(mockUsersDataProvider) .userPresenceManager(mockUserPresenceManager); }
51. public class UsersPack { public void pack() { UsersPack usersPack = build(); UsersDataProvider usersDataProvider = usersPack.usersDataProvider(); UserPresenceManager userPresenceManager = usersPack.userPresenceManager(); DndInfoDataProvider dndInfoDataProvider = usersPack.dndInfoDataProvider(); // More components... List<User> users = usersPack.users(); for (User user : users) { when(usersDataProvider.getUser(user.id())) .thenReturn(Observable.just(user)); when(userPresenceManager.isUserActive(user.id(), true)) .thenReturn(true); when(dndInfoDataProvider.getDndInfo(user.id())) .thenReturn(Observable.just(new DndInfo(/* params */))); // More mocking... } } }
52. Under the hood public ChannelsPack.Builder channelsPack() { return ChannelsPack.builder() .accountManager(mockAccountManager) .featureFlagStore(mockFeatureFlagStore) .lastOpenedMsgChannelIdStore(mockLastOpenedMsgChannelIdStore) .mpdmDisplayNameHelper(mockMpdmDisplayNameHelper) .persistentStore(mockPersistentStore) .prefsManager(mockPrefsManager) .slackApi(mockSlackApi) .channelListDataProvider(mockChannelListDataProvider) .channelNameHelper(mockChannelNameHelper) .channelsPaneDataProvider(mockChannelsPaneDataProvider) .channelsDataProvider(mockChannelsDataProvider) .messageCountManager(mockMessageCountManager) .notificationPrefsDataProvider(mockNotificationPrefsDataProvider) .notificationPrefsManager(mockNotificationPrefsManager) .userChannelListDataProvider(mockUserChannelListDataProvider);
53. @Before @Override public void setUp() throws Exception { super.setUp(); slackPowerPack = new SlackPowerPack(context.getApplicationContext()); slackPowerPack.usersPack() .addUser(User.builder().../* setters */...build()) .addUser(User.builder().../* setters */...build()) .pack(); slackPowerPack.channelsPack() .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .pack(); launchActivity(); }
54. Wins • *Pack classes encapsulate all mockito statements • Tests benefit from updates in one place • Test authors focus on writing their actual test • Fluent APIs
55. Developer Feedback Workflow Ease of Testing App Quality Release Process “Ease of test writing: this has improved a lot. It would be nice to take the time to update some of our old tests to use PowerPack/our latest patterns." 2.5/4 2.92/4 “I would also love an easy way to set up a team.”
56. Focus on the pattern
58. The Solution ++
59. @Before @Override public void setUp() throws Exception { super.setUp(); User user = User.builder().../* setters */...build(); when(mockUsersDataProvider.getUser(USER_ID)) .thenReturn(Observable.just(user)); when(mockUserPresenceManager.isUserActive(USER_ID, true)) .thenReturn(true); when(mockDndInfoDataProvider.getDndInfo(userId)) .thenReturn(Observable.just(new DndInfo(/* params */))); Channel channel = Channel.builder().../* setters */...build(); when(mockChannelsDataProvider.getMessagingChannelObservable(CHANNEL_ID)) .thenReturn(Observable.just(channel)); when(mockChannelNameHelper.getDisplayNameObservable(eq(channel))) .thenReturn(Observable.just(CHANNEL_NAME)); launchActivity(); }
60. @Before @Override public void setUp() throws Exception { super.setUp(); slackPowerPack = new SlackPowerPack(context.getApplicationContext()); slackPowerPack.usersBuilder() .addUser(User.builder().../* setters */...build()) .addUser(User.builder().../* setters */...build()) .pack(); slackPowerPack.channelsBuilder() .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .addChannel(Channel.builder().../* setters */...build()) .pack(); launchActivity(); }
61. @Before @Override public void setUp() throws Exception { super.setUp(); slackPowerPack = new SlackPowerPack(context.getApplicationContext(), "team_a_config.json"); launchActivity(); }
62. JSON Config { “logged_in_user”: String “team”: { } “users”: [ ], “channels”: [ ], “history”: [ { “channel_id”: String “messages”: [ ] } ] }
63. public class WorkspaceConfig { private String logged_in_user; private Team team; private List<Channel> channels; private List<DM> dms; private List<User> users; private List<History> history; }
65. { "ok": true, "channels": [ { "id": "C0G9QF9GW", "name": "random", "is_channel": true, "created": 1449709280, "creator": "U0G9QF9C6", "is_archived": false, "is_general": false, "name_normalized": "random", "is_shared": false, "is_org_shared": false, "is_member": true, "is_private": false, "is_mpim": false, "members": [ "U0G9QF9C6", "U0G9WFXNZ" ], "topic": { "value": "Other stuff", "creator": "U0G9QF9C6", "last_set": 1449709352 }, "previous_names": [], "num_members": 2 }, JSON Config { “logged_in_user”: String “team”: { } “users”: [ ], “channels”: [ ], “history”: [ { “channel_id”: String “messages”: [ ] } ] }
66. Charging the Power Pack E2E Test record API Data Replay Test (e.g. Perf) Real Backend Hermetic Functional Test profit replay generate fetch
67. @valera_zakharov @kevin_lai14