E2E2U: Slack’s journey to developer-owned end-to-end testing

Two years ago we presented on an approach that makes authoring reliable Android UI tests easier. To date, Slack android developers have written over 700 UI tests and all of them run as part of a merge-blocking (and, most importantly, reliable) pull-request check. Having conquered this level, we set our sights on the top of our test pyramid - automated e2e tests. This test suite was still being maintained by a single hero developer, who, through sheer grit and determination, didn’t let this important part of our release process die. Armed with experience in functional UI testing, we embarked on an adventure to make our end-to-end test suite a shared responsibility of the entire team. Our journey has taken us through the fields of dagger, the bog of espresso IdlingResources, the inferno of proguard and, perhaps the most challenging of them all, cultural aspects of quality assurance. We look forward to sharing where we are in our never-ending testing story.

Transcript

1. E2E2U SLACK’S JOURNEY TO DEVELOPER-OWNED E2E TESTING @valera_zakharov
3. STABLE E2E TESTS ON ANDROID IS SUCH AT THING EVEN POSSIBLE? @valera_zakharov
4. NO
5. THANK YOU FOR COMING, ANY QUESTIONS? @valera_zakharov
9. SLACK LAUNCHED 2013 2015 2019 +JASON 2016 2017 2018
12. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 2017 2018
13. 2015 ▸ calabash (ruby) ▸ xamarin (now appcenter.ms) ▸ cross platform https://www.everwilde.com/store/Calabash-Gourd-Seeds.html E2E TESTS
14. FIRST E2E TESTS LOOKS LIKE THIS Feature: User Login Scenario: Unsuccessful user login Given the app has launched Then I wait for the "Login" button to appear When I enter "tstuser" into the "Username" field And I enter "qwerty" into the "Password" field And I touch "Login" Then I should see "Username you entered is incorrect" Scenario: Successful user login Given the app has launched Then I wait for the "Login" button to appear When I enter "testeruser" into the "Username" field And I enter "qwerty" into the "Password" field And I touch "Login" Then I should see "Hey testeruser!" https://www.toptal.com/mobile/android-ios-ui-testing-calabash
15. FIRST E2E TESTS WHAT IS AN END-TO-END TEST ▸ covers important user journeys ▸ talks to the backend https://aws.amazon.com/solutions/case-studies/slack/
16. FIRST E2E TESTS PROBLEMS ▸ blackbox = sign in every time ▸ unstable ▸ decoupled from dev workflow ▸ no synchronization with the app under test ▸ really slow http://ruffledfeathersandspilledmilk.com/wp-content/uploads/2013/11/Bella-272.jpg
17. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA ESPRESSO 2017 2018
18. TARGETED & HERMETIC ESPRESSO
20. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA ESPRESSO +KEVIN POWER PACK 2017 2018
21. 2017 public void setUp() throws Exception { super.setUp(); User user = User.builder()…/* setters */…build(); when(mockUsersDataProvider.getUser(USER_ID)) .thenReturn(Observable.just(user)); when(mockUserPresenceManager.isUserActive(USER_ID, true)) .thenReturn(true); when(mockDndInfoDataProvider.getDndInfo(userId)) .thenReturn(Observable.just(new DndInfo(/* params */))); Channel channel = Channel.builder()…/* setters */…build(); when(mockChannelsDataProvider.getMessagingChannelObservable(CHANNEL_ID)) .thenReturn(Observable.just(channel)); when(mockChannelNameHelper.getDisplayNameObservable(eq(channel))) .thenReturn(Observable.just(CHANNEL_NAME)); launchActivity(); } slackPowerPack = new SlackPowerPack(context.getApplicationContext()); slackPowerPack.users() .addUser(User.builder()…/* setters */…build()) .addUser(User.builder()…/* setters */…build()) .pack(); slackPowerPack.channels() .addChannel(Channel.builder()…/* setters */…build()) .addChannel(Channel.builder()…/* setters */…build()) .pack(); POWER PACK
23. AUTHORED AND MAINTAINED BY DEVELOPERS
26. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO 2017 2018
27. 1. EASY TO READ & MAINTAIN 2. STABLE & FAST E2E TO ESPRESSO
28. E2E TO ESPRESSO ROBOTS @Test fun signInWithValidCredentialsAndSignOut() { signInRobot .signIn(signedInUser) .verifyToolbarTeamName(signedInUser.teamDomain) .tapOverflowSettingsOption() .signOutOfCurrentTeam() } public ToolbarRobot signIn(TestUser testUser, boolean waitForHomeActivity) { tapSignInButton(); tapSignInManuallyButton(); enterTeamURL(testUser.teamDomain); tapNextButtonOnTeamURL(); enterEmailAddress(testUser.email); tapNextButtonOnEnterEmailAddress(); enterPassword(testUser.password); tapNextButtonOnEnterPassword(); if (waitForHomeActivity) { // Wait until activity transition between FirstSignInActivity to HomeActivity onView(isRoot()).perform(waitForActivity(HomeActivity.class, MAX_TIMEOUT)); } return new ToolbarRobot(); } https://jakewharton.com/testing-robots/
29. E2E TO ESPRESSO KOTLIN ROBOTS? toolbarRobot .tapOverflowSettingsOption() // transition to settings .tapAdvancedSettings() // transition to advanced settings .tapResetLocalStorage() // transition back to home activity .verifyToolbarTeamName(signedInUser.teamDomain) toolbar { overflow(SETTINGS) } settings { setting(ADVANCED) advanced { tapResetLocalStorage() } } toolbar { verify { toolbarTeamName(signedInUser.teamDomain) } } fun toolbar(func: ToolbarRobot.() -> Unit) = ToolbarRobot().apply { func() } ¯\_(ツ)_/¯
30. 1. EASY TO READ & MAINTAIN 2. STABLE & FAST E2E TO ESPRESSO
31. E2E TO ESPRESSO E2E TESTS CAN BE TARGETED public abstract class SignedInBaseTest { @Override protected void createObjectGraph() { slackApp = ApplicationProvider.getApplicationContext(); appComponent = EndToEndAppComponentBuilder.get(slackApp, getMdmComponent()); appComponent.endpointsHelper().setApiUrl(getApiUrl()); slackApp.resetAppComponent(appComponent); appComponent.inject(this); accountManager = appComponent.accountManager(); signInDataProvider = appComponent.signInDataProvider(); setupUserAndTeamData(); Account account = accountManager.getAccountWithTeamId(signedInUser.teamId); EndToEndUserComponent userComponent = appComponent.endToEndUserComponent() .loggedInUser(LoggedInUser.create(account.userId(), account.teamId(), account.enterpriseId(), account.authToken())) .idlingResourceModule(new IdlingResourceModule(IdlingResourceTestProvider.END_TO_END_IDLING_COUNTER_PROVIDER)) .build(); slackApp.addUserGraph(signedInUser.teamId, userComponent); inject(userComponent); } POINT APP AT THE GIVEN BACKEND SET UP SIGNED IN STATE
32. E2E TO ESPRESSO SYNCHRONIZE WITH IDLINGRESOURCES ▸ OkHttp ▸ RxIdler ▸ IdlingExecutors https://github.com/JakeWharton/okhttp-idling-resource https://github.com/square/RxIdler
33. E2E TO ESPRESSO IDLING EXECUTORS
34. E2E TO ESPRESSO IDLING EXECUTORS https://github.com/google/guava PACKAGE PRIVATE
35. E2E TO ESPRESSO IDLING EXECUTORS public final class WrappedCallable<T> implements Callable<T> { private final Callable<T> delegate; private final CountingIdlingResource idlingResource; public WrappedCallable(Callable<T> delegate, CountingIdlingResource idlingResource) { this.delegate = delegate; this.idlingResource = idlingResource; } @Override public T call() throws Exception { Timber.v("%s(IdlingResource).increment", idlingResource.getName()); idlingResource.increment(); T call; try { call = delegate.call(); } finally { Timber.v("%s(IdlingResource).decrement", idlingResource.getName()); idlingResource.decrement(); } return call; } }
36. E2E TO ESPRESSO IDLING EXECUTORS @Provides @UserScopeSingleton static EventDispatcher provideMSEventDispatcher(EventHandlerFactory eventHandlerFactory, PersistentStore persistentStore, JsonInflater jsonInflater, @Named(EspressoModule.EVENT_DISPATCHER_IDLING_RESOURCE) CountingIdlingResource idlingResource) { PausableThreadPoolExecutor msEventsExecutor = PausableThreadPoolExecutorImpl.newSingleThreadExecutor( new NamedThreadFactory("ms-event-dispatcher-")); if (BuildConfig.DEBUG) { msEventsExecutor = new IdlingPausableThreadPoolExecutor(msEventsExecutor, idlingResource); } return new EventDispatcherImpl(eventHandlerFactory, msEventsExecutor, persistentStore, jsonInflater); }
37. E2E TO ESPRESSO PLUGGING DIRECTLY INTO PROD CODE internal fun filter(constraint: CharSequence?): List<UniversalResult> { countingIdlingResource?.increment() return universalResultDataProviderLazy.get().getResults(constraint.toString(), options) .onErrorReturn { emptyList } .blockingGet() } … override fun publishResults(constraint: CharSequence?, results: FilterResults) { // This method is called by a delayed handler message, so previous results might emit // one after another. We want to ignore these backpressure updates and only take the latest. // See Filter.RequestHandler for implementation details. if (constraint == latestConstraint) { @Suppress("UNCHECKED_CAST") // We suppress this cast warning because we know it's always going to be a list of UniversalResults. displayList = results.values?.let { results.values as List<UniversalResult> } ?: emptyList notifyDataSetChanged() } countingIdlingResource?.decrement() }
38. E2E TO ESPRESSO CONSIDER BUSYBEE https://github.com/americanexpress/busybee
39. E2E TO ESPRESSO A SPRINKLING OF HACKS IN ROBOTS ▸ Activity transitions ▸ Hacky “synchronization” with waitForElement fun enterChannelName(channelName: String): CreateWorkspaceRobot { waitForElementDisplayed(allOf(withId(R.id.header), withText(R.string.what_is_a_project))) waitAndTypeText(allOf(withId(R.id.input_field), withHint(R.string.project_name_example), isDisplayed()), channelName) return this } // Reduce flakiness with transition between FirstSignInActivity to HomeActivity onView(isRoot()).perform(waitForActivity(HomeActivity.class, MAX_TIMEOUT));
40. E2E TO ESPRESSO PROMISING RESULTS ▸ vastly improved stability ▸ speed up of 7x ▸ ability for devs to debug locally & contribute
41. E2E TO ESPRESSO BUT STILL NOT OWNED BY DEVS ▸ not enough stability for pre-merge ▸ no official handover
42. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO PERF TESTING E2E2U 2017 2018
43. HANDOVER TO DEVS E2E2U PR-BLOCKING CHECK ~100% STABILITY
44. RUN E2E TESTS AGAINST RELEASE VARIANT release { minifyEnabled true …
45. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT
46. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT ./gradlew app:assembleInternalRelease app:assembleInternalReleaseAndroidTest -PtestBuildTypeRelease=true 18:08:57 gcloud: 18:08:57 app: /mnt/tmp/android-e2e-full/TEST-ONLY-app-internal-x86-release.apk 18:08:57 test: /mnt/tmp/android-e2e-full/TEST-ONLY-app-internal-release-androidTest.apk
47. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT 6 MIN 30 MIN
48. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT https://issuetracker.google.com/issues/126429384
49. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT
50. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT
51. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT
52. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT ▸ We need to build an apk that ▸ is minified ▸ can be used to run instrumentation tests ▸ has code coverage turned on staging { initWith release signingConfig signingConfigs.debug // Enable code coverage for staging build variant testCoverageEnabled true proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro', 'proguard-staging-rules.pro' matchingFallbacks = ['release'] }
53. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT if (BuildConfig.DEBUG) { enableStrictMode(); ThreadUtils.enableThreadChecks(); } if (buildConfig.isDebug() && !buildConfig.isStaging()) { enableStrictMode(); ThreadUtils.enableThreadChecks(); }
54. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT https://nationalmarker.com/product/end-detour-sign-5643/tm232k/
55. HANDOVER TO DEVS PR-BLOCKING CHECK ~100% STABILITY
56. E2E2U STABILITY: RUN AGAINST PRODUCTION BACKEND ▸ staging env = somewhat isolated production ▸ much more stable than dev environments ▸ downside: no test apis ▸ can’t easily manipulate it for testing (e.g. creating users, etc) https://slack.engineering/how-slack-built-shared-channels-8d42c895b19f
57. E2E2U STABILITY: LIVING WITH SHARED STATE ▸ problem: multiple tests running in parallel using the same team/user ▸ shared state lives “in the cloud” ▸ not possible/practical to set up new test fixture for every test ▸ solution: provision many users and pick randomly from the set for each test
58. E2E2U STABILITY: LIVING WITH SHARED STATE private void setUpRandomUsers(int extraUserCount) { final TestUser.Generator generator = getTestUserGenerator(); signedInUser = generator.generate(); secondaryUser = generator.generate(); for (int i = 0; i < extraUserCount; i++) { userCollection[i] = generator.generate(); } } class DefaultGeneratorImpl { val excluded = mutableSetOf<String>() override fun generate(): TestUser { val user = random(*excluded.toTypedArray()) excluded += user.userId return user } fun random(vararg avoidUsersWithIds: String): TestUser { // attempt (up to maxUsers times) to generate a random user that does not collide with the provided list of users. // we'd have to be really unlucky, but it is theoretically possible that we won't get it, therefore // we handle that case by throwing a RE. for (x in 0 until maxUsers) { val i = Random.nextInt(from = 3, until = idMap.size) val userId = idMap[i] ?: throw NoSuchElementException("No user id for index $i") if (userId !in avoidUsersWithIds) { return TestUser(teamDomain, “$baseUserNamePrefix$i", “$baseEmail+$i@slack-corp.com", password, userId, teamId) } } throw IllegalStateException("Could not generate random user different from $avoidUsersWithIds") }
59. E2E2U STABILITY: THIS ONE EASY TRICK WILL MAKE YOUR TESTS NOT FLAKY https://github.com/TestArmada/flank gcloud app: $APK_PATH test: $TEST_APK_PATH use-orchestrator: true flaky-test-attempts: 4
60. E2E2U STABILITY: FLAKY TEST ANALYTICS
61. E2E2U EDUCATION AND SUPPORT
62. E2E2U EDUCATION AND SUPPORT https://www.positivediscipline.com/articles/letting-go-morning-hassles-and-responsibility
63. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO PERF TESTING E2E2U TODAY 2017 2018
64. TODAY HEALTHY TEST COUNT & RUNTIME 18:18:09 CostReport 18:18:09 Virtual devices 18:18:09 $1.75 for 1h 45m TODAY
65. TODAY DEVS ARE ADDING E2E TESTS
66. TODAY DEVS ARE ADDING E2E TESTS
67. TODAY STABILITY IS OK
68. TODAY BUT NOT AS GOOD AS IT SHOULD BE
69. TODAY MOST COMMON TYPE OF FAILURE: SERVER-SIDE FEATURE FLAG FLIP
70. TODAY BUT ALSO NEED MORE WORK ON SYNCHRONIZATION https://android.jlelse.eu/integrate-espresso-idling-resources-in-your-app-to-build-flexible-ui-tests-c779e24f5057
71. TODAY AND DAGGER public abstract class SignedInBaseTest { @Override protected void createObjectGraph() { slackApp = ApplicationProvider.getApplicationContext(); appComponent = EndToEndAppComponentBuilder.get(slackApp, getMdmComponent()); appComponent.endpointsHelper().setApiUrl(getApiUrl()); slackApp.resetAppComponent(appComponent); appComponent.inject(this); accountManager = appComponent.accountManager(); signInDataProvider = appComponent.signInDataProvider(); setupUserAndTeamData(); Account account = accountManager.getAccountWithTeamId(signedInUser.teamId); EndToEndUserComponent userComponent = appComponent.endToEndUserComponent() .loggedInUser(LoggedInUser.create(account.userId(), account.teamId(), account.enterpriseId(), account.authToken())) .idlingResourceModule(new IdlingResourceModule(IdlingResourceTestProvider.END_TO_END_IDLING_COUNTER_PROVIDER)) .build(); slackApp.addUserGraph(signedInUser.teamId, userComponent); inject(userComponent); }
72. TODAY BUT WE’VE COME A LONG WAY Not so long ago Hours of failure triage before every release Today Ship it on green (almost)
73. STABLE E2E TESTS ON ANDROID IS SUCH AT THING EVEN POSSIBLE? @valera_zakharov
74. YES
75. ARE THERE ANY QUESTIONS? @valera_zakharov
77. THANK YOU DROIDCON SF! QUESTIONS? @valera_zakharov