Evolution of Test Automation at Spotify

Quality is important. People love a high-quality application. So, how can we achieve it? Spotify wants to release an application faster. We have a one-week release cycle. Every day, hundreds of engineers are working on the same code base. We believe that test automation is one of the key factors to keep high quality in a sustainable way.

Test automation has three levels: unit test, integration test, and UI test. In this talk, we will present how Spotify has evolved test automation for Mobile. Especially, we're going to talk on integration and UI test level including our internal systems.

Transcript

1. Evolution of Test Automation Sangsoo Nam sangsoo@spotify.com
2. Test Pyramid Unit Tests Integration Tests E2E Tests
3. Login Test Case - Click email input field - Type email - Click password input field - Type password - Click "Login" button
4. Script Testing
5. Login Test Case Easy to do it manually, but How to automate testing?
6. Script Test Server Application Logic @Test public void testLogin() { clickEmailInputField(); type(“user@account.com"); clickPasswordInputField(); type(“password"); clickLoginButton(); }
7. Script Test Server Application Logic
8. Script Test Server Application Logic Script Engine
9. BeanShell Lightweight Scripting for Java Runs in the Java Runtime Environment Script Test Server Application Logic Script Engine
10. Script Test Server Application Logic Script Engine email = findView("input_text_email"); email.performClick(); email.setText("user@account.com") password = findView("input_text_password"); password.performClick(); password.setText("password") findView("button_login").performClick();
11. Page Object Pattern Page Object allows to do and see 
 that a user can on a page Hide the underlying mechanics and show abstract-level interfaces Page Page Object Test
12. Login View public class LoginView { public void clickLogin() { ... } public void typeEmail(String email) { ... } public void typePassword(String password) { ... }
13. Script Test Server Application Logic Script Engine @Test public void testLogin() { login.typeEmail(“user@account.com"); login.typePassword(“password"); login.clickLogin(); }
14. Login Test Case @Test public void testLogin() { LoginAuto login = Pages.remote(LoginAuto.class); login.typeEmail("user@account.com"); login.typePassword("password"); login.clickLogin(); HomeAuto home = Pages.remote(HomeAuto.class); assertTrue(home.isDisplayed()); }
15. Login Test Case @Test public void testLogin() { LoginAuto login = Pages.remote(LoginAuto.class); login.typeEmail("user@account.com"); login.typePassword("password"); login.clickLogin(); HomeAuto home = Pages.remote(HomeAuto.class); assertTrue(home.isDisplayed()); } Fail.. no guarantee to show immediately
16. Authentication Network request UI Rendering …
17. Login Test Case (Periodic Polling) @Test public void testLogin() { LoginAuto login = Pages.remote(LoginAuto.class); login.typeEmail("user@account.com"); login.typePassword("password"); login.clickLogin(); HomeAuto home = Pages.remote(HomeAuto.class); assertWaitTrue(home.isDisplayed(), Duration.ofSeconds(10).toMillis()); }
18. User Accounts
19. Account Issues Spoiled Accounts Empty state verifications Playlist, recently played, search history, … Concurrent Access Playback with Connect
20. TDS(Test Data Service) Provide a simple and scalable solution to acquire a Spotify user Use a user pool and support tagging users
 Don’t return the user currently being used
21. Acquire Tag: EMPTY_PLAYLIST EMPTY_PLAYLIST FREE PREMIUM TDS
22. Acquire Tag: EMPTY_PLAYLIST EMPTY_PLAYLIST FREE PREMIUM TDS
23. Acquire Tag: EMPTY_PLAYLIST EMPTY_PLAYLIST FREE PREMIUM Release TDS
24. Model Based Testing
25. Manual Testing Test Automation Model Based Testing
26. Model Abstraction of the real world Consist of expected behavior and states of the system under the test Test cases are automatically generated from models
29. Test Case Generation Smoke Test new AStarPath(new ReachedVertex(“v_Done”)) Functional Test new RandomPath(new EdgeCoverage(100)) Stability Test new RandomPath(new TimeDuration(30, TimeUnit.MINUTES))
30. Model Interface public interface LoginModel {
 public void e_Init(); public void v_LoginPrompted(); public void e_InValidCredentials(); public void v_LoginFailed(); public void e_ValidCredentials(); public void v_InitialView(); }
31. Model Interface Implementation public class LoginModelAPI implements LoginModel { private LoginAuto login; public void e_Init() { login = Pages.remote(LoginAuto.class); } public void e_ValidCredentials() { login.typeEmail("user@account.com"); login.typePassword("password"); login.clickLogin(); } ... }
32. Problematic Test Pyramid
33. Problematic Test Pyramid Unit Tests E2E Tests QA
34. Problematic Test Pyramid Unit Tests E2E Tests QA Running E2E is slow… Problem could be found after merging
35. Backend services SpotifyProvider
 (Custom Content Provider) Android Services Core(C++) Metadata Features … Playback Session Ads …
36. E2E is ONLY option to test integration 
 with platform level components Backend services SpotifyProvider
 (Custom Content Provider) Android Services Core(C++) Metadata Features … Playback Session Ads …
37. Cassette Testing A Fast Reproducible Integration Testing Environment With Robolectric
38. Cassette Capabilities - Perform, record, and reply network requests for any given user
 (support OKHttp and custom protocol used at Spotify) - Simulate service binds to a pre-configured Android services
 (control playback, product updates, …)
 - Simulate queries to a pre-configured ContentProvider - Check and assert on UI View hierarchy states - Simulate app states
 (foreground/background)
39. Backend services SpotifyProvider
 (Custom Content Provider) Android Services Core(C++) Metadata Features … Playback Session Ads …
40. Backend services SpotifyProvider
 (Custom Content Provider) Android Services Core(C++) Metadata Features … Playback Session Ads … Recordable
41. Backend services SpotifyProvider
 (Custom Content Provider) Android Services Core(C++) Metadata Features … Playback Session Ads … Recordable
42. Fast Stable Network Core (C++) Android Services E2E Unit Cassette
43. public class ArtistHubFragmentIntegrationTest extends IntegrationTestCase { @Rule private final SpotifyCassetteRule mSpotifyCassetteRule = new SpotifyCassetteRule(); @Override protected Fragment createStartFragment(Flags flags, String testUsername) { return ArtistHubFragment.create("spotify:artist:0SfsnGyD8FpIN4U4WCkBZ5", "ARTIST", flags); } @Test @Cassette(user = @User(flags = AndroidUserFlags.PREMIUM)) public void testClickRelatedArtists() throws Throwable { Bdd.given((Setup<ArtistAuto>) () -> { return new ArtistAutoImpl((ArtistHubFragment) getFragment()); }).when((Action<ArtistAuto, ArtistRelatedArtistsAuto>) auto -> { auto.clickRelatedArtists(); return new RelatedArtistsAutoImpl((ArtistRelatedArtistsFragment) getNextFragment())); }).then(auto -> { assertTrue("Related Artists page did not load.", auto.isLoaded()) }); }
44. public class ArtistHubFragmentIntegrationTest extends IntegrationTestCase { @Rule private final SpotifyCassetteRule mSpotifyCassetteRule = new SpotifyCassetteRule(); @Override protected Fragment createStartFragment(Flags flags, String testUsername) { return ArtistHubFragment.create("spotify:artist:0SfsnGyD8FpIN4U4WCkBZ5", "ARTIST", flags); } @Test @Cassette(user = @User(flags = AndroidUserFlags.PREMIUM)) public void testClickRelatedArtists() throws Throwable { Bdd.given((Setup<ArtistAuto>) () -> { return new ArtistAutoImpl((ArtistHubFragment) getFragment()); }).when((Action<ArtistAuto, ArtistRelatedArtistsAuto>) auto -> { auto.clickRelatedArtists(); return new RelatedArtistsAutoImpl((ArtistRelatedArtistsFragment) getNextFragment())); }).then(auto -> { assertTrue("Related Artists page did not load.", auto.isLoaded()) }); }
45. 20x Times faster than before
46. Scaling Android application testing at Spotify Sean Kenny
 https://atscaleconference.com/videos/scaling-android-application-testing-at-spotify/
47. Test Pyramid Unit Tests Integration Tests E2E Tests
48. Espresso Testing
49. Espresso - Concise, beautiful, and reliable Android UI tests - Support synchronization with idling resources - Instrumented test: Have access to instrumentation information. 
 e.g. the context of the app - Result with a recorded video, performance analysis, and more @Test public void greeterSaysHello() { onView(withId(R.id.name_field)).perform(typeText("Steve")); onView(withId(R.id.greet_button)).perform(click()); onView(withText("Hello Steve!")).check(matches(isDisplayed())); }
50. Test Pyramid Unit Tests Integration Tests E2E Tests Espresso Test
 Recurring Cassette Test
 On Pre-merge
51. Culture
52. Test Certified for Client
53. Test Certified for Client Knowing • 100% code and test ownership registration • Number, type, and phase of current tests is known 
 Improving • Healthy distribution of unit tests, integration tests, and E2E tests • No static analysis warnings, no suppressed linter warnings 
 Maintaining • Do not decrease code coverage • Bug fixes require automated regression tests • New tests are written at the lowest level of complexity that provides the best value
54. Quality = Tools + Culture
55. Thank you. Sangsoo Nam sangsoo@spotify.com