Sustainable Test Automation - DroidCon Stockholm 2015

Automated Functional Testing of an application on Android is genetically fragile. Have you ever had a feeling that the effort to keep them green seems to be higher than the added value?

This talk is a story of two years of developing Spotify Android application from a perspective of Test Automator. And our approach to reducing the cost of writing and maintaining automated functional tests.

Transcript

1. Sustainable Test Automation Mikhail Sedov
2. Who am I? Mikhail Sedov Test Automator / Android Developer Android Platform Team At Spotify since April 2013
3. Agenda Model-Based Testing Page Object Test Interface
4. Test Automation in Spotify
5. How to write and maintain functional tests which can run for hours making meaningful actions
6. Model-Based Testing
7. https://github.com/KristianKarl/GraphWalker-Examples
8. https://github.com/KristianKarl/GraphWalker-Examples
9. Paused Playing Minimized + Paused Minimized + Playing
10. Graphwalker 3 MIT licensed Open Source project.
11. Implementation public class NowPlayingModel {
 
 public void e_Play() { … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }

12. Implementation public class NowPlayingModel {
 
 public void e_Play() { … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); Player.setPlaying(true);
13. Implementation public class NowPlayingModel {
 
 public void e_Play() { … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); assertFalse(      Player.isPlaying()); Player.setPlaying(true); assertFalse(      Player.isPlaying());
14. @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())
 .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
15. @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())
 .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
16. @Test
 public void runFunctionalTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())
 .setPathGenerator( new RandomPath( new EdgeCoverage(100)))
 .setStart("e_Init")
 .execute();
 }
17. @Test
 public void runStabilityTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())
 .setPathGenerator( new RandomPath( new TimeDuration(30, TimeUnit.MINUTES)))
 .setStart("e_Init")
 .execute();
 }
18. Models
19. Page Object Pattern
20. Page Object Pattern • Allows to do anything and see anything that a human can on a page • Hides the underlying mechanics required to find and manipulate UI
21. public class NowPlayingView {
 
 public void play() { … }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
22. public class NowPlayingView {
 
 public void play() { … }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
23. public class NowPlayingView {
 
 public void play() { … }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
24. Functional Tests Application
25. Functional Tests Application Remote Control
26. Functional Tests Application Remote Control mServerSocket = new ServerSocket(mPort);
27. Functional Tests Application Remote Control Scripting Engine mServerSocket = new ServerSocket(mPort);
28. Functional Tests Application Remote Control Scripting Engine mServerSocket = new ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
29. Functional Tests Application Remote Control Scripting Engine Application Logic Android Internals mServerSocket = new ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
30. mDriver.getCurrentActivity()
 .findViewById(R.id.button_login)
 .performClick(); Command to click view by ID
31. > 30 developers 400 commits per week 1000 commits per release
32. Models Tests Page Objects
33. Models Tests Page Objects
34. Models Tests Page Objects
35. Application Functional Tests Models Tests Page Objects
36. Test Interface
37. Functional Tests Application Page Objects
38. Functional Tests Application Page Objects
39. Functional Tests Application Test Interfaces
40. Functional Tests Application Test Interfaces apply plugin: 'java'
41. /**
 * Class represents Player page
 */
 @AutomationInterface
 public interface PlayerAuto {
 
 public enum RepeatMode {
 NONE, ALL, ONE;
 }
 
 public boolean isPlaying();
 
 public void setPlaying(boolean playing);
 
 public RepeatMode getRepeatMode();
 
 public void pressRepeatButton(); …
42. public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton; …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
43. public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton; …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
44. PlayerAuto player = Pages.remote(PlayerAuto.class);
 player.isPlaying(); Dynamic Proxy
45. public class Pages implements InvocationHandler {
 
 @Override
 public Object invoke(Object proxy, Method method, Object[] args) {
 
 String name = method.getName();
 Class<?> returnType = method.getReturnType();
 
 String command = buildCommand(name, args);
 int ret = transport.send(command);
 
 return buildObject(transport.reply(), returnType);
 } Dynamic Proxy 1 2 3
46. Wins • A contract between functional tests and an application • No tests depending on the app implementation
49. Wins • Sharing the burden of maintenance with developers
50. Wins • Advanced Page Objects • Faster • More Stable • More actions • Simpler
51. Caveats: false negative • Click on invisible things • Click-Through • Bypassing UI (e.g. not using performClick) A test result that is incorrect because the test failed to recognize an existing condition or finding.
52. Point is: Development productivity > Formal testing
53. • Simple abstractions to reduce Test Automation complexity (States, Actions, Models) • Programmable interface on top of user-facing features TL;DR
54. THANK YOU Mikhail Sedov sedovmik@spotify.com Sept 3rd, 2015
55. • http://martinfowler.com/bliki/PageObject.html • http://graphwalker.org/ • https://en.wikipedia.org/wiki/A*_search_algorithm • https://en.wikipedia.org/wiki/False_positives_and_false_negatives • https://www.hakkalabs.co/articles/protect-your-baby-how-spotify-does-testing- for-mobile (Sean Kenny, Android developer at Spotify, covers testing levels) • Spotify Engineering Culture: • Part 1: https://vimeo.com/85490944 • Part 2: https://vimeo.com/94950270