$30 off During Our Annual Pro Sale. View Details »

Sustainable Test Automation - DroidCon Stockholm 2015

sedovmik
September 03, 2015

Sustainable Test Automation - DroidCon Stockholm 2015

Automated Functional Testing of an application on Android is genetically fragile. Have you ever had a feeling that the effort to keep them green seems to be higher than the added value?

This talk is a story of two years of developing Spotify Android application from a perspective of Test Automator. And our approach to reducing the cost of writing and maintaining automated functional tests.

sedovmik

September 03, 2015
Tweet

Other Decks in Programming

Transcript

  1. Sustainable Test Automation Mikhail Sedov

  2. Who am I? Mikhail Sedov Test Automator / Android Developer

    Android Platform Team At Spotify since April 2013
  3. Agenda Model-Based Testing Page Object Test Interface

  4. Test Automation in Spotify

  5. How to write and maintain functional tests which can run

    for hours making meaningful actions
  6. Model-Based Testing

  7. https://github.com/KristianKarl/GraphWalker-Examples

  8. https://github.com/KristianKarl/GraphWalker-Examples

  9. Paused Playing Minimized + Paused Minimized + Playing

  10. Graphwalker 3 MIT licensed Open Source project.

  11. Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }

  12. Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); Player.setPlaying(true);
  13. Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); assertFalse(      Player.isPlaying()); Player.setPlaying(true); assertFalse(      Player.isPlaying());
  14. @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
  15. @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
  16. @Test
 public void runFunctionalTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new RandomPath( new EdgeCoverage(100)))
 .setStart("e_Init")
 .execute();
 }
  17. @Test
 public void runStabilityTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new RandomPath( new TimeDuration(30, TimeUnit.MINUTES)))
 .setStart("e_Init")
 .execute();
 }
  18. Models

  19. Page Object Pattern

  20. Page Object Pattern • Allows to do anything and see

    anything that a human can on a page • Hides the underlying mechanics required to find and manipulate UI
  21. public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  22. public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  23. public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  24. Functional Tests Application

  25. Functional Tests Application Remote Control

  26. Functional Tests Application Remote Control mServerSocket = new ServerSocket(mPort);

  27. Functional Tests Application Remote Control Scripting Engine mServerSocket = new

    ServerSocket(mPort);
  28. Functional Tests Application Remote Control Scripting Engine mServerSocket = new

    ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
  29. Functional Tests Application Remote Control Scripting Engine Application Logic Android

    Internals mServerSocket = new ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
  30. mDriver.getCurrentActivity()
 .findViewById(R.id.button_login)
 .performClick(); Command to click view by ID

  31. > 30 developers 400 commits per week 1000 commits per

    release
  32. Models Tests Page Objects

  33. Models Tests Page Objects

  34. Models Tests Page Objects

  35. Application Functional Tests Models Tests Page Objects

  36. Test Interface

  37. Functional Tests Application Page Objects

  38. Functional Tests Application Page Objects

  39. Functional Tests Application Test Interfaces

  40. Functional Tests Application Test Interfaces apply plugin: 'java'

  41. /**
 * Class represents Player page
 */
 @AutomationInterface
 public interface

    PlayerAuto {
 
 public enum RepeatMode {
 NONE, ALL, ONE;
 }
 
 public boolean isPlaying();
 
 public void setPlaying(boolean playing);
 
 public RepeatMode getRepeatMode();
 
 public void pressRepeatButton(); …
  42. public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton;

    …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
  43. public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton;

    …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
  44. PlayerAuto player = Pages.remote(PlayerAuto.class);
 player.isPlaying(); Dynamic Proxy

  45. public class Pages implements InvocationHandler {
 
 @Override
 public Object

    invoke(Object proxy, Method method, Object[] args) {
 
 String name = method.getName();
 Class<?> returnType = method.getReturnType();
 
 String command = buildCommand(name, args);
 int ret = transport.send(command);
 
 return buildObject(transport.reply(), returnType);
 } Dynamic Proxy 1 2 3
  46. Wins • A contract between functional tests and an application

    • No tests depending on the app implementation
  47. None
  48. None
  49. Wins • Sharing the burden of maintenance with developers

  50. Wins • Advanced Page Objects • Faster • More Stable

    • More actions • Simpler
  51. Caveats: false negative • Click on invisible things • Click-Through

    • Bypassing UI (e.g. not using performClick) A test result that is incorrect because the test failed to recognize an existing condition or finding.
  52. Point is: Development productivity > Formal testing

  53. • Simple abstractions to reduce Test Automation complexity (States, Actions,

    Models) • Programmable interface on top of user-facing features TL;DR
  54. THANK YOU Mikhail Sedov sedovmik@spotify.com Sept 3rd, 2015

  55. • http://martinfowler.com/bliki/PageObject.html • http://graphwalker.org/ • https://en.wikipedia.org/wiki/A*_search_algorithm • https://en.wikipedia.org/wiki/False_positives_and_false_negatives • https://www.hakkalabs.co/articles/protect-your-baby-how-spotify-does-testing-

    for-mobile (Sean Kenny, Android developer at Spotify, covers testing levels) • Spotify Engineering Culture: • Part 1: https://vimeo.com/85490944 • Part 2: https://vimeo.com/94950270