Sustainable Test Automation - DroidCon Stockholm 2015

E54870616ec100ee98aa57e96f76fde9?s=47 sedovmik
September 03, 2015

Sustainable Test Automation - DroidCon Stockholm 2015

Automated Functional Testing of an application on Android is genetically fragile. Have you ever had a feeling that the effort to keep them green seems to be higher than the added value?

This talk is a story of two years of developing Spotify Android application from a perspective of Test Automator. And our approach to reducing the cost of writing and maintaining automated functional tests.

E54870616ec100ee98aa57e96f76fde9?s=128

sedovmik

September 03, 2015
Tweet

Transcript

  1. 2.

    Who am I? Mikhail Sedov Test Automator / Android Developer

    Android Platform Team At Spotify since April 2013
  2. 5.

    How to write and maintain functional tests which can run

    for hours making meaningful actions
  3. 11.

    Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }

  4. 12.

    Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); Player.setPlaying(true);
  5. 13.

    Implementation public class NowPlayingModel {
 
 public void e_Play() {

    … }
 
 public void e_Pause() { … }
 
 public void v_Playing() { … }
 
 public void v_Paused() { … }
 Player.setPlaying(true); assertFalse(      Player.isPlaying()); Player.setPlaying(true); assertFalse(      Player.isPlaying());
  6. 14.

    @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
  7. 15.

    @Test
 public void runSmokeTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new AStarPath( new ReachedVertex("v_Done")))
 .setStart("e_Init")
 .execute();
 }
  8. 16.

    @Test
 public void runFunctionalTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new RandomPath( new EdgeCoverage(100)))
 .setStart("e_Init")
 .execute();
 }
  9. 17.

    @Test
 public void runStabilityTest() {
 new TestBuilder()
 .setModel(MODEL_PATH)
 .setContext(new NowPlayingModel())


    .setPathGenerator( new RandomPath( new TimeDuration(30, TimeUnit.MINUTES)))
 .setStart("e_Init")
 .execute();
 }
  10. 18.
  11. 20.

    Page Object Pattern • Allows to do anything and see

    anything that a human can on a page • Hides the underlying mechanics required to find and manipulate UI
  12. 21.

    public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  13. 22.

    public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  14. 23.

    public class NowPlayingView {
 
 public void play() { …

    }
 
 public void pause() { … }
 
 public boolean isPlaying() { … }
 
 public void pressNext() { … }
 
 public void pressPrevious() { … }
 
 public Track getCurrentTrack() { … }
  15. 28.

    Functional Tests Application Remote Control Scripting Engine mServerSocket = new

    ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
  16. 29.

    Functional Tests Application Remote Control Scripting Engine Application Logic Android

    Internals mServerSocket = new ServerSocket(mPort); public void handleMessage(final Message msg) {
 String command = (String) msg.obj;
 Object result = mInterpreter.eval(command);
 send("200 OK", serialize(result));
 }
  17. 41.

    /**
 * Class represents Player page
 */
 @AutomationInterface
 public interface

    PlayerAuto {
 
 public enum RepeatMode {
 NONE, ALL, ONE;
 }
 
 public boolean isPlaying();
 
 public void setPlaying(boolean playing);
 
 public RepeatMode getRepeatMode();
 
 public void pressRepeatButton(); …
  18. 42.

    public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton;

    …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
  19. 43.

    public class PlayerFragment extends Fragment {
 
 private Button mRepeatButton;

    …
 
 public class AutoDelegateImpl implements PlayerAuto {
 
 @Override
 public boolean isPlaying() {
 final PlayerState state = mPlayerPresenter.getLastPlayerState();
 return state != null && state.isPlaying();
 }
 
 @Override
 public void pressRepeatButton() {
 mRepeatButton.performClick();
 } … }
  20. 45.

    public class Pages implements InvocationHandler {
 
 @Override
 public Object

    invoke(Object proxy, Method method, Object[] args) {
 
 String name = method.getName();
 Class<?> returnType = method.getReturnType();
 
 String command = buildCommand(name, args);
 int ret = transport.send(command);
 
 return buildObject(transport.reply(), returnType);
 } Dynamic Proxy 1 2 3
  21. 46.

    Wins • A contract between functional tests and an application

    • No tests depending on the app implementation
  22. 47.
  23. 48.
  24. 51.

    Caveats: false negative • Click on invisible things • Click-Through

    • Bypassing UI (e.g. not using performClick) A test result that is incorrect because the test failed to recognize an existing condition or finding.
  25. 53.

    • Simple abstractions to reduce Test Automation complexity (States, Actions,

    Models) • Programmable interface on top of user-facing features TL;DR
  26. 55.

    • http://martinfowler.com/bliki/PageObject.html • http://graphwalker.org/ • https://en.wikipedia.org/wiki/A*_search_algorithm • https://en.wikipedia.org/wiki/False_positives_and_false_negatives • https://www.hakkalabs.co/articles/protect-your-baby-how-spotify-does-testing-

    for-mobile (Sean Kenny, Android developer at Spotify, covers testing levels) • Spotify Engineering Culture: • Part 1: https://vimeo.com/85490944 • Part 2: https://vimeo.com/94950270