Automated Testing @Square - Droidcon NYC 2014

Scratching the surface on what appears to be a very difficult problem to get right on Android. Goals, techniques and useful tips to get started.

Transcript

1. Automated Testing
   Dimitris K.
   

   View Slide

2. Quality Brand
   Automated Testing
   

   View Slide

3. Automated Testing
   Extremely important to us
   

   View Slide

4. Quality Yesterday
   Automated Testing
   

   View Slide

5. Quality Yesterday
   Manual Testing
   Automated Testing
   

   View Slide

6. Unit Testing
   Quality Yesterday
   Automated Testing
   

   View Slide

7. Quality Yesterday
   “Add ability to compile and run tests”
   - Eric Burke
   Automated Testing
   

   View Slide

8. Quality Today
   Automated Testing
   

   View Slide

9. No Flakiness
   Quality Today
   Automated Testing
   

   View Slide

10. Easy to write tests
    Quality Today
    Automated Testing
    

    View Slide

11. Catch edge cases
    Quality Today
    Automated Testing
    

    View Slide

12. Quality Today
    Automated Testing
    Graybox testing is preferred
    

    View Slide

13. Quality Today
    Automated Testing
    Tests are red == master is red
    

    View Slide

14. Robots
    Automated Testing
    

    View Slide

15. Robots
    Thin layer above UI framework
    Automated Testing
    Fluent API
    Single point to update your code
    

    View Slide

16. Robots
    public class InvoiceScreenRobot extends ScreenRobot {
    public InvoiceScreenRobot typeTitle(String title) {
    onView(withId(R.id.title)).perform(typeText(title));
    return this;
    }
    // More here...
    public void clickSend() {
    onView(withId(R.id.send)).perform(click());
    }
    }
    Automated Testing
    

    View Slide

17. Robots
    public void testSendInvoice() {
    withRobot(InvoiceScreenRobot.class) //
    .typeTitle("Sample Title") //
    .typeNumber("#000001") //
    .typeName("Square User") //
    .typeEmail("squareuser@squareup.com") //
    .clickSend();
    }
    Automated Testing
    

    View Slide

18. Robots
    public void testSendInvoice() {
    withRobot(InvoiceScreenRobot.class) //
    .checkIsDisabled(R.id.send) //
    .typeTitle("Sample Title") //
    .typeNumber("#000001") //
    .typeName("Square User") //
    .typeEmail("squareuser@squareup.com") //
    .checkIsEnabled(R.id.send) //
    .clickSend();
    }
    Automated Testing
    

    View Slide

19. Mock Mode
    Automated Testing
    

    View Slide

20. Mock Mode
    Eliminate network flakiness
    Automated Testing
    Rapid development
    Test network edge cases
    

    View Slide

21. Mock Mode
    @Provides @Singleton MyService provideMyService(RestAdapter adapter) {
    return adapter.create(MyService.class);
    }
    Automated Testing
    

    View Slide

22. Mock Mode
    @Provides @Singleton
    MyService provideMyService(RestAdapter adapter) {
    MyService real = restAdapter.create(MyService.class);
    MyService mock = new MockMyService();
    return ServiceProxy.newInstance(MyService.class, mock, real);
    }
    Automated Testing
    

    View Slide

23. Mock Mode
    public class ServiceProxy implements InvocationHandler {
    @Override public Object invoke(Object proxy, Method
    method, Object[] args) throws Throwable {
    Object service = mockModeOn() ? mock : real;
    try {
    return method.invoke(service, args);
    } catch (InvocationTargetException e) {
    throw e.getTargetException();
    }
    }
    Automated Testing
    

    View Slide

24. Mock Mode
    public class MockMyService implements MyService {
    @Override public Response doSomething(Request request) {
    delay();
    boolean success = true;
    return new Response(success);
    }
    }
    Automated Testing
    

    View Slide

25. Test Matrix
    Automated Testing
    

    View Slide

26. Test Matrix
    Automated Testing
    Card Not Present Flow
    

    View Slide

27. Test Matrix
    7 card types
    Automated Testing
    4 languages
    4 receipt types
    

    View Slide

28. Test Matrix
    Automated Testing
    112 tests to write
    

    View Slide

29. Burst
    Automated Testing
    

    View Slide

30. Burst
    Test Generator
    Automated Testing
    

    View Slide

31. Burst
    Variation Proximity
    Automated Testing
    

    View Slide

32. Burst
    Type Safety
    Automated Testing
    

    View Slide

33. Burst
    Use Everywhere
    Automated Testing
    

    View Slide

34. Burst
    Automated Testing
    Not
    @Parameterized.Parameters(name = "{0}")
    public static List parameters() {
    return Arrays.asList(new Object[] {
    new Factory() {
    @Override public BufferedSink create(Buffer data) {
    return data;
    }
    @Override public String toString() {
    return "Buffer";
    }
    },
    }, new Object[] {
    new Factory() {
    @Override public BufferedSink create(Buffer data) {
    return new RealBufferedSink(data);
    }
    @Override public String toString() {
    return "RealBufferedSink";
    }
    }
    });
    }
    @Parameterized.Parameter
    public Factory factory;
    

    View Slide

35. Burst
    public enum Soda {
    COKE,
    PEPSI,
    RC_COLA
    }
    @RunWith(BurstJUnit4.class)
    public class SampleTest {
    @Test public void sodas(Soda soda) {
    // Test will run 3 times
    assertNotNull(soda);
    }
    }
    Automated Testing
    

    View Slide

36. Burst
    public enum Soda {
    COKE,
    PEPSI,
    RC_COLA
    }
    @RunWith(BurstJUnit4.class)
    public class SampleTest {
    @Test public void sodasAndSnacks(Soda soda,
    Snack snack) {
    // Test will run 9 times
    assertNotNull(soda);
    assertNotNull(snack);
    }
    }
    public enum Snack {
    CHIPS,
    KITKAT,
    CANDY
    }
    Automated Testing
    

    View Slide

37. Burst
    Friendly Test Naming
    Automated Testing
    

    View Slide

38. Burst
    sodasAndSnacks[Soda.COKE, Snack.CHIPS]
    sodasAndSnacks[Soda.COKE, Snack.KITKAT]
    sodasAndSnacks[Soda.COKE, Snack.CANDY]
    Automated Testing
    

    View Slide

39. Burst
    @TabletOnly
    public void testTabletFeature() {
    // This test will only be executed on tablets.
    }
    Automated Testing
    

    View Slide

40. Burst
    @TabletOnly
    public void testTabletFeature() {
    // This test will only be executed on tablets.
    }
    @PhoneOnly
    public void testPhoneFeature() {
    // This test will only be executed on phones.
    }
    Automated Testing
    

    View Slide

41. Automated Testing
    @TabletOnly
    public void testTipBringsAmountOver25() {
    loginAsAndStartActivity("bart");
    performAuthSteps(2499L);
    // Tip
    withRobot(TipScreenRobot.class).clickTipOptionOne();
    // Sign
    withRobot(SignScreenRobot.class)
    .performSign()
    .clickContinue();
    // Receipt
    withRobot(ReceiptScreenRobot.class)
    .performEmailReceipt("bart@squareup.com")
    .clickNewSale();
    }
    

    View Slide

42. Tips
    Automated Testing
    

    View Slide

43. Tips
    Animations OFF
    Automated Testing
    

    View Slide

44. Don’t Sleep()
    Tips
    Automated Testing
    

    View Slide

45. Tips
    Screenshots
    Automated Testing
    

    View Slide

46. Tips
    Automated Testing
    @Override protected void setUp() throws Exception {
    super.setUp();
    setFailureHandler(new RegisterFailureHandler(this));
    }
    

    View Slide

47. Tips
    Automated Testing
    public class RegisterFailureHandler implements FailureHandler {
    private final TestCase testCase;
    public RegisterFailureHandler(TestCase testCase) {
    this.testCase = testCase;
    }
    @Override public void handle(Throwable throwable,
    Matcher viewMatcher) {
    // Very useful for finding out what went wrong...
    takeAndUploadScreenshot(testCase.getName());
    // Delegate to real failure handler here.
    }
    }
    

    View Slide

48. Tips
    Automated Testing
    public class ScreenShotProxy implements InvocationHandler {
    @Override
    public Object invoke(Object proxy, Method method, Object[]
    args) throws Throwable {
    try {
    Object result = method.invoke(proxy, args);
    takeScreenShot(); // <----- magic.
    return result;
    } catch (InvocationTargetException e) {
    throw e.getTargetException();
    }
    }
    }
    

    View Slide

49. Tips
    Keyboard :(
    Automated Testing
    

    View Slide

50. A
    https://github.com/square/burst
    https://squareup.com/careers
    https://github.com/square/burst
    

    View Slide