Clean and Useful Android Testing, with Spock, Espresso and UI Automator

6c22b6758d41f80de6064ee3eac16ee8?s=47 Jon Reeve
September 11, 2015

Clean and Useful Android Testing, with Spock, Espresso and UI Automator

Clean up your tests!
Presented at Droidcon Greece on 11th September 2015.

6c22b6758d41f80de6064ee3eac16ee8?s=128

Jon Reeve

September 11, 2015
Tweet

Transcript

  1. 10-12 September 2015 droidcon Greece Thessaloniki

  2. Clean and Useful Android Testing, with Spock, Espresso and UI

    Automator! Jon Reeve, Wasabi Code, @themightyjon
  3. (Gartner) Hype Cycle Gartner Research's Hype Cycle diagram / Jeremy

    Kemp / CC-BY-SA-3.0
  4. Testing Enthusiasm Curve Plains of Indifference Peak of Religious Zeal

    Trough of Disillusionment Plateau of Productivity ( / Pragmatism)
  5. • Android testing samples & blueprint https:/ /github.com/googlesamples/android-testing-templates https:/ /github.com/googlesamples/android-testing

    • Others https:/ /github.com/AndroidBootstrap/android-bootstrap https:/ /github.com/JakeWharton/u2020 • Many more... Where to Start? Examples...
  6. Mike Cohn's Testing Pyramid What Do We Test?

  7. The often encountered “Testing Ice-Cream Cone” What Do We Test?

  8. • Why? • Challenges • Helpful Architecture • Bad Test

    → Good Tests • Spock Unit Tests
  9. • Value more obvious to developers → Can suffer under

    pressure to do things quickly. • Drive clean code: loose coupling, separation of concerns, etc. • Immediate & clear failure: where and why. Why We Write Unit Tests
  10. • Can be hard. Has got better: • Real unit

    tests vs “Unit Android Tests”, off-device, faster. • Mockable Android JAR. Death to RuntimeException: “Stub!” • But still: • Abstract classes over interfaces • Big classes, combining many concerns • Hidden internals: constants, functions, whole classes. • “final” things Android Unit Testing Challenges
  11. • MVP • Wrap & Abstract • Dependency Inversion •

    Hexagonal Architecture / Ports & Adapters / “Clean Architecture” • Rule out & wall off “hard to test” • ... Basically, separation of concerns, divide by layers. How To Overcome?
  12. • Activity or Fragment is simple, dumb “View”. • Business

    logic in Presenter, isolated from platform. Unit testable. • Good examples elsewhere: • Shazam: http:/ /www.infoq.com/presentations/mobile-shazam • Teh Interwebs: http:/ /lmgtfy.com/?q=mvp+android MVP
  13. • Wrap static function call with Interface + implementation, to

    “loosen” coupling. • System.currentTimeMillis() → Clock { long currentTimeMillis() } • Wrap a system class to hide use of Android classes or functions • Bit much? • Mobile device, varying spec (low end devices). • GC, tight loops. • Be aware of context, and use where appropriate. (I ❤ enum) Wrap + Abstract
  14. • Why? • Challenges • Helpful Architecture • Bad Test

    → Good Tests • Spock Unit Tests
  15. public void sharedPreferencesHelper_SaveAndReadPersonalInformation() { // Save the personal information to

    SharedPreferences boolean success = mMockSharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); assertThat("Checking that SharedPreferenceEntry.save... returns true", success, is(true)); // Read personal information from SharedPreferences SharedPreferenceEntry savedSharedPreferenceEntry = mMockSharedPreferencesHelper.getPersonalInfo(); // Make sure both written and retrieved personal information are equal. assertThat("Checking that SharedPreferenceEntry.name has been persisted and read correctly", mSharedPreferenceEntry.getName(), is(equalTo(savedSharedPreferenceEntry.getName()))); assertThat("Checking that SharedPreferenceEntry.dateOfBirth has been persisted and read " + "correctly", mSharedPreferenceEntry.getDateOfBirth(), is(equalTo(savedSharedPreferenceEntry.getDateOfBirth()))); assertThat("Checking that SharedPreferenceEntry.email has been persisted and read " + "correctly", mSharedPreferenceEntry.getEmail(), is(equalTo(savedSharedPreferenceEntry.getEmail()))); } “Wall of Code” You have to read comments, not code Testing too much at once Failure messages adding noise to test itself
  16. public boolean savePersonalInfo(SharedPreferenceEntry sharedPreferenceEntry){ // Start a SharedPreferences transaction. SharedPreferences.Editor

    editor = mSharedPreferences.edit(); editor.putString(KEY_NAME, sharedPreferenceEntry.getName()); editor.putLong(KEY_DOB, sharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); editor.putString(KEY_EMAIL, sharedPreferenceEntry.getName()); // Commit changes to SharedPreferences. return editor.commit(); } Mistakes...
  17. # ./gradlew unit:BasicSample:app:testDebug com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperTest > sharedPreferencesHelper_SavePersonalInformationFailed_ReturnsFalse PASSED com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperTest > sharedPreferencesHelper_SaveAndReadPersonalInformation

    PASSED BUILD SUCCESSFUL Run the Tests!
  18. WTF?

  19. If the test fails, and you can't immediately tell why...

    The test is bad, unhelpful. If you can mess up the code under test, and the test DOESN'T fail... The test is actively harmful. Unhelpful and Harmful Tests
  20. private SharedPreferencesHelper createMockSharedPreference() { // Mocking reading the SharedPreferences as

    if mMockSharedPreferences was previously written // correctly. when(mMockSharedPreferences.getString(eq(SharedPreferencesHelper.KEY_NAME), anyString())) .thenReturn(mSharedPreferenceEntry.getName()); when(mMockSharedPreferences.getString(eq(SharedPreferencesHelper.KEY_EMAIL), anyString())) .thenReturn(mSharedPreferenceEntry.getEmail()); when(mMockSharedPreferences.getLong(eq(SharedPreferencesHelper.KEY_DOB), anyLong())) .thenReturn(mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); // Mocking a successful commit. when(mMockEditor.commit()).thenReturn(true); // Return the MockEditor when requesting it. when(mMockSharedPreferences.edit()).thenReturn(mMockEditor); return new SharedPreferencesHelper(mMockSharedPreferences); } Why Does it Fail?
  21. @Test public void savesPersonalInformation() { // Given sharedPrefsSaveWillSucceed(); // When

    boolean saveSuccess = sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then verify(mMockEditor).putString(KEY_NAME, mSharedPreferenceEntry.getName()); verify(mMockEditor).putString(KEY_EMAIL, mSharedPreferenceEntry.getEmail()); verify(mMockEditor).putLong(KEY_DOB, mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); verify(mMockEditor).commit(); assertThat(saveSuccess, is(true)); } @Test public void readsPersonalInformation() { // Given givenSharedPrefsContains(mSharedPreferenceEntry); // When SharedPreferenceEntry retrievedInfo = sharedPreferencesHelper.getPersonalInfo(); // Then assertThat(retrievedInfo.getName(), is(equalTo(mSharedPreferenceEntry.getName()))); assertThat(retrievedInfo.getDateOfBirth(), is(equalTo(mSharedPreferenceEntry.getDateOfBirth()))); assertThat(retrievedInfo.getEmail(), is(equalTo(mSharedPreferenceEntry.getEmail()))); } Split the Test, Check What Matters
  22. # ./gradlew unit:BasicSample:app:testDebug com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperTest > savesPersonalInformation FAILED org.mockito.exceptions.verification.junit.ArgumentsAreDifferent at SharedPreferencesHelperTest.java:94

    # ./gradlew unit:BasicSample:app:testDebug --info com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperTest > savesPersonalInformation FAILED Argument(s) are different! Wanted: mMockEditor.putString( "key_email", "test@email.com" ); -> at com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperTest.savesPersonalInformation(Sh aredPreferencesHelperTest.java:94) Actual invocation has different arguments: mMockEditor.putString( "key_email", "Test name" ); -> at ... Run the Tests!
  23. @Test public void savesPersonalInformation() { // Given sharedPrefsSaveWillSucceed(); // When

    boolean saveSuccess = sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then verify(mMockEditor).putString(KEY_NAME, mSharedPreferenceEntry.getName()); verify(mMockEditor).putString(KEY_EMAIL, mSharedPreferenceEntry.getEmail()); verify(mMockEditor).putLong(KEY_DOB, mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); verify(mMockEditor).commit(); assertThat(saveSuccess, is(true)); } The Save Test
  24. @Test public void reportsSuccessSavingPersonalInformation() { // Given sharedPrefsSaveWillSucceed(); // When

    boolean saveSuccess = sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then assertThat(saveSuccess, is(true)); } @Test public void savesPersonalInformation() { // Given sharedPrefsSaveWillSucceed(); // When sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then verify(mMockEditor).putString(KEY_NAME, mSharedPreferenceEntry.getName()); verify(mMockEditor).putString(KEY_EMAIL, mSharedPreferenceEntry.getEmail()); verify(mMockEditor).putLong(KEY_DOB, mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); verify(mMockEditor).commit(); } Split It!
  25. @Test public void savesName() { // When sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then

    verify(mMockEditor).putString(KEY_NAME, mSharedPreferenceEntry.getName()); verify(mMockEditor).commit(); } @Test public void savesEmail() { // When sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then verify(mMockEditor).putString(KEY_EMAIL, mSharedPreferenceEntry.getEmail()); verify(mMockEditor).commit(); } @Test public void savesDateOfBirth() { // When sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then verify(mMockEditor).putLong(KEY_DOB, mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); verify(mMockEditor).commit(); } Split... MORE!
  26. @Test public void savesName() { // When sharedPreferencesHelper.savePersonalInfo(mSharedPreferenceEntry); // Then

    verifySavedAndCommitted(KEY_NAME, mSharedPreferenceEntry.getName()); } @Test public void savesEmail() { ... verifySavedAndCommitted(KEY_EMAIL, mSharedPreferenceEntry.getEmail()); } @Test public void savesDateOfBirth() { ... verifySavedAndCommitted(KEY_DOB, mSharedPreferenceEntry.getDateOfBirth().getTimeInMillis()); } private void verifySavedAndCommitted(final String keyName, final String value) { verify(mMockEditor).putString(keyName, value); verify(mMockEditor).commit(); } private void verifySavedAndCommitted(String keyName, long value) { ... } Less Duplication, More Readable
  27. @Test public void readsPersonalInformation() { // Given givenSharedPrefsContains(mSharedPreferenceEntry); // When

    SharedPreferenceEntry retrievedInfo = sharedPreferencesHelper.getPersonalInfo(); // Then assertThat(retrievedInfo.getName(), is(equalTo(mSharedPreferenceEntry.getName()))); assertThat(retrievedInfo.getDateOfBirth(), is(equalTo(mSharedPreferenceEntry.getDateOfBirth()))); assertThat(retrievedInfo.getEmail(), is(equalTo(mSharedPreferenceEntry.getEmail()))); } The Read Test
  28. @Test public void readsPersonalInformation() { // Given givenSharedPrefsContains(mSharedPreferenceEntry); // When

    SharedPreferenceEntry retrievedPersonalInfo = sharedPreferencesHelper.getPersonalInfo(); // Then assertThat(retrievedPersonalInfo, matchesEntry(mSharedPreferenceEntry)); } Single Assertion (Hamcrest Matchers)
  29. public class SharedPreferenceEntryMatchers { public static Matcher<? super SharedPreferenceEntry> matchesEntry(final

    SharedPreferenceEntry entry) { return allOf( matchesNameOf(entry), matchesEmailOf(entry), matchesDateOfBirthOf(entry) ); } private static FeatureMatcher<SharedPreferenceEntry, String> matchesNameOf(final SharedPreferenceEntry entry) { return new FeatureMatcher<SharedPreferenceEntry, String>(equalTo(entry.getName()), "name", "name") { @Override protected String featureValueOf(final SharedPreferenceEntry sharedPreferenceEntry) { return sharedPreferenceEntry.getName(); } }; } ... } Matchers Caveat...
  30. • Natural, expressive; read many x more than write. •

    SNR • Assert / check / test one thing. Matchers can help. • Test boundaries, don't poke internals (package scope included) Should be able to change implementation and tests still pass. Should NOT be able to BREAK implementation and still have tests pass. Recap: Top Testing Tips
  31. • Code modification >>> creation → save time each time

    test fails, is looked at. • Unit tests lowest level → failure must be clear why and where. • Good failures: • test name • message, info given • test one thing Failure
  32. • Spock Framework • https:/ /github.com/spockframework/spock • http:/ /spockframework.org/ •

    No logo yet... • Decent documentation... on Google Code... Going Further... Logically IMAGINE A FANTASTIC LOGO HERE!
  33. def "subscribers receive published events at least once"() { when:

    publisher.send(event) then: (1.._) * subscriber.receive(event) where: event << ["started", "paused", "stopped"] } def "can cope with misbehaving subscribers"() { given: firstSubscriber.receive(_) >> { throw new Exception() } when: publisher.send("event1") publisher.send("event2") then: 1 * latterSubscriber.receive("event1") 1 * latterSubscriber.receive("event2") } Some Examples
  34. def "reads personal info"() { given: sharedPrefsContains(entry) when: def retrievedEntry

    = sharedPreferencesHelper.getPersonalInfo() then: expect retrievedEntry, matchesEntry(entry) } Spock Hamcrest Support
  35. def "reads personal info"() { given: mockSharedPrefs.has(expectedEntry) // extension method

    when: def entry = sharedPreferencesHelper.getPersonalInfo() then: resulting entry matches expectedEntry // resulting(entry).matches(expectedEntry) } Groovy DSL (+ Metaprogramming) Magic
  36. com.example.android.testing.unittesting.BasicSample.SharedPreferencesHelperSpec > reads personal info FAILED Condition not satisfied: val.email

    == expected.email | | | | | | | | | email@email.com | | | com.example.android.testing.unittesting.BasicSample.SharedPreferenceEntry@71ea020a | | false | | 9 differences (62% similarity) | | email@email.com(_AN_ERROR) | | email@email.com(---------) | email@email.com_AN_ERROR com.example.android.testing.unittesting.BasicSample.SharedPreferenceEntry@1e6abe27 at com.example..SharedPreferencesHelperSpec.resulting_closure2(SharedPreferencesHelperSpec.groovy:122) at com.example..SharedPreferencesHelperSpec.reads personal info(SharedPreferencesHelperSpec.groovy:87) With Amazing Failure Info!
  37. def "uses spy"() { when: thing.doSomethingWith(collaborator) then: 1 * collaborator.callWeExpect()

    } def "uses stub"() { given: stubRandomNumberGenerator.getValue() >> 4 // chosen at random when: def result = thing.doThingWith(stubRandomNumberGenerator) then: result == 42 } Wildcards and ranges: _.receive(event) // receive is called on any mock object 0 * _._ // No more interactions 0 * _ // Same thing 3 * subscriber.receive(event) // exactly 3 times (1.._) * subscriber.receive(event) // at least 1 time (_..2) * subscriber.receive(event) // at most 2 times 0 * _./set.*/(_) // no setter is called on any mock Mocking, Stubbing, Spying
  38. pickySubscriber.receive(_) >> { Event event -> event.priority > 3 ?

    "ok" : "fail" } badSubscriber.receive(_) >> { throw new InternalError("I'm a troublemaker!") } subscriber.receive(_) >>> ["ok", "fail", "ok"] >> { throw new InternalError() } >> "ok" 1 * subscriber.receive(!null) // arg must not be null 1 * subscriber.receive(!unwantedEvent) // arg must not be equal to unwantedEvent 1 * subscriber.receive( { it.priority >= 5 } ) // arg must have priority 5 or above More: https:/ /spockframework.github.io/spock/docs/1.0/interaction_based_testing.html More Fakery, Argument Constraints
  39. @Unroll def "maximum of #a and #b is #result"() {

    expect: Math.max(a, b) == result where: a | b || result 0 | 0 || 0 0 | 9 || 9 -2 | 1 || 3 } ....SharedPreferencesHelperSpec > maximum of -2 and 1 is 3 FAILED Condition not satisfied: Math.max(a, b) == result | | | | | 1 -2 1 | 3 false at com.example..MyClassSpec.maximum of #a and #b is #result(SharedPreferencesHelperSpec.groovy:93) Data Driven Tests
  40. when: thing.addValue(DUMMY_VALUE) then: thing.previousValue == old(thing.newestValue) Goddamn magic How? What?

    Huh?
  41. • Groovy-based • “Kind of Java” • Expressive DSL abilities

    (http:/ /docs.groovy-lang.org/docs/latest/html/documentation/core-domain-specific-languages.html) • Freakin' Magic • Built-in expressive mocking • Readable, unrolling, data-driven tests (https:/ /spockframework.github.io/spock/docs/1.0/data_driven_testing.html) Spock Features
  42. None
  43. • Similar to unit tests, but wider scope • Test

    domain model satisfies business logic • May test interaction of many units. Focused on use cases instead. • Still off-device (fast), isolate domain from platform Domain / “Service” / API Tests
  44. None
  45. • Why? • Challenges • Tips UI Tests

  46. • Obvious business value, on complete product • Test UI

    (shallow) • Test core use cases, main interactions (deep) • Test specific bugs (fix, prevent regressions) • Link to BDD, conversation → spec BUT: • Can end up with too many, taking too long and breaking too often Why We Write UI Tests
  47. • Speed (lack of) • Synchronisation issues • Unclear failure

    causes • Brittleness • Readability UI Test Challenges
  48. • Avoid Thread.sleep(millis), -> Espresso synch & IdlingResource • Developer

    machine: • Genymotion, x86(_64) > ARM • Many emulators & test sharding (e.g. Fork by Shazam) • On CI: • Many devices (also sharding) Fixes – Slow / Synchronisation Issues
  49. Custom failure handler: BetterFailureHandler failureHandler = new BetterFailureHandler(); Espresso.setFailureHandler(failureHandler); Spoon

    screenshot: Spoon.screenshot(activity, sanitise(tag)); UiDevice screenshot: final Instrumentation instrumentation = InstrumentationRegistry.getInstrumentation(); final Context targetContext = InstrumentationRegistry.getTargetContext(); final UiDevice uiDevice = UiDevice.getInstance(instrumentation); final File screenshotFile = new File(testScreenshotsDir, screenshotName); uiDevice.takeScreenshot(screenshotFile); Fixes – Unclear Failure Causes
  50. • Slow • Synchronisation issues • Unclear failure reason •

    Readability • Brittleness • Flakiness / reproducibility UI Test Challenges
  51. @Test public void changeText() { // Type text and then

    press the button. onView(withId(R.id.editTextUserInput)) .perform(typeText(STRING_TO_BE_TYPED), closeSoftKeyboard()); onView(withId(R.id.changeTextBt)).perform(click()); // Check that the text was changed. onView(withId(R.id.textToBeChanged)).check(matches(withText(STRING_TO_BE_TYPED))); } Readability: A Bitter Little Espresso Comments because code isn't readable enough Overly verbose, backward English Could do with a better name
  52. @Test public void submittingTextChangesTextDisplayed() { type(DUMMY_STRING).into(inputTextField()).then(closeSoftKeyboard()); clickOn(submitButton()); checkThat(displayText(), hasText(STRING_TO_BE_TYPED)); }

    Work In Progress, please contribute! https:/ /github.com/jonreeve/espresso-sugar Readability: Adding Some Sugar
  53. • android-spock • https:/ /github.com/pieces029/android-spock • Runs on device •

    Uses Spock, Groovy • Trade-off: • DSL gets better! • Auto-completion gets worse... (at least for now) Readability: Spock, Again!
  54. @Test public void showsWelcomeOnFirstLogin() { type(USERNAME).into(view(withId(R.id.username_text))); type(PASSWORD).into(view(withId(R.id.password_text))); clickOn(view(withId(R.id.login_button))); checkThat(view(withId(R.id.welcome)), isDisplayed());

    } OR... @Test public void showsWelcomeOnFirstLogin() { final MainScreen mainScreen = onLoginScreen().loginAs(FIRST_TIME_USER); checkThat(mainScreen.welcomeMessage(), isDisplayed()) } Readability + Brittleness: Page / Screen Models
  55. @Test public void showsWelcomeOnFirstLogin() { when(FIRST_TIME_USER).logsIn(); then(welcomeMessage(), isDisplayed()) } Also

    see Gwen, by Shazam: https:/ /github.com/shazam/gwen Readability + Brittleness: Actor Models, Other
  56. "Hermetic" test environment Magical, mystical. Astrology, Alchemy... Completely sealed off

    (as if by magic) Identify sources of unpredictability, and Fake them (wrappers, mocking, hexagonal architecture, etc.) Flakiness / Reliability
  57. No such thing! But... • Is it worth it? •

    Is there a better level to test it at? • Is there a better place to introduce a seam and isolate? • Pairing and rubber-ducking helps maintain perspective Untestable?
  58. • Speed: Espresso, many emulators / devices + sharding (Fork)

    • Make tests readable! Page / actor models, DSL, android-spock • Make failures clear, add useful info + screenshots • Flakiness: Define boundaries, fake the unpredictable! • Hard to test? Best way to test? Pairing, rubber-ducking, etc. Recap: UI Testing Tips
  59. Jon Reeve Wasabi Code @themightyjon Thank you! droidcon Greece Thessaloniki