Guide to the jungle of testing frameworks

Slide 1

Slide 1 text

Guide to the jungle of testing frameworks Tomáš Kypta @TomasKypta

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Unit Testing Experience on Android

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Android apps are difficult to test

Slide 6

Slide 6 text

Android apps were difficult to test

Slide 7

Slide 7 text

Types of Android tests

Slide 8

Slide 8 text

Types of Android tests Instrumentation tests Local unit tests

Slide 9

Slide 9 text

Android test code • project sources • ${module}/src/main/java • instrumentation tests • ${module}/src/androidTest/java • unit tests • ${module}/src/test/java • full Gradle and Android Studio support

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

• the essential piece of both instrumentation and unit tests • alone can be used only for pure Java code • doesn’t provide any mocks or Android APIs

Slide 12

Slide 12 text

Instrumentation Tests

Slide 13

Slide 13 text

Instrumentation Tests • running on physical device or emulator • gradle connectedAndroidTest • ${module}/build/reports/androidTests/connected/ index.html

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Instrumentation Tests Legacy instrumentation tests or Testing Support Library

Slide 16

Slide 16 text

Legacy Instrumentation Tests • JUnit3 • Tests extend from TestCase • AndroidTestCase • ActivityInstrumentationTestCase2 • ServiceTestCase • … deprecated since API level 24

Slide 17

Slide 17 text

Testing Support Library • JUnit4 compatible • AndroidJUnitRunner android { defaultConfig { testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner" } } dependencies { androidTestCompile 'com.android.support.test:runner:0.5' }

Slide 18

Slide 18 text

Testing Support Library • JUnit test rules • AndroidTestRule • ServiceTestRule • DisableOnAndroidDebug • LogLogcatRule • … androidTestCompile 'com.android.support.test:rules:0.5'

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

• framework for functional UI tests • part of Android Testing Support Library androidTestCompile 'com.android.support.test.espresso:espresso-core:2.2.2'

Slide 21

Slide 21 text

@Test public void sayHello() { onView(withId(R.id.edit_text)) .perform(typeText(STRING_TO_BE_TYPED), closeSoftKeyboard()); onView(withText("Say hello!")) .perform(click()); String expectedText = "Hello, " + STRING_TO_BE_TYPED + "!"; onView(withId(R.id.textView)) .check(matches(withText(expectedText))); }

Slide 22

Slide 22 text

Problems • testing on device is not isolated • device state affects the result • e.g. screen on/off might affect test result onView(withId(R.id.my_view)) .check(matches(isDisplayed()));

Slide 23

Slide 23 text

Some annoyances android.support.test.espresso.NoActivityResumedException: No activities in stage RESUMED. Did you forget to launch the activity. (test.getActivity() or similar)?

Slide 24

Slide 24 text

UI Automator

Slide 25

Slide 25 text

UI Automator • APIs for building UI tests • interaction with both your apps and system apps • Android 4.3+ androidTestCompile 'com.android.support.test.uiautomator:uiautomator-v18:2.1.2'

Slide 26

Slide 26 text

UI Automator • several parts • API for information retrieval and performing operations • API for cross-app testing • uiautomatorviewer tool

Slide 27

Slide 27 text

UI Automator @RunWith(AndroidJUnit4.class) public class AnActivityTest { private static final String MY_PACKAGE = "com.example.helloworld"; private static final int LAUNCH_TIMEOUT = 5000; UiDevice mDevice; @Before public void setUP() { mDevice = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation()); mDevice.pressHome(); Context context = InstrumentationRegistry.getContext(); Intent intent = context.getPackageManager().getLaunchIntentForPackage(MY_PACKAGE); intent.addFlags(Intent.FLAG_ACTIVITY_CLEAR_TASK); context.startActivity(intent); // Wait for the app to appear mDevice.wait(Until.hasObject(By.pkg(MY_PACKAGE).depth(0)), LAUNCH_TIMEOUT); } // do the testing }

Slide 28

Slide 28 text

UI Automator @RunWith(AndroidJUnit4.class) public class AnActivityTest { // a setup @Test public void testText() throws Exception { UiObject textLabel = mDevice.findObject( new UiSelector().packageName(MY_PACKAGE).text("Hello World!")); UiObject button = mDevice.findObject( new UiSelector().packageName(MY_PACKAGE).text("Change text")); button.click(); UiObject2 textResult = mDevice.findObject(By.res(MY_PACKAGE, "txt_label")); Assert.assertEquals("Hello Minsk!", textResult.getText()); } }

Slide 29

Slide 29 text

Java API

Slide 30

Slide 30 text

• use with UI Automator • direct manipulation with sensor values on Genymotion devices • allows to omit mocking sensor values androidTestCompile 'com.genymotion.api:genymotion-api:1.0.2'

Slide 31

Slide 31 text

if (!GenymotionManager.isGenymotionDevice()) { return; //don't execute this test }

Slide 32

Slide 32 text

if (!GenymotionManager.isGenymotionDevice()) { return; //don't execute this test } GenymotionManager genymotion = Genymotion.getGenymotionManager( getInstrumentation().getContext());

Slide 33

Slide 33 text

if (!GenymotionManager.isGenymotionDevice()) { return; //don't execute this test } GenymotionManager genymotion = Genymotion.getGenymotionManager( getInstrumentation().getContext()); genymotion .getRadio().call("555123456");

Slide 34

Slide 34 text

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

• for iOS, Android, Windows apps • based on WebDriver protocol • alternative to UI Automator • aims to automate apps from any language and any test framework • …with access to back-end APIs and DBs • Android 4.1+

Slide 37

Slide 37 text

Instrumentation tests are kinda SLOOOOOW

Slide 38

Slide 38 text

Unit Tests

Slide 39

Slide 39 text

Unit Tests • run on JVM • mockable android.jar • gradle test • ${module}/build/reports/tests/${variant}/index.html

Slide 40

Slide 40 text

...

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

• Helps rarely • returns 0, false, null, … Method ... not mocked. android {  testOptions {  unitTests.returnDefaultValues = true }  }

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

• mocking framework • easy to use • compatible with Android unit testing testCompile 'org.mockito:mockito-core:2.2.11'

Slide 45

Slide 45 text

• can be used also in instrumentation tests • needs dexmaker androidTestCompile 'org.mockito:mockito-core:2.2.11' androidTestCompile 'com.google.dexmaker:dexmaker:1.2' androidTestCompile 'com.google.dexmaker:dexmaker-mockito:1.2'

Slide 46

Slide 46 text

@RunWith(JUnit4.class) public class ContextManagerTest { @Mock Context mAppContext; }

Slide 47

Slide 47 text

@RunWith(JUnit4.class) public class ContextManagerTest { @Mock Context mAppContext; @Before public void setUp() { MockitoAnnotations.initMocks(this); } }

Slide 48

Slide 48 text

@RunWith(JUnit4.class) public class ContextManagerTest { @Mock Context mAppContext; @Before public void setUp() { MockitoAnnotations.initMocks(this); } @Test public void testWithContext() { … } }

Slide 49

Slide 49 text

@RunWith(MockitoJUnitRunner.class) public class ContextManagerTest { @Mock Context mAppContext; @Test public void testWithContext() { … } }

Slide 50

Slide 50 text

@RunWith(JUnit4.class) public class ContextManagerTest { @Test public void testWithContext() { Context appContext = mock(Context.class); } }

Slide 51

Slide 51 text

@RunWith(JUnit4.class) public class ContextManagerTest { @Test public void testWithContext() { Context appContext = mock(Context.class); Mockito.when(appContext.getPackageName()) .thenReturn(“com.example.app”); … } }

Slide 52

Slide 52 text

• Mockito.spy() • wrapping a real object • Mockito.verify() • verify that special condition are met • e.g. method called, method called twice, …

Slide 53

Slide 53 text

Limitations • final classes • opt-in incubating support in Mockito 2 • anonymous classes • primitive types • static methods

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

• Can mock static methods • Can be used together with Mockito

Slide 56

Slide 56 text

@RunWith(PowerMockRunner.class) @PrepareForTest(Static.class); PowerMockito.mockStatic(Static.class); Mockito.when(Static.staticMethod()) .thenReturn(value); PowerMockito.verifyStatic(Static.class);

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

• functional testing framework • runs on JVM • at first, might be difficult to use • the ultimate mock of Android APIs • provides mocks of system managers • allows custom shadows

Slide 59

Slide 59 text

• possible to use for UI testing • better to use for business logic @RunWith(RobolectricTestRunner.class) public class MyTest { … }

Slide 60

Slide 60 text

• Robolectric • RuntimeEnvironment • Shadows • ShadowApplication • ShadowLooper

Slide 61

Slide 61 text

Potential problems • difficult to search for solutions • long history of bigger API changes • many obsolete posts

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

• cross-platform BDD framework • human-like test definitions testCompile ‘junit:junit:4.12' testCompile ‘info.cukes:cucumber-java:1.2.5' testCompile 'info.cukes:cucumber-junit:1.2.5'

Slide 64

Slide 64 text

• describe the desired behaviour Feature: CoffeeMaker    Scenario: a few coffees  Given I previously had 3 coffees  When I add one coffee  Then I had 4 coffees

Slide 65

Slide 65 text

• create the mapping public class CoffeeMakerDefs {  CoffeeMaker mCoffeeMaker = new CoffeeMaker();          }

Slide 66

Slide 66 text

Slide 67

Slide 67 text

• create the mapping public class CoffeeMakerDefs {  CoffeeMaker mCoffeeMaker = new CoffeeMaker();    @Given("^I previously had (\\d+) coffees$")  public void hadCoffeesPreviously(int coffees) {  mCoffeeMaker.setCoffeeCount(coffees);  }    @When("^I add one coffee$")  public void addCoffee() {  mCoffeeMaker.addCoffee();  }      }

Slide 68

Slide 68 text

Slide 69

Slide 69 text

• place definition and mapping at the same paths! • ${module}/src/test/java/com/example/MyMapping.java • ${module}/src/test/resources/com/example/ MyDefinition.feature @RunWith(Cucumber.class)  public class RunCucumberTest {  }

Slide 70

Slide 70 text

Code Coverage

Slide 71

Slide 71 text

Code Coverage • instrumentation tests • JaCoCo • EMMA • obsolete • unit tests • JaCoCo

Slide 72

Slide 72 text

Instrumentation Tests & Code Coverage • has to be explicitly enabled • gradle createDebugCoverageReport • ${module}/build/reports/coverage/debug/index.html • ${module}/build/outputs/code-coverage/connected/$ {deviceName}-coverage.ec • doesn’t work on some devices!!! buildTypes {  debug {  testCoverageEnabled true  } }

Slide 73

Slide 73 text

JaCoCo

Slide 74

Slide 74 text

JaCoCo • enabled by default for unit tests • gradle test • generates binary report in build/jacoco • ${module}/build/jacoco/testDebugUnitTest.exec • it’s necessary to create a JacocoReport task to obtain a readable report

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

Good tests • run in any order • run in isolation • run consistently • run fast • are orthogonal

Slide 78

Slide 78 text

Rules of thumb • prefer pure Java • abstract away from Android APIs • separate business logic and UI • don’t write business logic into activities and fragments • MVP, MVVM is a way to go • try avoid static and final • use dependency injection

Slide 79

Slide 79 text

Questions?