Android API Integration Tests with Mocked Endpoints Chris Koeberle @kodi Bottle Rocket bit.ly/cdk-mock-pdf

Agenda • Problem statement • Requirements • Testing network calls • Mocking network responses

The Problem • Android apps fetch data from HTTP endpoints to display it on the device • If backend systems are designed with mobile in mind, this can be straightforward and easy • When working with one or more legacy systems, just constructing the request and parsing the response can be a challenge • Reducing the iteration time when trying to diagnose weird hijinks is paramount

Solutions • The worst way to do this is to run the app • Tools like Postman, Charles, and Fiddler can help solve problems but don’t leave you with working code • Integration tests allow you to test the full stack without worrying about UI or waiting to install the APK • In principle, you can have the entire networking layer proved out without a single Activity • In order for the integration tests to have ongoing value, they must be able to distinguish between a local failure and a remote failure • If we’re calling an endpoint that has side effects, we might want to limit how often we actually call the endpoint

Requirements • Dependency Injection or Service Locator • OkHttp • Some way of getting off the main thread • If they way you get off the main thread has problems with the Looper, you might need to mock your own Looper • Patience dependencies {
 compile 'com.android.support:appcompat-v7:25.0.1'
 compile fileTree(dir: 'libs', include: ['*.jar'])
 testCompile fileTree(include: ['*.jar'], dir: 'testlibs')
 testCompile 'junit:junit:4.12'
 testCompile 'org.mockito:mockito-core:1.10.19'
 compile ‘io.reactivex.rxjava2:rxandroid:2.0.1'
 compile ‘io.reactivex.rxjava2:rxjava:2.1.1'
 compile 'com.bottlerocketstudios:groundcontrol:1.1.3'
 compile ‘com.squareup.okhttp3:okhttp:3.8.1'
 compile ‘com.google.code.gson:gson:2.8.0’
 compile 'joda-time:joda-time:2.7'
 compile 'com.jakewharton.timber:timber:4.4.0'
 }

The Example • Most confusing, archaic legacy systems are private • It’s harder to see what’s going on when you’re looking at something confusing, anyway • Being able to call POST endpoints that create content generally requires login • Login will also make it harder to see the important stuff • So let’s look at Github, which will let us create an anonymous gist • This simpler example covers most of the difficulties I’ve faced in supporting a hundred different endpoints • Follow along at bit.ly/cdk-mock-git

Dependency Injection • Dependency injection allows us to change how things are mocked in tests on the fly. • This example uses ServiceLocator pattern instead of dependency injection public class ServiceLocator {
 Map, Object> mLocatorMap;
 
 private ServiceLocator() {
 mLocatorMap = new HashMap<>();
 }
 
 private static class SingletonHolder {
 static final ServiceLocator instance = new ServiceLocator();
 }
 
 private static ServiceLocator getInstance() {
 return SingletonHolder.instance;
 }
 
 public static void put(Class type, T instance) {
 if (type == null) { throw new NullPointerException(); }
 getInstance().mLocatorMap.put(type, instance);
 }
 
 public static T get(Class type) {
 return (T) getInstance().mLocatorMap.get(type);
 }
 } public class ServiceInjector {
 public static T resolve(Class extends T> type) {
 return ServiceLocator.get(type);
 }
 } ServiceLocator.put(RxEndpoints.class, new RxEndpointsImpl()); Flowable flowable = ServiceInjector .resolve(RxEndpoints.class) .getUser("bottlerocketapps");

OkHttp • Need to set up OkHttpClient so it can be injected • This is a good practice for other reasons - TLS support on 4.4, etc. public class OkHttpClientUtil {
 private static final long READ_TIMEOUT = 120;
 
 public static OkHttpClient getOkHttpClient(Context context, MockBehavior mock) {
 OkHttpClient okHttpClient = null;
 OkHttpClient.Builder builder = new OkHttpClient.Builder();
 if(mock != MockBehavior.DO_NOT_MOCK) {
 builder.addInterceptor(new MockedApiInterceptor(context, mock));
 }
 okHttpClient = builder
 .readTimeout(READ_TIMEOUT, TimeUnit.SECONDS)
 .retryOnConnectionFailure(false)
 .build();
 return okHttpClient;
 }
 } … ServiceLocator.put(OkHttpClient.class, OkHttpClientUtil.getOkHttpClient(null, MockBehavior.MOCK)); … subscriber.onNext(ServiceInjector.resolve(OkHttpClient.class).newCall(request).execute());

RxJava • RxJava is very flexible. This isn’t the only way; it’s likely not the best way. • This uses RxJava 2.1.1 private Flowable getResponse(final HttpUrl url) { return Flowable.fromCallable(new Callable() { @Override public Response call() throws Exception { System.out.println(url); Request request = ServiceInjector.resolve(ServiceConfiguration.class).getRequestBuilder() .url(url) .build(); return ServiceInjector.resolve(OkHttpClient.class).newCall(request).execute(); } }); } @Override
 public Flowable getUser(String userName) {
 HttpUrl url = ServiceInjector.resolve(ServiceConfiguration.class).getUrlBuilder()
 .addPathSegment(USER)
 .addPathSegment(userName)
 .build();
 return getResponse(url)
 .flatMap(new FetchString())
 .flatMap(new ToJson(UserImpl.class));
 }

RxJava • These functions support the flatMaps on the previous page private class FetchString implements Function> { @Override public Flowable apply(final Response response) { return Flowable.fromCallable(new Callable() { @Override public String call() throws Exception { if (!response.isSuccessful()) { throw new IOException(response.message()); } String responseString = response.body().string(); System.out.println(responseString); return responseString; } }); } } private class ToJson implements Function> { private final Class mTargetClass; private ToJson(Class mTargetClass) { this.mTargetClass = mTargetClass; } @Override public Flowable apply(final String s) { return Flowable.fromCallable(new Callable() { @Override public T call() throws Exception { return (T) ServiceInjector.resolve(Gson.class).fromJson(s, mTargetClass); } }); } }

Testing the Observable • The only testing framework we need is Junit • We have a few dependencies that stay the same for all the tests • So we’ll have a base test that sets those up public class BaseApiTest {
 @Before
 public void setup() {
 ServiceLocator.put(RxEndpoints.class, new RxEndpointsImpl());
 ServiceLocator.put(ServiceConfiguration.class, new ServiceConfigurationImpl());
 ServiceLocator.put(Gson.class, GsonUtil.getGson());
 }
 } • RxEndpoints is our collection of RxJava observable creators • ServiceConfiguration would allow us to change to a staging environment • Gson can be frustrating, but it makes setting up examples very fast!

Testing the Observable • Each test injects its own OkHttpClient so it can control mocking • This is not a unit test; just validate enough to know that the response makes sense. • Here, for example, we just make sure that: • The call completed • We found one organization with the orgName bottlerocketstudios • Its name is Bottle Rocket Studios @Test public void testOrganization() { ServiceLocator.put(OkHttpClient.class, OkHttpClientUtil.getOkHttpClient(null, MockBehavior.MOCK)); Flowable flowable = ServiceInjector.resolve(RxEndpoints.class).getOrg("bottlerocketstudios"); TestSubscriber testSubscriber = new TestSubscriber<>(); flowable.subscribe(testSubscriber); testSubscriber.assertComplete(); List orgList = testSubscriber.values(); assertEquals(orgList.size(), 1); assertEquals(orgList.get(0).getName(), "Bottle Rocket Studios"); }

Testing the Observable • Sometimes, we need to chain calls - retrieving a list of items, then fetching detail about one item • Here, for example, we get a list of gists, then we retrieve detail for the first one in the list and expect it to have the same description • Again, the goal here is not to prove anything about the objects, but to validate that we’re calling the endpoints successfully public class GistTest extends BaseApiTest { @Test public void testAllGists() { ServiceLocator.put(OkHttpClient.class, OkHttpClientUtil.getOkHttpClient(null, MockBehavior.MOCK)); Flowable flowable = ServiceInjector.resolve(RxEndpoints.class).getGists(); TestSubscriber testSubscriber = new TestSubscriber<>(); flowable.subscribe(testSubscriber); testSubscriber.assertComplete(); List gists = testSubscriber.values(); Gist gist = gists.get(0)[0]; Flowable gistFlowable = ServiceInjector.resolve(RxEndpoints.class).getGist(gist.getId()); TestSubscriber gistTestSubscriber = new TestSubscriber<>(); gistFlowable.subscribe(gistTestSubscriber); Gist detailGist = (Gist) gistTestSubscriber.values().get(0); assertEquals(detailGist.getDescription(), gist.getDescription()); } }

Testing the Observable • What if we make a call with side effects? • Every time I run this test, it will create a new, identical gist • Eventually, GitHub will flag my entire IP for spamming! private static final String CREATE_FILE_NAME = "AbstractMockedInterceptor.java"; private static final String CREATE_DESCRIPTION = "An OkHttp Interceptor that returns mocked results if it has them."; @Test public void createGist() throws IOException { ServiceLocator.put(OkHttpClient.class, OkHttpClientUtil.getOkHttpClient(null, MockBehavior.MOCK_ONLY)); Gist gist = new GistImpl(); gist.setDescription(CREATE_DESCRIPTION); gist.addFile(CREATE_FILE_NAME, readFromAsset("mocks/javaclass")); Flowable flowable = ServiceInjector.resolve(RxEndpoints.class).createGist(gist); TestSubscriber testSubscriber = new TestSubscriber<>(); flowable.subscribe(testSubscriber); testSubscriber.assertComplete(); List gistList = testSubscriber.values(); Gist resultGist = gistList.get(0); Flowable gistFlowable = ServiceInjector.resolve(RxEndpoints.class).getGist(resultGist.getId()); TestSubscriber gistTestSubscriber = new TestSubscriber<>(); gistFlowable.subscribe(gistTestSubscriber); Gist detailGist = gistTestSubscriber.values().get(0); assertEquals(detailGist.getDescription(), CREATE_DESCRIPTION); }

A Mocked Interceptor • http://tinyurl.com/cdk-mock public abstract class AbstractMockedApiInterceptor implements Interceptor {
 public AbstractMockedApiInterceptor(Context context, MockBehavior mock) {
 if (context != null) { mContext = context.getApplicationContext(); } else { mContext = null; }
 mMockBehavior = mock;
 }
 
 @Override
 public Response intercept(Chain chain) throws IOException {
 Request request = chain.request();
 NetworkCallSpec mockFound = null;
 for (NetworkCallSpec spec : mResponseList) {
 if (spec.matches(request.url(), request.method(), stringifyRequestBody(request))) {
 mockFound = spec;
 Response.Builder builder = new Response.Builder();
 String bodyString = resolveAsset("mocks/"+spec.mResponseFilename);
 bodyString = substituteStrings(bodyString, request);
 if(bodyString != null) {
 ResponseBody body = ResponseBody.create(MediaType.parse(getMockedMediaType()), bodyString);
 builder = builder.body(body).request(request).protocol(Protocol.HTTP_1_1).code(spec.mResponseCode).message(spec.mResponseMessage);
 }
 if (!ignoreExistingMocks()) {
 noteThatThisFileWasUsed(spec.mResponseFilename);
 return builder.build();
 }
 }
 }
 if (fetchNetworkResults()) {
 Response response = chain.proceed(request);
 response = memorializeRequest(request, response, mockFound);
 return response;
 }
 throw new IOException("Unable to handle request with current mocking strategy");
 }
 
 protected boolean fetchNetworkResults() { return mMockBehavior != MockBehavior.MOCK_ONLY; }
 protected boolean ignoreExistingMocks() { return mMockBehavior == MockBehavior.LOG_ONLY; }
 protected String getMockedMediaType() { return "json"; }
 }

A Mocked Interceptor • The basic flow: • Iterate through the list of response specifications • If one matches, build a response from it and return it • If none matches, make the network call and save it for future use public abstract class AbstractMockedApiInterceptor implements Interceptor {
 @Override
 public Response intercept(Chain chain) throws IOException {
 Request request = chain.request();
 NetworkCallSpec mockFound = null;
 for (NetworkCallSpec spec : mResponseList) {
 if (spec.matches(request.url(), request.method(), stringifyRequestBody(request))) {
 mockFound = spec;
 Response.Builder builder = new Response.Builder();
 String bodyString = resolveAsset("mocks/"+spec.mResponseFilename);
 bodyString = substituteStrings(bodyString, request);
 if(bodyString != null) {
 ResponseBody body = ResponseBody.create(MediaType.parse(getMockedMediaType()), bodyString);
 builder = builder.body(body).request(request).protocol(Protocol.HTTP_1_1).code(spec.mResponseCode).message(spec.mResponseMessage);
 }
 if (!ignoreExistingMocks()) {
 noteThatThisFileWasUsed(spec.mFilename);
 return builder.build();
 }
 }
 }
 if (fetchNetworkResults()) {
 Response response = chain.proceed(request);
 response = memorializeRequest(request, response, mockFound);
 return response;
 }
 throw new IOException("Unable to handle request with current mocking strategy");
 }
 }

Matching Requests public static class NetworkCallSpec {
 private final String mRequestUrlPattern;
 private String mRequestMethod;
 private Map mRequestQueryParameters;
 private String mRequestBody;
 private Set mRequestBodyContains;
 private int mResponseCode;
 private String mResponseMessage;
 private final String mResponseFilename;
 
 public boolean matches(HttpUrl url, String method, String body) {
 if (!url.encodedPath().matches(mRequestUrlPattern)) { return false; }
 if (!mRequestMethod.equalsIgnoreCase(method)) { return false; }
 if (mRequestMethod.equalsIgnoreCase("POST") && !TextUtils.isEmpty(mRequestBody) && !mRequestBody.equalsIgnoreCase(body)) {
 return false;
 }
 … • NetworkCallSpec defines what requests will match a given response • Start with the easiest, fastest cases - if the url doesn’t match, if the method doesn’t match, if we require a particular POST body

Matching Requests • Then do the more expensive checks - if we’re looking for a string inside the POST body, or if we need to find particular parameters • Header values can use regular expressions for matching … for (String contains : mRequestBodyContains) {
 if (!body.contains(contains)) {
 return false;
 }
 }
 }
 for (Map.Entry kvp : mRequestQueryParameters.entrySet()) {
 boolean foundKey = false;
 boolean foundValue = false;
 for (String key : url.queryParameterNames()) {
 if (key.matches(kvp.getKey())) {
 foundKey = true;
 String value = url.queryParameter(key);
 if (value != null && value.matches(kvp.getValue())) {
 foundValue = true;
 }
 if (value == null && (kvp.getValue() == null || kvp.getValue() == "" || kvp.getValue().equalsIgnoreCase("null"))) {
 foundValue = true;
 }
 }
 }
 if (!foundKey || !foundValue) {
 return false;
 }
 }
 return true;
 }

Building Responses • Once we have identified a mocked response, we do our best to make it look like it originally did - within the limits of what our app cares about • If we’re running as a JUnit test, we don’t have a context, and we read from test/resources/mocks/ • If we’re running in the app, we read from assets/mocks/ • Use build.gradle Copy task to copy from assets/ into test/resources/ Response.Builder builder = new Response.Builder();
 String bodyString = resolveAsset("mocks/"+spec.mFilename);
 bodyString = substituteStrings(bodyString, request);
 if(bodyString != null) {
 ResponseBody body = ResponseBody.create(MediaType.parse(getMockedMediaType()), bodyString);
 builder = builder.body(body).request(request).protocol(Protocol.HTTP_1_1).code(spec.mResponseCode).message(spec.mResponseMessage);
 }
 if (!ignoreExistingMocks()) {
 noteThatThisFileWasUsed(spec.mFilename);
 return builder.build();
 } private String resolveAsset(String filename) {
 if (mContext != null) {
 return getAssetAsString(mContext, filename);
 } else {
 try {
 return readFromAsset(filename);
 } catch (IOException e) {
 Timber.e(e, "Error reading from asset - this should only be called in tests.");
 }
 }
 return null;
 }


Saving New Mocks • When we retrieve a new file, we save it so we can mock it • Build up the code to register the NetworkCallSpec, and save off the file • Add everything that could be used to specify the endpoint, and then manually delete things that don’t make sense private Response memorializeRequest(Request request, Response response, NetworkCallSpec mockFound) {
 Response.Builder newResponseBuilder = response.newBuilder();
 try {
 String responseString = response.body().string();
 List segments = request.url().encodedPathSegments();
 String endpointName = segments.get(segments.size() - 1);
 String callSpecString = "mResponseList.add(new NetworkCallSpec(\""+request.url().encodedPath()+"\", \"::REPLACE_ME::\")";
 if (response.code() != HttpURLConnection.HTTP_OK) {
 callSpecString += ".setResponseCode("+response.code()+")";
 endpointName += "-"+response.code();
 }
 if (!TextUtils.isEmpty(response.message()) && !response.message().equalsIgnoreCase("OK")) {
 callSpecString += ".setResponseMessage(\""+response.message()+"\")";
 }
 if (!request.method().equalsIgnoreCase("GET")) {
 callSpecString += ".setRequestMethod(\""+request.method()+"\")";
 endpointName += "-"+request.method();
 }
 if (request.url().querySize()>0) {
 for (String key : request.url().queryParameterNames()) {
 callSpecString += ".addRequestQueryParameter(\""+key.replace("[", "\\\\[").replace("]", "\\\\]")+"\", \""+request.url().queryParameter(key)+"\")";
 }
 }
 … mResponseList.add(new NetworkCallSpec("/users/bottlerocketstudios", "bottlerocketstudios"));

Saving New Mocks • Save the entire body of a POST request, even though it’s not a good practice to try to match on the whole body - at least it’s available • Make the name unique so that the new file won’t overwrite something • If we’re on device, write to our FilesDir. In test, it just goes to the app root … String body = stringifyRequestBody(request);
 if (body != null) {
 callSpecString += ".addRequestBody(\""+body.replace("\"", "\\\"").replace("\\u003d", "\\\\u003d")+"\")";
 endpointName += "-"+body.hashCode();
 }
 requestSpecString += ");";
 if (endpointName.length()>100) {
 endpointName = ""+endpointName.hashCode();
 }
 endpointName = getUniqueName(endpointName);
 callSpecString = callSpecString.replace("::REPLACE_ME::", endpointName);
 if (mockFound != null) {
 callSpecString += " // duplicate of existing mock "+mockFound.mPattern;
 if (!TextUtils.isEmpty(mockFound.mRequestBody)) {
 callSpecString += " with body "+mockFound.mRequestBody;
 }
 }
 callSpecString += "\n";
 writeToFile(callSpecString, responseString, endpointName);
 newResponseBuilder.body(ResponseBody.create(response.body().contentType(), responseString));
 } catch (IOException e) {
 Timber.e("Unable to save request to "+request.url().toString()+" : ", e);
 }
 return newResponseBuilder.build();
 }

Substitutions • Sometimes, the response must be modified before being returned, for example, the expiration date of a token must be modified to be in the future • NetworkCallSpec can include a substitution pattern that will make these replacements private static interface StringSubstitutor {
 String replaceOneString(String body, Request request);
 boolean matchesFound(String body);
 } private String substituteStrings(String bodyString, Request request) {
 // Because each match can get replaced with something different, we have to reset the matcher after every replacement.
 // This way of doing things happens to enforce this in a non-obvious way, because we create a new matcher every time.
 for (StringSubstitutor substitutor : mSubstitutorList) {
 while (substitutor.matchesFound(bodyString)) {
 bodyString = substitutor.replaceOneString(bodyString, request);
 }
 }
 return bodyString;
 } private static final Pattern DATE = Pattern.compile(“%DATE[^%]*%"); private static class DateSubstitutor implements StringSubstitutor {
 @Override
 public String replaceOneString(String body, Request request) {
 Matcher dateMatcher = DATE.matcher(body);
 dateMatcher.find();
 String match = dateMatcher.group();
 Map query = getQueryFromUri(match);
 LocalDate date = new LocalDate();
 if(query.containsKey(OFFSET_PARAMETER)) {
 date = date.plusDays(Integer.parseInt(query.get(OFFSET_PARAMETER)));
 }
 body = dateMatcher.replaceFirst(date.toString());
 return body;
 } }

Back to the Test with Side Effects • Now, we set up this test to run MockBehavior.MOCK_ONLY • If the Interceptor finds a mock, it will return that mock • If not, it will return an error private static final String CREATE_FILE_NAME = "AbstractMockedInterceptor.java";
 private static final String CREATE_DESCRIPTION = "An OkHttp Interceptor that returns mocked results if it has them.";
 @Test
 public void createGist() throws IOException {
 ServiceLocator.put(OkHttpClient.class, OkHttpClientUtil.getOkHttpClient(null, MockBehavior.MOCK_ONLY));
 Gist gist = new GistImpl();
 gist.setDescription(CREATE_DESCRIPTION);
 gist.addFile(CREATE_FILE_NAME, readFromAsset("mocks/javaclass"));
 Observable observable = ServiceInjector.resolve(RxEndpoints.class).createGist(gist);
 TestSubscriber testSubscriber = new TestSubscriber<>();
 observable.subscribe(testSubscriber);
 testSubscriber.assertCompleted();
 List gistList = testSubscriber.getOnNextEvents();
 Gist resultGist = gistList.get(0);
 Observable gistObservable = ServiceInjector.resolve(RxEndpoints.class).getGist(resultGist.getId());
 TestSubscriber gistTestSubscriber = new TestSubscriber<>();
 gistObservable.subscribe(gistTestSubscriber);
 Gist detailGist = gistTestSubscriber.getOnNextEvents().get(0); assertEquals(detailGist.getDescription(), CREATE_DESCRIPTION);
 }

Photo Credits • Pete - Storage Array
 https://www.flickr.com/photos/comedynose/7048321621 • Copyright: Joi Ito, Attribution 2.0 Generic - Vacuum Tubes
 https://www.flickr.com/photos/joi/494429939 • Copyright: Sven, Attribution 2.0 Generic - Giant Heat Sink
 https://www.flickr.com/photos/flickrsven/2790410292 • Copyright: Andrew Magill, Attribution 2.0 Generic - Patient Dog
 https://www.flickr.com/photos/amagill/7798874692/ • Neil Dalphin - Spider
 https://www.flickr.com/photos/136758431@N05/32665099125 • Joe deSousa - Gears
 https://www.flickr.com/photos/mustangjoe/22711070429 • Bernard Spragg. NZ - Bull Kelp
 https://www.flickr.com/photos/volvob12b/9667991371 • Mark Jones - Eye
 https://www.flickr.com/photos/131211911@N03/16958804881 • Susan Young - Northern Mockingbird
 https://www.flickr.com/photos/95782365@N08/24329013583 • Bernard Spragg. NZ - Sunset on the Roof
 https://www.flickr.com/photos/volvob12b/14284648729 • Chris Koeberle - Building • Chris Koeberle - Metronome • Michael D Beckwith - Leeds Corn Exchange
 https://www.flickr.com/photos/118118485@N05/12644596004 • Smithsonian Institution - Supernova Bubble
 https://www.flickr.com/photos/smithsonian/5393241197

Questions?