Why testing a framework for Android is different from testing an app

"Why testing a framework for Android is different" @ droidconVIE 2016
At PSPDFKit (https://pspdfkit.com) we're building, maintaining and evolving a large PDF framework (for viewing and editing) with more than 500 Java classes, and an even bigger number of classes in native C++ code. Iterating quickly, we try to keep up a bi-weekly release cycle – along with a lot of refactorings to keep control of the code. After almost two years we are looking back at a journey of testing. With more than 1.7k distinctive tests (including unit tests, screenshot tests, UI integration tests), a full test run would take about 40 minutes and we use a lot of tricks to keep the turnaround time low. The talk will outline several of the issues that we encountered (method count limits in library test APKs, Espresso test flakiness, slow execution speeds of instrumentation tests, etc.) and how we managed to overcome them.

Transcript

1. Why testing a framework for Android is different David Schreiber-Ranner / PSPDFKit © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 1
2. PSPDFKit • A commercial library for Android, iOS, Chrome OS, Web • Enables developers to build apps with PDFs • Several wrappers • SYNCKit © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 2
3. Try our beta app: pspdfk.it/ viewer © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 3
4. Numbers, numbers, numbers • 7 developers (on the Android stack) • Version 1.0, February 2015 • 39 releases • Currently v2.5.1 © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 4
5. A couple of tests • Android • 144 local unit tests • 814 instrumentation tests • C++ • 753 unit tests © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 5
6. How to test an app? © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 6
7. Example: App on Google Play © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 7
8. Testing setup • Local unit tests • JUnit • Robolectric • Instrumentation tests • Manual testing © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 8
9. Feedback loop • People using your app • Crash reports • Analytics • Feedback © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 9
10. How to test a library? © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 10
11. Example: Library via Maven repository © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 11
12. Testing setup • Local unit tests • JUnit • Robolectric • Instrumentation tests • Manual testing © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 12
13. Feedback loop • No immediate end-users • No runnable app • No analytics • Feedback © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 13
14. Second-hand feedback © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 14
15. Recipe for library testing Cooking with 4 ingredients © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 15
16. Ingredient 1 Write an app © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 16
17. Write an app! • Write an app that uses your library • Add a sample/ app to your project • Use it for development • Use it as documentation © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 17
18. Ingredient 2 Create a test app! © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 18
19. Usual structure my-project/ ├── library/ | ├── main/ | ├── test/ | └── androidTest/ ... © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 19
20. Resulting apk library-debug-tests.apk ├── classes.dex ├── assets/ ├── res/ ├── lib/ ... © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 20
21. Problems with that approach © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 21
22. ! 65536 © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 22
23. ! ... hmm, interesting. final Context context = InstrumentationRegistry.getContext(); final Context targetContext = InstrumentationRegistry.getTargetContext(); © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 23
24. Improved structure my-project/ ├── library/ | ├── main/ | └── test/ ├── library-tests/ | ├── main/ | └── androidTest/ ... © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 24
25. Resulting apk library-debug.apk ├── classes.dex ├── assets/ ├── res/ ├── lib/ ... library-debug-tests.apk ├── classes.dex ├── assets/ ├── res/ ├── lib/ ... © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 25
26. Ingredient 3 Test your end product © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 26
27. BuildConfig.java public final class BuildConfig { public static final boolean DEBUG = Boolean.parseBoolean("false"); public static final String BUILD_TYPE = "release"; // Your custom build config fields. public static final boolean IS_DEMO = false; public static final boolean IS_RELEASE_BUILD = true; ... } © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 27
28. app/build.gradle android { buildType.release { minifiedEnabled true ... } } © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 28
29. Ingredient 4 Dogfood your library © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 29
30. Dogfooding is short for “Eating your own dog food” which represents the practice of using your own products. © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 30
31. How to dogfood • Write apps using your library • Serve your apps to end users • Tests your app in different scenarios © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 31
32. Summary • Create apps • Split off your tests • Test your end product • Eat all your dog food! © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 32
33. So long, and thanks for all the dog food! © David Schreiber-Ranner 2016 / @flashmasterdash / pspdfkit.com 33