To ∞ (~65K) and beyond!

It's 2016 even for us Android developers. And among many of the problems that we carry over from the past years, the loathed, dreadful, why-in-the-name-of-Andy-Rubin limit of 65536 methods is the most notable of them.

This talk focuses on analyzing this "condition" from a pragmatic and down-to-earth perspective for developers. You will get to understand the problem and its possible solutions, each one of them presented with pros and cons. Bonus round: useful tips and some sarcasm.

Transcript

1. To ∞ (~65K) and beyond! Sebastiano Gottardo, Android Engineer @ Musixmatch +SebastianoGottardo - @rotxed 1
2. Overview 2 • Context • 65K methods limit • Possible approaches • MultiDex • Secondary DEX • Shrinking (and reasoning) • Bottom line
3. Context 3
4. Context • OSS, community, libraries, frameworks
 The Android open-source community is enormous, and brings to the table thousands of libraries and frameworks. • ✓ Faster development, contributions and improvements
 These tools allows you to build better apps quicker. In addition, contributions by other users lead to improvements. • ✗ Using too many can lead to heavy APK, long loading times, bugs 
 As we will learn, there are tradeoffs. 4
5. 5 65536
6. Dalvik Executable (DEX) • Container of the compiled source code
 .java files are compiled into .class files, then packed into a .dex file 6
7. Where is the limit • Not runtime, not DEX, but Dalvik’s instruction set!
 7 https://goo.gl/ajlxwX
8. Crossing the limit • Unable to build an APK • Panic • Console output 8 Older build system Newer build system
9. Is this going to change? 9
10. Context • No(t likely)
 Lose retro-compatibility! • ART (Android Runtime)
 The new runtime used starting with Lollipop provides a way around. • What to do then?
 There are different solutions/approaches. 10
11. Approaches • MultiDex
 Google’s own solution. • Secondary DEX
 Get your hands dirty. • Shrinking / Reasoning
 ProGuard is your friend, and so is your brain. 11
12. MultiDex 12
13. MultiDex • Google’s way around this limitation
 aka what’s in the docs • Still not the “recommended” approach
 More on that later on. • Different working mechanisms depending on platform
 Dalvik ≠ ART. 13
14. MultiDex: Dalvik VS ART 14 ART Dalvik
15. MultiDex with ART • App’s source code is compiled and packed into multiple DEX files
 Ultimately packed in the APK • At install time, the OS compiles the code of the DEX files into a single OAT file (ELF)
 AOT compilation instead of JIT compilation 15
16. MultiDex with Dalvik • App’s source code is compiled and packed into a single DEX file
 Ultimately packed in the APK • Dalvik is unaware of DEX files other than the main one
 classes.dex • Multiple DEX files must be loaded at runtime
 Can be troublesome (a.k.a. sucks) 16
17. Setup 1/3 1. Gradle-wise setup 17 android {
 compileSdkVersion 23
 buildToolsVersion "23.0.2"
 
 defaultConfig {
 ...
 minSdkVersion 15
 targetSdkVersion 23
 ...
 
 // Enabling multidex support.
 multiDexEnabled true
 }
 ...
 }
18. Setup 2/3 2. Include the MultiDex support library 18 
 dependencies {
 compile 'com.android.support:multidex:1.0.0'
 }
19. Setup 3/3 3. Enable MultiDex in your code 19 <manifest xmlns:android="http://schemas.android.com/apk/res/android"
 package="com.example.android.multidex.myapplication">
 <application
 ...
 android:name="android.support.multidex.MultiDexApplication">
 ...
 </application>
 </manifest> a public class YourAwesomeApplication extends MultiDexApplication { ... } b @Override
 protected void attachBaseContext(Context base) {
 super.attachBaseContext(base);
 MultiDex.install(this);
 } c
20. Advantages • Well integrated with the build chain
 No need to pull off complicated tricks. • Seamless support across platform versions
 Being a support library. One code, one love! • Extremely easy to enable
 Literally takes 3 minutes. 20
21. Disadvantages • Limited in choosing what goes where
 Esp. true for libraries that sport native components. • MultiDex on Dalvik is unreliable
 For 14 < API < 21 crashes may occur. • Secondary DEX files are loaded at runtime
 Startup time is increased! (unless you defer installation) • Build times increase
 Variants may help (dev — min 21, speeds things up) 21
22. Secondary DEX 22
23. Secondary DEX • Historically, the first workaround
 A blog post by Fred Chung, a Googler at the time (2007) • Relied on customizing the build chain
 Some Ant magic • Used an interface-based approach
 23
24. How does it work • Put one or more libraries into a separate DEX file
 Limited to .jar files, .aar not supported • At runtime, load the secondary DEX on a custom ClassLoader
 Different from the default ClassLoader! • Invoke the libraries’ methods
 Interface, Reflection 24
25. Quick example w/ Reflection 25 twitter = new TwitterFactory().getInstance();
 twitter.setOAuthConsumer(TWITTER_APP_KEY, TWITTER_APP_SECRET);
 twitter.setOAuthAccessToken(new AccessToken(token, tokenSecret));
26. Quick example w/ Reflection 26 ...
 twitter = twitterManager.initializeTwitter(TWITTER_APP_KEY, TWITTER_APP_SECRET, token, tokenSecret); ... public Object initializeTwitter(String TWITTER_APP_KEY, String TWITTER_APP_SECRET, String token, String tokenSecret) {
 
 try {
 Object twitterFactoryInstance = getClassCached(twitterFactoryFQN).newInstance();
 
 Method getInstanceMethod = getMethodCached(twitterFactoryFQN, "getInstance", null);
 Object twitterInstance = getInstanceMethod.invoke(twitterFactoryInstance, null);
 
 Method setOAuthConsumerMethod = getMethodCached(twitterFQN, "setOAuthConsumer", String.class, String.class);
 setOAuthConsumerMethod.invoke(twitterInstance, TWITTER_APP_KEY, TWITTER_APP_SECRET);
 
 Class atClass = getClassCached(accessTokenFQN);
 Object atInstance = atClass.getDeclaredConstructor(new Class[] {String.class, String.class}).newInstance(token, tokenSecret);
 
 Method setOAuthAccessTokenMethod = getMethodCached(twitterFQN, "setOAuthAccessToken", atClass);
 setOAuthAccessTokenMethod.invoke(twitterInstance, atInstance);
 
 return twitterInstance;
 
 } catch (Exception e) {
 e.printStackTrace();
 }
 
 return null;
 }
27. Advantages • Choose exactly what goes where
 Take that, MultiDex
 • Choose exactly when to load secondary DEX
 So not to impact cold start times 27
28. Disadvantages • Quite hard to setup
 Copy DEX to the appropriate dir, create ClassLoader instance • Quite hard to work with
 Esp. if you rely on Reflection • Versioning!
 How to handle updates? 28
29. Improvements • Same approach as Google’s MultiDex
 Patches the Class Loader so to look for other DEX files • Specify the dependencies as ‘provided’
 So to be resolved at compile/build time, stripped in the final APK • Best of two worlds (kind of)
 Removes the need for ifaces, Reflection, still a bit cumbersome 29
30. Shrinking / Reasoning 30
31. — Google MultiDex online documentation Before configuring your app to enable use of 65K or more method references, you should take steps to reduce the total number of references called by your app code, including methods defined by your app code or included libraries. 31
32. ProGuard in pills • Shrink, optimize, obfuscate • Works both for your code and for third-party libraries • Has an impact on APK size, methods number, performance and security 32
33. ProGuard ProTips • Get familiar with writing appropriate rules
 The stricter, the better 33 -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;} • What’s wrong with the following keep rule? -keep class com.google.** { *; } -keepnames class com.google.** { *; } -keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; } • Adopts an exclusion-based approach
34. ProGuard ProTips 34 -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;} -keep class com.google.** { *; } -keepnames class com.google.** { *; } -keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; } • Adopts an exclusion-based approach • Get familiar with writing appropriate rules
 The stricter, the better • What’s wrong with the following keep rule?
35. ProGuard: default rules 35 android { ... buildTypes { release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' } } } • ProGuard already includes common rules for common cases
 ~/sdk/tools/proguard/lib/proguard-android.txt -keepclasseswithmembernames class * { native <methods>; }
36. DexGuard: beefier, for beefier companies 36 • ProGuard’s bigger (and pricier) brother • Advanced features: res/strings/.so encryption, stronger obfuscation,
 tamper detection, its own multidex! and more • Pluggable encryption!
 I mean, seriously, that exclamation mark has a reason • Default rules are much more extensive
37. ProGuard and third-party libs 37 • Many provide the ProGuard statement (a.k.a RTFM)
 e.g., Square, Facebook, … • Pay attention to ‘keep-all’ rules!
 especially for libraries with many methods
38. Bad example 38 • Stetho (Facebook) # Keep Calm and Don’t Warn
39. … wait, how do I count libraries methods? 39 … how do I count methods at all?!
40. inloop’s “APK method count” 40
41. inloop’s “APK method count” 41
42. inloop’s “APK method count” • Simple website with drag-and-drop interface • Drop the APK and see the (locally) computed count
 No APKs uploaded :+1: • http://inloop.github.io/apk-method-count/
 42 ✓ Easy, immediate, private
 
 ✗ Build every time
43. ClassyShark • Useful Java tool to inspect APKs
 Written by Boris Farber (Googler) • Inspect number of classes, members, methods, dependencies, …
 Acts as a simple decompiler, too
 
 43
44. ClassyShark 44
45. ClassyShark 45
46. ClassyShark • Useful Java tool to inspect APKs
 Written by Boris Farber (Googler) • Inspect number of classes, members, methods, dependencies, …
 Acts as a simple decompiler, too
 
 46 ✓ Easy, plenty of info, multi-purpose
 
 ✗ Build every time, more oriented to agnostic debugging
47. dexcount-gradle-plugin • Gradle plugin, developed by KeepSafe • See the current methods count every time you build
 Just open the Gradle console • Outputs per-package information in a ${variant}.txt file
 
 47
48. dexcount-gradle-plugin 48
49. dexcount-gradle-plugin 49
50. dexcount-gradle-plugin • Gradle plugin, developed by KeepSafe • See the current methods count every time you build
 Just open the Gradle console • Outputs per-package information in a ${variant}.txt file
 
 50 ✓ Keep the count controlled at a glance
 
 ✗ Cumbersome to see single deps count
51. MethodsCount • Developed by Sebastiano Gottardo, Nicola Miotto, Dario Marcato
 • Allows to see methods and dependencies for libraries beforehand
 • Compare different libraries in terms of “lightweight-ness” 51
52. MethodsCount 52 • More than 15000 libraries and dependencies
 What’s missing is calculated upon the first request • For each entry 1. dex/jar size 2. direct count + dependencies count 3. multiple versions (monitor changes over time)

53. MethodsCount 53
54. MethodsCount 54
55. MethodsCount 55
56. MethodsCount - AS plugin 56 • Parses `build.gradle` and directly shows the count • Convenient when you drop a dependency right away
57. MethodsCount - AS plugin 57 • Parses `build.gradle` and directly shows the count • Convenient when you drop a dependency right away
58. MethodsCount 58 ✓ See the count before having to hit the limit
 
 ✓ Compare different libraries
 Worth having 4000 methods instead of 900?
 
 ✓ Analyze dependencies
 No, Guava is not a good excuse to have 15000+ methods
 
 ✓ Integrated in AS, thanks to the plugin
 ✗ The count is before ProGuard → Upper bound!
59. Bottom line 59
60. Bottom line • The limit is here, at least for some time
 Learn to deal with it • The recommended way is “Do NOT cross the limit!”
 And use ProGuard. Your nerves will thank you later! • If you do need to cross, there are solutions (with drawbacks)
 MultiDex, secondary DEX 60
61. Routine 1. PM shows interest in a new library
 :eyes_roll: 2. With dex-gradle-plugin, check the current count 3. Check MethodsCount to see how many methods, size and deps 4. Evaluate whether fits a secondary DEX 5. Consider alternatives (again, MethodsCount) 61
62. Useful resources • Timothy Mellor: https://github.com/tmelz/multidex_notes 62 @rotxed +SebastianoGottardo Follow me