To ∞ (~65K) and beyond!

To ∞ (~65K) and beyond!

It's 2016 even for us Android developers. And among many of the problems that we carry over from the past years, the loathed, dreadful, why-in-the-name-of-Andy-Rubin limit of 65536 methods is the most notable of them.

This talk focuses on analyzing this "condition" from a pragmatic and down-to-earth perspective for developers. You will get to understand the problem and its possible solutions, each one of them presented with pros and cons. Bonus round: useful tips and some sarcasm.

E7ffd068afa95697e06f519f3ebd0240?s=128

Sebastiano Gottardo

April 08, 2016
Tweet

Transcript

  1. 1.

    To ∞ (~65K) and beyond! Sebastiano Gottardo, Android Engineer @

    Musixmatch +SebastianoGottardo - @rotxed 1
  2. 2.

    Overview 2 • Context • 65K methods limit • Possible

    approaches • MultiDex • Secondary DEX • Shrinking (and reasoning) • Bottom line
  3. 4.

    Context • OSS, community, libraries, frameworks
 The Android open-source community

    is enormous, and brings to the table thousands of libraries and frameworks. • ✓ Faster development, contributions and improvements
 These tools allows you to build better apps quicker. In addition, contributions by other users lead to improvements. • ✗ Using too many can lead to heavy APK, long loading times, bugs 
 As we will learn, there are tradeoffs. 4
  4. 5.
  5. 6.

    Dalvik Executable (DEX) • Container of the compiled source code


    .java files are compiled into .class files, then packed into a .dex file 6
  6. 7.

    Where is the limit • Not runtime, not DEX, but

    Dalvik’s instruction set!
 7 https://goo.gl/ajlxwX
  7. 8.

    Crossing the limit • Unable to build an APK •

    Panic • Console output 8 Older build system Newer build system
  8. 10.

    Context • No(t likely)
 Lose retro-compatibility! • ART (Android Runtime)


    The new runtime used starting with Lollipop provides a way around. • What to do then?
 There are different solutions/approaches. 10
  9. 11.

    Approaches • MultiDex
 Google’s own solution. • Secondary DEX
 Get

    your hands dirty. • Shrinking / Reasoning
 ProGuard is your friend, and so is your brain. 11
  10. 13.

    MultiDex • Google’s way around this limitation
 aka what’s in

    the docs • Still not the “recommended” approach
 More on that later on. • Different working mechanisms depending on platform
 Dalvik ≠ ART. 13
  11. 15.

    MultiDex with ART • App’s source code is compiled and

    packed into multiple DEX files
 Ultimately packed in the APK • At install time, the OS compiles the code of the DEX files into a single OAT file (ELF)
 AOT compilation instead of JIT compilation 15
  12. 16.

    MultiDex with Dalvik • App’s source code is compiled and

    packed into a single DEX file
 Ultimately packed in the APK • Dalvik is unaware of DEX files other than the main one
 classes.dex • Multiple DEX files must be loaded at runtime
 Can be troublesome (a.k.a. sucks) 16
  13. 17.

    Setup 1/3 1. Gradle-wise setup 17 android {
 compileSdkVersion 23


    buildToolsVersion "23.0.2"
 
 defaultConfig {
 ...
 minSdkVersion 15
 targetSdkVersion 23
 ...
 
 // Enabling multidex support.
 multiDexEnabled true
 }
 ...
 }
  14. 18.

    Setup 2/3 2. Include the MultiDex support library 18 


    dependencies {
 compile 'com.android.support:multidex:1.0.0'
 }
  15. 19.

    Setup 3/3 3. Enable MultiDex in your code 19 <manifest

    xmlns:android="http://schemas.android.com/apk/res/android"
 package="com.example.android.multidex.myapplication">
 <application
 ...
 android:name="android.support.multidex.MultiDexApplication">
 ...
 </application>
 </manifest> a public class YourAwesomeApplication extends MultiDexApplication { ... } b @Override
 protected void attachBaseContext(Context base) {
 super.attachBaseContext(base);
 MultiDex.install(this);
 } c
  16. 20.

    Advantages • Well integrated with the build chain
 No need

    to pull off complicated tricks. • Seamless support across platform versions
 Being a support library. One code, one love! • Extremely easy to enable
 Literally takes 3 minutes. 20
  17. 21.

    Disadvantages • Limited in choosing what goes where
 Esp. true

    for libraries that sport native components. • MultiDex on Dalvik is unreliable
 For 14 < API < 21 crashes may occur. • Secondary DEX files are loaded at runtime
 Startup time is increased! (unless you defer installation) • Build times increase
 Variants may help (dev — min 21, speeds things up) 21
  18. 23.

    Secondary DEX • Historically, the first workaround
 A blog post

    by Fred Chung, a Googler at the time (2007) • Relied on customizing the build chain
 Some Ant magic • Used an interface-based approach
 23
  19. 24.

    How does it work • Put one or more libraries

    into a separate DEX file
 Limited to .jar files, .aar not supported • At runtime, load the secondary DEX on a custom ClassLoader
 Different from the default ClassLoader! • Invoke the libraries’ methods
 Interface, Reflection 24
  20. 25.

    Quick example w/ Reflection 25 twitter = new TwitterFactory().getInstance();
 twitter.setOAuthConsumer(TWITTER_APP_KEY,

    TWITTER_APP_SECRET);
 twitter.setOAuthAccessToken(new AccessToken(token, tokenSecret));
  21. 26.

    Quick example w/ Reflection 26 ...
 twitter = twitterManager.initializeTwitter(TWITTER_APP_KEY, TWITTER_APP_SECRET,

    token, tokenSecret); ... public Object initializeTwitter(String TWITTER_APP_KEY, String TWITTER_APP_SECRET, String token, String tokenSecret) {
 
 try {
 Object twitterFactoryInstance = getClassCached(twitterFactoryFQN).newInstance();
 
 Method getInstanceMethod = getMethodCached(twitterFactoryFQN, "getInstance", null);
 Object twitterInstance = getInstanceMethod.invoke(twitterFactoryInstance, null);
 
 Method setOAuthConsumerMethod = getMethodCached(twitterFQN, "setOAuthConsumer", String.class, String.class);
 setOAuthConsumerMethod.invoke(twitterInstance, TWITTER_APP_KEY, TWITTER_APP_SECRET);
 
 Class atClass = getClassCached(accessTokenFQN);
 Object atInstance = atClass.getDeclaredConstructor(new Class[] {String.class, String.class}).newInstance(token, tokenSecret);
 
 Method setOAuthAccessTokenMethod = getMethodCached(twitterFQN, "setOAuthAccessToken", atClass);
 setOAuthAccessTokenMethod.invoke(twitterInstance, atInstance);
 
 return twitterInstance;
 
 } catch (Exception e) {
 e.printStackTrace();
 }
 
 return null;
 }
  22. 27.

    Advantages • Choose exactly what goes where
 Take that, MultiDex


    • Choose exactly when to load secondary DEX
 So not to impact cold start times 27
  23. 28.

    Disadvantages • Quite hard to setup
 Copy DEX to the

    appropriate dir, create ClassLoader instance • Quite hard to work with
 Esp. if you rely on Reflection • Versioning!
 How to handle updates? 28
  24. 29.

    Improvements • Same approach as Google’s MultiDex
 Patches the Class

    Loader so to look for other DEX files • Specify the dependencies as ‘provided’
 So to be resolved at compile/build time, stripped in the final APK • Best of two worlds (kind of)
 Removes the need for ifaces, Reflection, still a bit cumbersome 29
  25. 31.

    — Google MultiDex online documentation Before configuring your app to

    enable use of 65K or more method references, you should take steps to reduce the total number of references called by your app code, including methods defined by your app code or included libraries. 31
  26. 32.

    ProGuard in pills • Shrink, optimize, obfuscate • Works both

    for your code and for third-party libraries • Has an impact on APK size, methods number, performance and security 32
  27. 33.

    ProGuard ProTips • Get familiar with writing appropriate rules
 The

    stricter, the better 33 -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;} • What’s wrong with the following keep rule? -keep class com.google.** { *; } -keepnames class com.google.** { *; } -keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; } • Adopts an exclusion-based approach
  28. 34.

    ProGuard ProTips 34 -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;} -keep class com.google.**

    { *; } -keepnames class com.google.** { *; } -keepclassmembernames class * { @com.google.android.gms.common.annotation.KeepName *; } • Adopts an exclusion-based approach • Get familiar with writing appropriate rules
 The stricter, the better • What’s wrong with the following keep rule?
  29. 35.

    ProGuard: default rules 35 android { ... buildTypes { release

    { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' } } } • ProGuard already includes common rules for common cases
 ~/sdk/tools/proguard/lib/proguard-android.txt -keepclasseswithmembernames class * { native <methods>; }
  30. 36.

    DexGuard: beefier, for beefier companies 36 • ProGuard’s bigger (and

    pricier) brother • Advanced features: res/strings/.so encryption, stronger obfuscation,
 tamper detection, its own multidex! and more • Pluggable encryption!
 I mean, seriously, that exclamation mark has a reason • Default rules are much more extensive
  31. 37.

    ProGuard and third-party libs 37 • Many provide the ProGuard

    statement (a.k.a RTFM)
 e.g., Square, Facebook, … • Pay attention to ‘keep-all’ rules!
 especially for libraries with many methods
  32. 39.

    … wait, how do I count libraries methods? 39 …

    how do I count methods at all?!
  33. 42.

    inloop’s “APK method count” • Simple website with drag-and-drop interface

    • Drop the APK and see the (locally) computed count
 No APKs uploaded :+1: • http://inloop.github.io/apk-method-count/
 42 ✓ Easy, immediate, private
 
 ✗ Build every time
  34. 43.

    ClassyShark • Useful Java tool to inspect APKs
 Written by

    Boris Farber (Googler) • Inspect number of classes, members, methods, dependencies, …
 Acts as a simple decompiler, too
 
 43
  35. 46.

    ClassyShark • Useful Java tool to inspect APKs
 Written by

    Boris Farber (Googler) • Inspect number of classes, members, methods, dependencies, …
 Acts as a simple decompiler, too
 
 46 ✓ Easy, plenty of info, multi-purpose
 
 ✗ Build every time, more oriented to agnostic debugging
  36. 47.

    dexcount-gradle-plugin • Gradle plugin, developed by KeepSafe • See the

    current methods count every time you build
 Just open the Gradle console • Outputs per-package information in a ${variant}.txt file
 
 47
  37. 50.

    dexcount-gradle-plugin • Gradle plugin, developed by KeepSafe • See the

    current methods count every time you build
 Just open the Gradle console • Outputs per-package information in a ${variant}.txt file
 
 50 ✓ Keep the count controlled at a glance
 
 ✗ Cumbersome to see single deps count
  38. 51.

    MethodsCount • Developed by Sebastiano Gottardo, Nicola Miotto, Dario Marcato


    • Allows to see methods and dependencies for libraries beforehand
 • Compare different libraries in terms of “lightweight-ness” 51
  39. 52.

    MethodsCount 52 • More than 15000 libraries and dependencies
 What’s

    missing is calculated upon the first request • For each entry 1. dex/jar size 2. direct count + dependencies count 3. multiple versions (monitor changes over time)

  40. 56.

    MethodsCount - AS plugin 56 • Parses `build.gradle` and directly

    shows the count • Convenient when you drop a dependency right away
  41. 57.

    MethodsCount - AS plugin 57 • Parses `build.gradle` and directly

    shows the count • Convenient when you drop a dependency right away
  42. 58.

    MethodsCount 58 ✓ See the count before having to hit

    the limit
 
 ✓ Compare different libraries
 Worth having 4000 methods instead of 900?
 
 ✓ Analyze dependencies
 No, Guava is not a good excuse to have 15000+ methods
 
 ✓ Integrated in AS, thanks to the plugin
 ✗ The count is before ProGuard → Upper bound!
  43. 60.

    Bottom line • The limit is here, at least for

    some time
 Learn to deal with it • The recommended way is “Do NOT cross the limit!”
 And use ProGuard. Your nerves will thank you later! • If you do need to cross, there are solutions (with drawbacks)
 MultiDex, secondary DEX 60
  44. 61.

    Routine 1. PM shows interest in a new library
 :eyes_roll:

    2. With dex-gradle-plugin, check the current count 3. Check MethodsCount to see how many methods, size and deps 4. Evaluate whether fits a secondary DEX 5. Consider alternatives (again, MethodsCount) 61