To ∞ (~65K) and beyond!

It's 2016 even for us Android developers. And among many of the problems that we carry over from the past years, the loathed, dreadful, why-in-the-name-of-Andy-Rubin limit of 65536 methods is the most notable of them.

This talk focuses on analyzing this "condition" from a pragmatic and down-to-earth perspective for developers. You will get to understand the problem and its possible solutions, each one of them presented with pros and cons. Bonus round: useful tips and some sarcasm.

Transcript

1. To ∞ (~65K) and beyond!
   Sebastiano Gottardo, Android Engineer @ Musixmatch
   +SebastianoGottardo - @rotxed
   1
   

   View Slide

2. Overview
   2
   • Context
   • 65K methods limit
   • Possible approaches
   • MultiDex
   • Secondary DEX
   • Shrinking (and reasoning)
   • Bottom line
   

   View Slide

3. Context
   3
   

   View Slide

4. Context
   • OSS, community, libraries, frameworks

   The Android open-source community is enormous, and brings to the
   table thousands of libraries and frameworks.
   • ✓ Faster development, contributions and improvements

   These tools allows you to build better apps quicker. In addition,
   contributions by other users lead to improvements.
   • ✗ Using too many can lead to heavy APK, long loading times, bugs 

   As we will learn, there are tradeoffs.
   4
   

   View Slide

5. View Slide

6. Dalvik Executable (DEX)
   • Container of the compiled source code

   .java files are compiled into .class files, then packed into a .dex file
   6
   

   View Slide

7. Where is the limit
   • Not runtime, not DEX, but Dalvik’s instruction set!

   7
   https://goo.gl/ajlxwX
   

   View Slide

8. Crossing the limit
   • Unable to build an APK
   • Panic
   • Console output
   8
   Older build system Newer build system
   

   View Slide

9. Is this going to change?
   9
   

   View Slide

10. Context
    • No(t likely)

    Lose retro-compatibility!
    • ART (Android Runtime)

    The new runtime used starting with Lollipop provides a way around.
    • What to do then?

    There are different solutions/approaches.
    10
    

    View Slide

11. Approaches
    • MultiDex

    Google’s own solution.
    • Secondary DEX

    Get your hands dirty.
    • Shrinking / Reasoning

    ProGuard is your friend, and so is your brain.
    11
    

    View Slide

12. MultiDex
    12
    

    View Slide

13. MultiDex
    • Google’s way around this limitation

    aka what’s in the docs
    • Still not the “recommended” approach

    More on that later on.
    • Different working mechanisms depending on platform

    Dalvik ≠ ART.
    13
    

    View Slide

14. MultiDex: Dalvik VS ART
    14
    ART
    Dalvik
    

    View Slide

15. MultiDex with ART
    • App’s source code is compiled and packed into multiple DEX files

    Ultimately packed in the APK
    • At install time, the OS compiles the code of the DEX files into a single
    OAT file (ELF)

    AOT compilation instead of JIT compilation
    15
    

    View Slide

16. MultiDex with Dalvik
    • App’s source code is compiled and packed into a single DEX file

    Ultimately packed in the APK
    • Dalvik is unaware of DEX files other than the main one

    classes.dex
    • Multiple DEX files must be loaded at runtime

    Can be troublesome (a.k.a. sucks)
    16
    

    View Slide

17. Setup 1/3
    1. Gradle-wise setup
    17
    android {

    compileSdkVersion 23

    buildToolsVersion "23.0.2"

    

    defaultConfig {

    ...

    minSdkVersion 15

    targetSdkVersion 23

    ...

    

    // Enabling multidex support.

    multiDexEnabled true

    }

    ...

    }
    

    View Slide

18. Setup 2/3
    2. Include the MultiDex support library
    18
    

    dependencies {

    compile 'com.android.support:multidex:1.0.0'

    }
    

    View Slide

19. Setup 3/3
    3. Enable MultiDex in your code
    19
    package="com.example.android.multidex.myapplication">

    ...

    android:name="android.support.multidex.MultiDexApplication">

    ...

    

    
    a
    public class YourAwesomeApplication extends MultiDexApplication {
    ...
    }
    b
    @Override

    protected void attachBaseContext(Context base) {

    super.attachBaseContext(base);

    MultiDex.install(this);

    }
    c
    

    View Slide

20. Advantages
    • Well integrated with the build chain

    No need to pull off complicated tricks.
    • Seamless support across platform versions

    Being a support library. One code, one love!
    • Extremely easy to enable

    Literally takes 3 minutes.
    20
    

    View Slide

21. Disadvantages
    • Limited in choosing what goes where

    Esp. true for libraries that sport native components.
    • MultiDex on Dalvik is unreliable

    For 14 < API < 21 crashes may occur.
    • Secondary DEX files are loaded at runtime

    Startup time is increased! (unless you defer installation)
    • Build times increase

    Variants may help (dev — min 21, speeds things up)
    21
    

    View Slide

22. Secondary DEX
    22
    

    View Slide

23. Secondary DEX
    • Historically, the first workaround

    A blog post by Fred Chung, a Googler at the time (2007)
    • Relied on customizing the build chain

    Some Ant magic
    • Used an interface-based approach

    23
    

    View Slide

24. How does it work
    • Put one or more libraries into a separate DEX file

    Limited to .jar files, .aar not supported
    • At runtime, load the secondary DEX on a custom ClassLoader

    Different from the default ClassLoader!
    • Invoke the libraries’ methods

    Interface, Reflection
    24
    

    View Slide

25. Quick example w/ Reflection
    25
    twitter = new TwitterFactory().getInstance();

    twitter.setOAuthConsumer(TWITTER_APP_KEY, TWITTER_APP_SECRET);

    twitter.setOAuthAccessToken(new AccessToken(token, tokenSecret));
    

    View Slide

26. Quick example w/ Reflection
    26
    ...

    twitter = twitterManager.initializeTwitter(TWITTER_APP_KEY, TWITTER_APP_SECRET, token, tokenSecret);
    ...
    public Object initializeTwitter(String TWITTER_APP_KEY, String TWITTER_APP_SECRET, String token, String tokenSecret) {

    

    try {

    Object twitterFactoryInstance = getClassCached(twitterFactoryFQN).newInstance();

    

    Method getInstanceMethod = getMethodCached(twitterFactoryFQN, "getInstance", null);

    Object twitterInstance = getInstanceMethod.invoke(twitterFactoryInstance, null);

    

    Method setOAuthConsumerMethod = getMethodCached(twitterFQN, "setOAuthConsumer", String.class, String.class);

    setOAuthConsumerMethod.invoke(twitterInstance, TWITTER_APP_KEY, TWITTER_APP_SECRET);

    

    Class atClass = getClassCached(accessTokenFQN);

    Object atInstance = atClass.getDeclaredConstructor(new Class[] {String.class, String.class}).newInstance(token, tokenSecret);

    

    Method setOAuthAccessTokenMethod = getMethodCached(twitterFQN, "setOAuthAccessToken", atClass);

    setOAuthAccessTokenMethod.invoke(twitterInstance, atInstance);

    

    return twitterInstance;

    

    } catch (Exception e) {

    e.printStackTrace();

    }

    

    return null;

    }
    

    View Slide

27. Advantages
    • Choose exactly what goes where

    Take that, MultiDex

    • Choose exactly when to load secondary DEX

    So not to impact cold start times
    27
    

    View Slide

28. Disadvantages
    • Quite hard to setup

    Copy DEX to the appropriate dir, create ClassLoader instance
    • Quite hard to work with

    Esp. if you rely on Reflection
    • Versioning!

    How to handle updates?
    28
    

    View Slide

29. Improvements
    • Same approach as Google’s MultiDex

    Patches the Class Loader so to look for other DEX files
    • Specify the dependencies as ‘provided’

    So to be resolved at compile/build time, stripped in the final APK
    • Best of two worlds (kind of)

    Removes the need for ifaces, Reflection, still a bit cumbersome
    29
    

    View Slide

30. Shrinking / Reasoning
    30
    

    View Slide

31. — Google MultiDex online documentation
    Before configuring your app to enable use of 65K or
    more method references, you should take steps to
    reduce the total number of references called by your
    app code, including methods defined by your app code
    or included libraries.
    31
    

    View Slide

32. ProGuard in pills
    • Shrink, optimize, obfuscate
    • Works both for your code and for third-party libraries
    • Has an impact on APK size, methods number, performance and
    security
    32
    

    View Slide

33. ProGuard ProTips
    • Get familiar with writing appropriate rules

    The stricter, the better
    33
    -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}
    • What’s wrong with the following keep rule?
    -keep class com.google.** { *; }
    -keepnames class com.google.** { *; }
    -keepclassmembernames class * {
    @com.google.android.gms.common.annotation.KeepName *;
    }
    • Adopts an exclusion-based approach
    

    View Slide

34. ProGuard ProTips
    34
    -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}
    -keep class com.google.** { *; }
    -keepnames class com.google.** { *; }
    -keepclassmembernames class * {
    @com.google.android.gms.common.annotation.KeepName *;
    }
    • Adopts an exclusion-based approach
    • Get familiar with writing appropriate rules

    The stricter, the better
    • What’s wrong with the following keep rule?
    

    View Slide

35. ProGuard: default rules
    35
    android {
    ...
    buildTypes {
    release {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'),
    'proguard-rules.pro'
    }
    }
    }
    • ProGuard already includes common rules for common cases

    ~/sdk/tools/proguard/lib/proguard-android.txt
    -keepclasseswithmembernames class * {
    native ;
    }
    

    View Slide

36. DexGuard: beefier, for beefier companies
    36
    • ProGuard’s bigger (and pricier) brother
    • Advanced features: res/strings/.so encryption, stronger obfuscation,

    tamper detection, its own multidex! and more
    • Pluggable encryption!

    I mean, seriously, that exclamation mark has a reason
    • Default rules are much more extensive
    

    View Slide

37. ProGuard and third-party libs
    37
    • Many provide the ProGuard statement (a.k.a RTFM)

    e.g., Square, Facebook, …
    • Pay attention to ‘keep-all’ rules!

    especially for libraries with many methods
    

    View Slide

38. Bad example
    38
    • Stetho (Facebook)
    # Keep Calm and Don’t Warn
    

    View Slide

39. … wait, how do I count libraries
    methods?
    39
    … how do I count methods at all?!
    

    View Slide

40. inloop’s “APK method count”
    40
    

    View Slide

41. inloop’s “APK method count”
    41
    

    View Slide

42. inloop’s “APK method count”
    • Simple website with drag-and-drop interface
    • Drop the APK and see the (locally) computed count

    No APKs uploaded :+1:
    • http://inloop.github.io/apk-method-count/

    42
    ✓ Easy, immediate, private

    

    ✗ Build every time
    

    View Slide

43. ClassyShark
    • Useful Java tool to inspect APKs

    Written by Boris Farber (Googler)
    • Inspect number of classes, members, methods, dependencies, …

    Acts as a simple decompiler, too

    

    43
    

    View Slide

44. ClassyShark
    44
    

    View Slide

45. ClassyShark
    45
    

    View Slide

46. ClassyShark
    • Useful Java tool to inspect APKs

    Written by Boris Farber (Googler)
    • Inspect number of classes, members, methods, dependencies, …

    Acts as a simple decompiler, too

    

    46
    ✓ Easy, plenty of info, multi-purpose

    

    ✗ Build every time, more oriented to agnostic debugging
    

    View Slide

47. dexcount-gradle-plugin
    • Gradle plugin, developed by KeepSafe
    • See the current methods count every time you build

    Just open the Gradle console
    • Outputs per-package information in a ${variant}.txt file

    

    47
    

    View Slide

48. dexcount-gradle-plugin
    48
    

    View Slide

49. dexcount-gradle-plugin
    49
    

    View Slide

50. dexcount-gradle-plugin
    • Gradle plugin, developed by KeepSafe
    • See the current methods count every time you build

    Just open the Gradle console
    • Outputs per-package information in a ${variant}.txt file

    

    50
    ✓ Keep the count controlled at a glance

    

    ✗ Cumbersome to see single deps count
    

    View Slide

51. MethodsCount
    • Developed by Sebastiano Gottardo, Nicola Miotto, Dario Marcato

    • Allows to see methods and dependencies for libraries beforehand

    • Compare different libraries in terms of “lightweight-ness”
    51
    

    View Slide

52. MethodsCount
    52
    • More than 15000 libraries and dependencies

    What’s missing is calculated upon the first request
    • For each entry
    1. dex/jar size
    2. direct count + dependencies count
    3. multiple versions (monitor changes over time)

    

    View Slide

53. MethodsCount
    53
    

    View Slide

54. MethodsCount
    54
    

    View Slide

55. MethodsCount
    55
    

    View Slide

56. MethodsCount - AS plugin
    56
    • Parses `build.gradle` and directly shows the count
    • Convenient when you drop a dependency right away
    

    View Slide

57. MethodsCount - AS plugin
    57
    • Parses `build.gradle` and directly shows the count
    • Convenient when you drop a dependency right away
    

    View Slide

58. MethodsCount
    58
    ✓ See the count before having to hit the limit

    

    ✓ Compare different libraries

    Worth having 4000 methods instead of 900?

    

    ✓ Analyze dependencies

    No, Guava is not a good excuse to have 15000+ methods

    

    ✓ Integrated in AS, thanks to the plugin

    ✗ The count is before ProGuard → Upper bound!
    

    View Slide

59. Bottom line
    59
    

    View Slide

60. Bottom line
    • The limit is here, at least for some time

    Learn to deal with it
    • The recommended way is “Do NOT cross the limit!”

    And use ProGuard. Your nerves will thank you later!
    • If you do need to cross, there are solutions (with drawbacks)

    MultiDex, secondary DEX
    60
    

    View Slide

61. Routine
    1. PM shows interest in a new library

    :eyes_roll:
    2. With dex-gradle-plugin, check the current count
    3. Check MethodsCount to see how many methods, size and deps
    4. Evaluate whether fits a secondary DEX
    5. Consider alternatives (again, MethodsCount)
    61
    

    View Slide

62. Useful resources
    • Timothy Mellor: https://github.com/tmelz/multidex_notes
    62
    @rotxed
    +SebastianoGottardo
    Follow me
    

    View Slide