$30 off During Our Annual Pro Sale. View Details »

To ∞ (~65K) and beyond!

To ∞ (~65K) and beyond!

It's 2016 even for us Android developers. And among many of the problems that we carry over from the past years, the loathed, dreadful, why-in-the-name-of-Andy-Rubin limit of 65536 methods is the most notable of them.

This talk focuses on analyzing this "condition" from a pragmatic and down-to-earth perspective for developers. You will get to understand the problem and its possible solutions, each one of them presented with pros and cons. Bonus round: useful tips and some sarcasm.

Sebastiano Gottardo

April 08, 2016
Tweet

More Decks by Sebastiano Gottardo

Other Decks in Programming

Transcript

  1. To ∞ (~65K) and beyond!
    Sebastiano Gottardo, Android Engineer @ Musixmatch
    +SebastianoGottardo - @rotxed
    1

    View Slide

  2. Overview
    2
    • Context
    • 65K methods limit
    • Possible approaches
    • MultiDex
    • Secondary DEX
    • Shrinking (and reasoning)
    • Bottom line

    View Slide

  3. Context
    3

    View Slide

  4. Context
    • OSS, community, libraries, frameworks

    The Android open-source community is enormous, and brings to the
    table thousands of libraries and frameworks.
    • ✓ Faster development, contributions and improvements

    These tools allows you to build better apps quicker. In addition,
    contributions by other users lead to improvements.
    • ✗ Using too many can lead to heavy APK, long loading times, bugs 

    As we will learn, there are tradeoffs.
    4

    View Slide

  5. 5
    65536

    View Slide

  6. Dalvik Executable (DEX)
    • Container of the compiled source code

    .java files are compiled into .class files, then packed into a .dex file
    6

    View Slide

  7. Where is the limit
    • Not runtime, not DEX, but Dalvik’s instruction set!

    7
    https://goo.gl/ajlxwX

    View Slide

  8. Crossing the limit
    • Unable to build an APK
    • Panic
    • Console output
    8
    Older build system Newer build system

    View Slide

  9. Is this going to change?
    9

    View Slide

  10. Context
    • No(t likely)

    Lose retro-compatibility!
    • ART (Android Runtime)

    The new runtime used starting with Lollipop provides a way around.
    • What to do then?

    There are different solutions/approaches.
    10

    View Slide

  11. Approaches
    • MultiDex

    Google’s own solution.
    • Secondary DEX

    Get your hands dirty.
    • Shrinking / Reasoning

    ProGuard is your friend, and so is your brain.
    11

    View Slide

  12. MultiDex
    12

    View Slide

  13. MultiDex
    • Google’s way around this limitation

    aka what’s in the docs
    • Still not the “recommended” approach

    More on that later on.
    • Different working mechanisms depending on platform

    Dalvik ≠ ART.
    13

    View Slide

  14. MultiDex: Dalvik VS ART
    14
    ART
    Dalvik

    View Slide

  15. MultiDex with ART
    • App’s source code is compiled and packed into multiple DEX files

    Ultimately packed in the APK
    • At install time, the OS compiles the code of the DEX files into a single
    OAT file (ELF)

    AOT compilation instead of JIT compilation
    15

    View Slide

  16. MultiDex with Dalvik
    • App’s source code is compiled and packed into a single DEX file

    Ultimately packed in the APK
    • Dalvik is unaware of DEX files other than the main one

    classes.dex
    • Multiple DEX files must be loaded at runtime

    Can be troublesome (a.k.a. sucks)
    16

    View Slide

  17. Setup 1/3
    1. Gradle-wise setup
    17
    android {

    compileSdkVersion 23

    buildToolsVersion "23.0.2"


    defaultConfig {

    ...

    minSdkVersion 15

    targetSdkVersion 23

    ...


    // Enabling multidex support.

    multiDexEnabled true

    }

    ...

    }

    View Slide

  18. Setup 2/3
    2. Include the MultiDex support library
    18

    dependencies {

    compile 'com.android.support:multidex:1.0.0'

    }

    View Slide

  19. Setup 3/3
    3. Enable MultiDex in your code
    19
    package="com.example.android.multidex.myapplication">

    ...

    android:name="android.support.multidex.MultiDexApplication">

    ...



    a
    public class YourAwesomeApplication extends MultiDexApplication {
    ...
    }
    b
    @Override

    protected void attachBaseContext(Context base) {

    super.attachBaseContext(base);

    MultiDex.install(this);

    }
    c

    View Slide

  20. Advantages
    • Well integrated with the build chain

    No need to pull off complicated tricks.
    • Seamless support across platform versions

    Being a support library. One code, one love!
    • Extremely easy to enable

    Literally takes 3 minutes.
    20

    View Slide

  21. Disadvantages
    • Limited in choosing what goes where

    Esp. true for libraries that sport native components.
    • MultiDex on Dalvik is unreliable

    For 14 < API < 21 crashes may occur.
    • Secondary DEX files are loaded at runtime

    Startup time is increased! (unless you defer installation)
    • Build times increase

    Variants may help (dev — min 21, speeds things up)
    21

    View Slide

  22. Secondary DEX
    22

    View Slide

  23. Secondary DEX
    • Historically, the first workaround

    A blog post by Fred Chung, a Googler at the time (2007)
    • Relied on customizing the build chain

    Some Ant magic
    • Used an interface-based approach

    23

    View Slide

  24. How does it work
    • Put one or more libraries into a separate DEX file

    Limited to .jar files, .aar not supported
    • At runtime, load the secondary DEX on a custom ClassLoader

    Different from the default ClassLoader!
    • Invoke the libraries’ methods

    Interface, Reflection
    24

    View Slide

  25. Quick example w/ Reflection
    25
    twitter = new TwitterFactory().getInstance();

    twitter.setOAuthConsumer(TWITTER_APP_KEY, TWITTER_APP_SECRET);

    twitter.setOAuthAccessToken(new AccessToken(token, tokenSecret));

    View Slide

  26. Quick example w/ Reflection
    26
    ...

    twitter = twitterManager.initializeTwitter(TWITTER_APP_KEY, TWITTER_APP_SECRET, token, tokenSecret);
    ...
    public Object initializeTwitter(String TWITTER_APP_KEY, String TWITTER_APP_SECRET, String token, String tokenSecret) {


    try {

    Object twitterFactoryInstance = getClassCached(twitterFactoryFQN).newInstance();


    Method getInstanceMethod = getMethodCached(twitterFactoryFQN, "getInstance", null);

    Object twitterInstance = getInstanceMethod.invoke(twitterFactoryInstance, null);


    Method setOAuthConsumerMethod = getMethodCached(twitterFQN, "setOAuthConsumer", String.class, String.class);

    setOAuthConsumerMethod.invoke(twitterInstance, TWITTER_APP_KEY, TWITTER_APP_SECRET);


    Class atClass = getClassCached(accessTokenFQN);

    Object atInstance = atClass.getDeclaredConstructor(new Class[] {String.class, String.class}).newInstance(token, tokenSecret);


    Method setOAuthAccessTokenMethod = getMethodCached(twitterFQN, "setOAuthAccessToken", atClass);

    setOAuthAccessTokenMethod.invoke(twitterInstance, atInstance);


    return twitterInstance;


    } catch (Exception e) {

    e.printStackTrace();

    }


    return null;

    }

    View Slide

  27. Advantages
    • Choose exactly what goes where

    Take that, MultiDex

    • Choose exactly when to load secondary DEX

    So not to impact cold start times
    27

    View Slide

  28. Disadvantages
    • Quite hard to setup

    Copy DEX to the appropriate dir, create ClassLoader instance
    • Quite hard to work with

    Esp. if you rely on Reflection
    • Versioning!

    How to handle updates?
    28

    View Slide

  29. Improvements
    • Same approach as Google’s MultiDex

    Patches the Class Loader so to look for other DEX files
    • Specify the dependencies as ‘provided’

    So to be resolved at compile/build time, stripped in the final APK
    • Best of two worlds (kind of)

    Removes the need for ifaces, Reflection, still a bit cumbersome
    29

    View Slide

  30. Shrinking / Reasoning
    30

    View Slide

  31. — Google MultiDex online documentation
    Before configuring your app to enable use of 65K or
    more method references, you should take steps to
    reduce the total number of references called by your
    app code, including methods defined by your app code
    or included libraries.
    31

    View Slide

  32. ProGuard in pills
    • Shrink, optimize, obfuscate
    • Works both for your code and for third-party libraries
    • Has an impact on APK size, methods number, performance and
    security
    32

    View Slide

  33. ProGuard ProTips
    • Get familiar with writing appropriate rules

    The stricter, the better
    33
    -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}
    • What’s wrong with the following keep rule?
    -keep class com.google.** { *; }
    -keepnames class com.google.** { *; }
    -keepclassmembernames class * {
    @com.google.android.gms.common.annotation.KeepName *;
    }
    • Adopts an exclusion-based approach

    View Slide

  34. ProGuard ProTips
    34
    -keep class !android.support.v7.internal.view.menu.**,android.support.** {*;}
    -keep class com.google.** { *; }
    -keepnames class com.google.** { *; }
    -keepclassmembernames class * {
    @com.google.android.gms.common.annotation.KeepName *;
    }
    • Adopts an exclusion-based approach
    • Get familiar with writing appropriate rules

    The stricter, the better
    • What’s wrong with the following keep rule?

    View Slide

  35. ProGuard: default rules
    35
    android {
    ...
    buildTypes {
    release {
    minifyEnabled true
    proguardFiles getDefaultProguardFile('proguard-android.txt'),
    'proguard-rules.pro'
    }
    }
    }
    • ProGuard already includes common rules for common cases

    ~/sdk/tools/proguard/lib/proguard-android.txt
    -keepclasseswithmembernames class * {
    native ;
    }

    View Slide

  36. DexGuard: beefier, for beefier companies
    36
    • ProGuard’s bigger (and pricier) brother
    • Advanced features: res/strings/.so encryption, stronger obfuscation,

    tamper detection, its own multidex! and more
    • Pluggable encryption!

    I mean, seriously, that exclamation mark has a reason
    • Default rules are much more extensive

    View Slide

  37. ProGuard and third-party libs
    37
    • Many provide the ProGuard statement (a.k.a RTFM)

    e.g., Square, Facebook, …
    • Pay attention to ‘keep-all’ rules!

    especially for libraries with many methods

    View Slide

  38. Bad example
    38
    • Stetho (Facebook)
    # Keep Calm and Don’t Warn

    View Slide

  39. … wait, how do I count libraries
    methods?
    39
    … how do I count methods at all?!

    View Slide

  40. inloop’s “APK method count”
    40

    View Slide

  41. inloop’s “APK method count”
    41

    View Slide

  42. inloop’s “APK method count”
    • Simple website with drag-and-drop interface
    • Drop the APK and see the (locally) computed count

    No APKs uploaded :+1:
    • http://inloop.github.io/apk-method-count/

    42
    ✓ Easy, immediate, private


    ✗ Build every time

    View Slide

  43. ClassyShark
    • Useful Java tool to inspect APKs

    Written by Boris Farber (Googler)
    • Inspect number of classes, members, methods, dependencies, …

    Acts as a simple decompiler, too


    43

    View Slide

  44. ClassyShark
    44

    View Slide

  45. ClassyShark
    45

    View Slide

  46. ClassyShark
    • Useful Java tool to inspect APKs

    Written by Boris Farber (Googler)
    • Inspect number of classes, members, methods, dependencies, …

    Acts as a simple decompiler, too


    46
    ✓ Easy, plenty of info, multi-purpose


    ✗ Build every time, more oriented to agnostic debugging

    View Slide

  47. dexcount-gradle-plugin
    • Gradle plugin, developed by KeepSafe
    • See the current methods count every time you build

    Just open the Gradle console
    • Outputs per-package information in a ${variant}.txt file


    47

    View Slide

  48. dexcount-gradle-plugin
    48

    View Slide

  49. dexcount-gradle-plugin
    49

    View Slide

  50. dexcount-gradle-plugin
    • Gradle plugin, developed by KeepSafe
    • See the current methods count every time you build

    Just open the Gradle console
    • Outputs per-package information in a ${variant}.txt file


    50
    ✓ Keep the count controlled at a glance


    ✗ Cumbersome to see single deps count

    View Slide

  51. MethodsCount
    • Developed by Sebastiano Gottardo, Nicola Miotto, Dario Marcato

    • Allows to see methods and dependencies for libraries beforehand

    • Compare different libraries in terms of “lightweight-ness”
    51

    View Slide

  52. MethodsCount
    52
    • More than 15000 libraries and dependencies

    What’s missing is calculated upon the first request
    • For each entry
    1. dex/jar size
    2. direct count + dependencies count
    3. multiple versions (monitor changes over time)


    View Slide

  53. MethodsCount
    53

    View Slide

  54. MethodsCount
    54

    View Slide

  55. MethodsCount
    55

    View Slide

  56. MethodsCount - AS plugin
    56
    • Parses `build.gradle` and directly shows the count
    • Convenient when you drop a dependency right away

    View Slide

  57. MethodsCount - AS plugin
    57
    • Parses `build.gradle` and directly shows the count
    • Convenient when you drop a dependency right away

    View Slide

  58. MethodsCount
    58
    ✓ See the count before having to hit the limit


    ✓ Compare different libraries

    Worth having 4000 methods instead of 900?


    ✓ Analyze dependencies

    No, Guava is not a good excuse to have 15000+ methods


    ✓ Integrated in AS, thanks to the plugin

    ✗ The count is before ProGuard → Upper bound!

    View Slide

  59. Bottom line
    59

    View Slide

  60. Bottom line
    • The limit is here, at least for some time

    Learn to deal with it
    • The recommended way is “Do NOT cross the limit!”

    And use ProGuard. Your nerves will thank you later!
    • If you do need to cross, there are solutions (with drawbacks)

    MultiDex, secondary DEX
    60

    View Slide

  61. Routine
    1. PM shows interest in a new library

    :eyes_roll:
    2. With dex-gradle-plugin, check the current count
    3. Check MethodsCount to see how many methods, size and deps
    4. Evaluate whether fits a secondary DEX
    5. Consider alternatives (again, MethodsCount)
    61

    View Slide

  62. Useful resources
    • Timothy Mellor: https://github.com/tmelz/multidex_notes
    62
    @rotxed
    +SebastianoGottardo
    Follow me

    View Slide