From ProGuard to R8: take care of your byte code

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
April 23, 2018

From ProGuard to R8: take care of your byte code

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

April 23, 2018
Tweet

Transcript

  1. From ProGuard to R8: take care of your bytecode Qian

    JIN | @bonbonking | qjin@xebia.fr 23 April 2018
  2. Me Configuring ProGuard (in 2016)

  3. Agenda • ProGuard in Android Build Process • Why ProGuard?

    • ProGuard Essentials • Understand the default ProGuard settings • ProGuard like a Pro • What do we know about R8? • Takeaways
  4. $ ./gradlew assembleRelease (Android Build Process)

  5. ProGuard is a open source Java class file shrinker, optimizer,

    obfuscator, and preverifier, created by Eric Lafortune, initially released in May 2002, latest version 6.0 was released in February 2018
  6. Reference: https://www.guardsquare.com/en/proguard/manual/introduction Input jars Shrunk code Optimized code Obfuscated code

    Output jars
  7. Shrinking Image reference: https://www.gutenberg.org/files/28847/28847-h/28847-h.htm Input jars Shrunk code Optimized code

    Obfuscated code Output jars
  8. Optimization Reference: https://xkcd.com/1691/ Input jars Shrunk code Optimized code Obfuscated

    code Output jars
  9. Obfuscation public class ExampleClass { private MySettings settings; private MyConfigurations

    configurations; private int result; private int calculateResult(int input) { // ... return result; } } public class a { private b a; private c b; private int c; private int a(int a) { // ... return c; } } Input jars Shrunk code Optimized code Obfuscated code Output jars
  10. Reference: https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html App source code (*.java or *.kt) JAR Library

    bytecode (*.class) AAR Library bytecode (*.class) App Java bytecode (*.class) javac / kotlinc ProGuard / R8 Optimized Java bytecode (*.class) DEX bytecode (*.dex) DX / D8 * D8 is the default DEX compiler in Android Studio 3.1 release * R8 is available for test starting from Android Studio 3.2 alpha 6
  11. Reference: https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html App source code (*.java or *.kt) JAR Library

    bytecode (*.class) AAR Library bytecode (*.class) App Java bytecode (*.class) javac / kotlinc ProGuard / R8 Optimized Java bytecode (*.class) DEX bytecode (*.dex) DX / D8 ProGuard Inputs (Rules) ProGuard Outputs (dump.txt, mapping.txt, seeds.txt, usage.txt)
  12. $ proguard -why (Why ProGuard?)

  13. Streamline Your Application! • Resource shrinking • Smaller APK size

    • Smaller download size • Smaller DEX size Reference: https://en.wikipedia.org/wiki/We_Can_Do_It!
  14. Debug (Without ProGuard, minifyEnabled=false, shrinkResources=false) Release (With ProGuard, minifyEnabled=true, shrinkResources=true)

    APK Size 30.7 MB 23.5 MB* Dex Method Count 151,235 109,634** Without/With ProGuard Reference: https://github.com/mihaip/dex-method-counts *app size is 30% smaller ** dex count is reduced by 37% v3.26.2
  15. Harder to Reverse Engineer Reference: https://pixabay.com/fr/masque-anonyme-face-panneau-1587566/

  16. Reference: https://github.com/skylot/jadx Kept Class

  17. Reference: https://github.com/skylot/jadx Obfuscated Class

  18. $ man proguard (ProGuard essentials)

  19. The ProGuard fundamental Entry Point Reference: https://www.guardsquare.com/en/proguard/manual/introduction

  20. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point

    Live Code
  21. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point

    Live Code
  22. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point

    Live Code Dead Code
  23. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point

    Live Code Dead Code
  24. How to define an Entry Point?

  25. Entry Point Example -keepclassmembers class * extends android.app.Activity { public

    void *(android.view.View); }
  26. -keepclassmembers class * extends android.app.Activity { public void *(android.view.View); }

    Entry Point Example Reference: https://www.guardsquare.com/en/proguard/manual/usage#classspecification Keep Option Class Specification
  27. -keep -keepclassmembers -keepclasswithmembers -keepnames -keepclassmembernames -keepclasswithmembernames -keep,-allowshrinking -keep,-allowoptimization -keep,-allow-obfuscation @Keep

    …. Reference: https://www.guardsquare.com/en/proguard/manual/usage#keepoptions
  28. None
  29. Reference: https://www.guardsquare.com/en/proguard/manual/usage#keepoverview From being removed or renamed From being renamed

    (Short for -keep,allowshrinking) -keep -keepnames -keepclassmembers -keepclassmembernames -keepclasseswithmembers -keepclasseswithmembernames
  30. proguard-android.txt @Keep annotation # Understand the @Keep support annotation. -keep

    class android.support.annotation.Keep -keep @android.support.annotation.Keep class * {*;} -keepclasseswithmembers class * { @android.support.annotation.Keep <methods>; } -keepclasseswithmembers class * { @android.support.annotation.Keep <fields>; } -keepclasseswithmembers class * { @android.support.annotation.Keep <init>(...); }
  31. getDefaultProguardFile (Understand the default ProGuard settings)

  32. ProGuard Inputs (Rules)

  33. /build/intermediates/proguard-files/

  34. proguard-android.txt proguard-android-optimize.txt 2 default ProGuard templates

  35. $ diff proguard-android.txt proguard-android-optimize.txt -dontoptimize -optimizations !code/ simplification/arithmetic,! code/simplification/cast,! field/*,!class/merging/*

    -optimizationpasses 5 -allowaccessmodification proguard-android.txt proguard-android-optimize.txt
  36. /app/build.gradle Generated by default buildTypes { release { proguardFiles getDefaultProguardFile(

    'proguard-android.txt'), 'proguard-rules.pro' } } ➡ This does nothing!
  37. /app/build.gradle Enable code shrinking buildTypes { release { minifyEnabled true

    proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } } ➡ minifyEnabled shrinks the code with the default proguard rules
  38. /app/build.gradle Enable code & resource shrinking buildTypes { release {

    minifyEnabled true shrinkResources true proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } }
  39. /app/build.gradle Enable code optimization buildTypes { release { minifyEnabled true

    shrinkResources true proguardFiles getDefaultProguardFile( 'proguard-android-optimize.txt'), 'proguard-rules.pro' } } ➡ Optimization is turned off by default. ➡ Dex does not like code run through the ProGuard optimization steps.
  40. /app/build.gradle Shrink code for Instant Run Reference: https://developer.android.com/studio/build/shrink-code.html#gradle-shrinker buildTypes {

    debug { minifyEnabled true useProguard false proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } }
  41. ProGuard Outputs dump.txt, mapping.txt, seeds.txt, usage.txt

  42. How to deobfuscate crash stack traces?

  43. Reference: Reference: https://developers.google.com/android-publisher/api-ref/edits/deobfuscationfiles/upload POST https://www.googleapis.com/upload/androidpublisher/v2/ applications/packageName/edits/editId/apks/apkVersionCode/ deobfuscationFiles/deobfuscationFileType deobfuscationFileType: ProGuard

  44. > proguard-rules.pro (ProGuard like a Pro)

  45. What should you keep?

  46. MagritteModel.kt version 1.0 Reference: https://github.com/xebia-france/magritte package fr.xebia.magritte.model data class MagritteModel(val

    id: String, val name: String, val description: String, val configuration: MagritteConfiguration)
  47. proguard.pro version 1.0 -keep class fr.xebia.magritte.model.** {*;} -dontwarn okio.** -dontwarn

    javax.annotation.** -keepclasseswithmembers class * { @com.squareup.moshi.* <methods>; } -keep @com.squareup.moshi.JsonQualifier interface *
  48. MagritteModel.kt version 2.0 Reference: https://github.com/xebia-france/magritte package fr.xebia.magritte.model import android.os.Parcelable import

    com.squareup.moshi.Json import kotlinx.android.parcel.Parcelize @Parcelize data class MagritteModel( @Json(name = "id") val id: String, @Json(name = "name") val name: String, @Json(name = "description") val description: String, @Json(name = "model_file_path") val modelFilePath: String, @Json(name = "model_type") val modelType: TFModelType, @Json(name = "configuration") val configuration: MagritteConfiguration ) : Parcelable
  49. Reference: https://speakerdeck.com/takhion/unleash-the-secret-power-of-kotlin-metadata Kotlin Metadata informations are GONE!

  50. proguard.pro version 2.0 Reference: https://github.com/square/moshi/issues/345 -keep class kotlin.Metadata { *;

    } -keepclassmembers class kotlin.Metadata { public <methods>; } -keep class fr.xebia.magritte.model.** {*;} -dontwarn okio.** -dontwarn javax.annotation.** -keepclasseswithmembers class * { @com.squareup.moshi.* <methods>; } -keep @com.squareup.moshi.JsonQualifier interface *
  51. None
  52. Reference: https://speakerdeck.com/takhion/unleash-the-secret-power-of-kotlin-metadata k -> kind mv -> meta data version

    bv -> byte code version d1 -> data 1 d2 -> data 2 xs -> extra string xi -> extra int
  53. How to configure ProGuard rules for 3rd party libraries?

  54. When to use -dontwarn? Example: Retrofit2 complains about can’t find

    annotation classes javax.annotation.Nullable (from JSR 305) Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74
  55. Reference: https://github.com/square/retrofit/blob/master/retrofit/pom.xml#L33-L37 <scope>provided</scope>

  56. proguard.pro -dontwarn okhttp3.** -dontwarn okio.** -dontwarn javax.annotation.** -dontwarn org.conscrypt.**

  57. android.enableR8=true (What do we know about R8?)

  58. Reference: https://twitter.com/JakeWharton/status/982382740019564544 Lars Bak who likes short code name?

  59. What we know about R8? • R8 & D8 share

    the same code base • R8 is a ProGuard replacement for whole-program optimization, shrinking and minification. • R8 uses the same ProGuard keep rule format & command line arguments • R8 implements different optimisations than ProGuard • R8 supports Instant Run*(?) • Available from Android Studio 3.2 alpha 6 • Build faster than ProGuard & decrease dex file size by 34% for some app** *Reference: https://developer.android.com/studio/build/shrink-code.html#gradle-shrinker **Reference: http://androidbackstage.blogspot.fr/2018/01/episode-86-its-gr8.html
  60. gradle.properties android.enableR8=true

  61. Reference: https://twitter.com/JakeWharton/status/973291095152087041

  62. R8 Benchmarks * Magritte: https://github.com/xebia-france/magritte ** Tested with Android Studio

    3.2 alpha 11 Magritte* Release (ProGuard without optimization) Release (R8** without optimization) Release (ProGuard with optimization) Release (R8** with optimization) APK Size 22.1MB 23.2MB 23MB 23MB Dex Count 25758 23199 (-10%) ? Build Time 52s 57s
  63. R8 Benchmarks * Magritte: https://github.com/xebia-france/magritte ** Tested with Android Studio

    3.2 alpha 11 Magritte* Release (ProGuard without optimization) Release (R8** without optimization) Release (ProGuard with optimization) Release (R8** with optimization) APK Size 22.1MB 23.2MB 23MB 23MB Dex Count 25758 23199 (-10%) 21527 18039 (-16.2%) Build Time 52s 57s 2m16s 55s (2.5x faster)
  64. R8 Benchmarks * Plaid: https://github.com/nickbutcher/plaid by Nick Butcher ** Tested

    with Android Studio 3.2 alpha 11 Plaid* Release (ProGuard with optimization) Release (R8** with optimization) APK Size 6.8MB 6.7MB Dex Count 19049 16460 (-14%) Build Time 1m49s 36s (2.75x faster)
  65. $ takeaway ls

  66. Takeways • Read the source Luke • Read the byte

    code too Luke • Exception log is your friend • Dex decompiler is also your friend • Commercialized alternative: DexGuard Reference: https://blog.codinghorror.com/learn-to-read-the-source-luke/
  67. Takeways • Consumer ProGuard file for your AAR library •

    Test with ProGuard • ProGuard for Instant App • …
  68. KEEP CALM AND ENABLE PROGUARD / R8

  69. Enjoy AndroidMakers 2018!