From ProGuard to R8: take care of your byte code

Presented at AndroidMakers Paris 2018
https://androidmakers.fr/schedule/#session-26

Video: https://youtu.be/w3bOfxYNY8o

Transcript

1. From ProGuard to R8: take care of your bytecode Qian JIN | @bonbonking | qjin@xebia.fr 23 April 2018
2. Me Configuring ProGuard (in 2016)
3. Agenda • ProGuard in Android Build Process • Why ProGuard? • ProGuard Essentials • Understand the default ProGuard settings • ProGuard like a Pro • What do we know about R8? • Takeaways
4. $ ./gradlew assembleRelease (Android Build Process)
5. ProGuard is a open source Java class file shrinker, optimizer, obfuscator, and preverifier, created by Eric Lafortune, initially released in May 2002, latest version 6.0 was released in February 2018
6. Reference: https://www.guardsquare.com/en/proguard/manual/introduction Input jars Shrunk code Optimized code Obfuscated code Output jars
7. Shrinking Image reference: https://www.gutenberg.org/files/28847/28847-h/28847-h.htm Input jars Shrunk code Optimized code Obfuscated code Output jars
8. Optimization Reference: https://xkcd.com/1691/ Input jars Shrunk code Optimized code Obfuscated code Output jars
9. Obfuscation public class ExampleClass { private MySettings settings; private MyConfigurations configurations; private int result; private int calculateResult(int input) { // ... return result; } } public class a { private b a; private c b; private int c; private int a(int a) { // ... return c; } } Input jars Shrunk code Optimized code Obfuscated code Output jars
10. Reference: https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html App source code (*.java or *.kt) JAR Library bytecode (*.class) AAR Library bytecode (*.class) App Java bytecode (*.class) javac / kotlinc ProGuard / R8 Optimized Java bytecode (*.class) DEX bytecode (*.dex) DX / D8 * D8 is the default DEX compiler in Android Studio 3.1 release * R8 is available for test starting from Android Studio 3.2 alpha 6
11. Reference: https://android-developers.googleblog.com/2017/08/next-generation-dex-compiler-now-in.html App source code (*.java or *.kt) JAR Library bytecode (*.class) AAR Library bytecode (*.class) App Java bytecode (*.class) javac / kotlinc ProGuard / R8 Optimized Java bytecode (*.class) DEX bytecode (*.dex) DX / D8 ProGuard Inputs (Rules) ProGuard Outputs (dump.txt, mapping.txt, seeds.txt, usage.txt)
12. $ proguard -why (Why ProGuard?)
13. Streamline Your Application! • Resource shrinking • Smaller APK size • Smaller download size • Smaller DEX size Reference: https://en.wikipedia.org/wiki/We_Can_Do_It!
14. Debug (Without ProGuard, minifyEnabled=false, shrinkResources=false) Release (With ProGuard, minifyEnabled=true, shrinkResources=true) APK Size 30.7 MB 23.5 MB* Dex Method Count 151,235 109,634** Without/With ProGuard Reference: https://github.com/mihaip/dex-method-counts *app size is 30% smaller ** dex count is reduced by 37% v3.26.2
15. Harder to Reverse Engineer Reference: https://pixabay.com/fr/masque-anonyme-face-panneau-1587566/
16. Reference: https://github.com/skylot/jadx Kept Class
17. Reference: https://github.com/skylot/jadx Obfuscated Class
18. $ man proguard (ProGuard essentials)
19. The ProGuard fundamental Entry Point Reference: https://www.guardsquare.com/en/proguard/manual/introduction
20. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point Live Code
21. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point Live Code
22. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point Live Code Dead Code
23. Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74 MyActivity MyClass1 MyClass2 SomeClass3 SomeClass4 Program Entry Point Live Code Dead Code
24. How to define an Entry Point?
25. Entry Point Example -keepclassmembers class * extends android.app.Activity { public void *(android.view.View); }
26. -keepclassmembers class * extends android.app.Activity { public void *(android.view.View); } Entry Point Example Reference: https://www.guardsquare.com/en/proguard/manual/usage#classspecification Keep Option Class Specification
27. -keep -keepclassmembers -keepclasswithmembers -keepnames -keepclassmembernames -keepclasswithmembernames -keep,-allowshrinking -keep,-allowoptimization -keep,-allow-obfuscation @Keep …. Reference: https://www.guardsquare.com/en/proguard/manual/usage#keepoptions
29. Reference: https://www.guardsquare.com/en/proguard/manual/usage#keepoverview From being removed or renamed From being renamed (Short for -keep,allowshrinking) -keep -keepnames -keepclassmembers -keepclassmembernames -keepclasseswithmembers -keepclasseswithmembernames
30. proguard-android.txt @Keep annotation # Understand the @Keep support annotation. -keep class android.support.annotation.Keep -keep @android.support.annotation.Keep class * {*;} -keepclasseswithmembers class * { @android.support.annotation.Keep <methods>; } -keepclasseswithmembers class * { @android.support.annotation.Keep <fields>; } -keepclasseswithmembers class * { @android.support.annotation.Keep <init>(...); }
31. getDefaultProguardFile (Understand the default ProGuard settings)
32. ProGuard Inputs (Rules)
33. /build/intermediates/proguard-files/
34. proguard-android.txt proguard-android-optimize.txt 2 default ProGuard templates
35. $ diff proguard-android.txt proguard-android-optimize.txt -dontoptimize -optimizations !code/ simplification/arithmetic,! code/simplification/cast,! field/*,!class/merging/* -optimizationpasses 5 -allowaccessmodification proguard-android.txt proguard-android-optimize.txt
36. /app/build.gradle Generated by default buildTypes { release { proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } } ➡ This does nothing!
37. /app/build.gradle Enable code shrinking buildTypes { release { minifyEnabled true proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } } ➡ minifyEnabled shrinks the code with the default proguard rules
38. /app/build.gradle Enable code & resource shrinking buildTypes { release { minifyEnabled true shrinkResources true proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } }
39. /app/build.gradle Enable code optimization buildTypes { release { minifyEnabled true shrinkResources true proguardFiles getDefaultProguardFile( 'proguard-android-optimize.txt'), 'proguard-rules.pro' } } ➡ Optimization is turned off by default. ➡ Dex does not like code run through the ProGuard optimization steps.
40. /app/build.gradle Shrink code for Instant Run Reference: https://developer.android.com/studio/build/shrink-code.html#gradle-shrinker buildTypes { debug { minifyEnabled true useProguard false proguardFiles getDefaultProguardFile( 'proguard-android.txt'), 'proguard-rules.pro' } }
41. ProGuard Outputs dump.txt, mapping.txt, seeds.txt, usage.txt
42. How to deobfuscate crash stack traces?
43. Reference: Reference: https://developers.google.com/android-publisher/api-ref/edits/deobfuscationfiles/upload POST https://www.googleapis.com/upload/androidpublisher/v2/ applications/packageName/edits/editId/apks/apkVersionCode/ deobfuscationFiles/deobfuscationFileType deobfuscationFileType: ProGuard
44. > proguard-rules.pro (ProGuard like a Pro)
45. What should you keep?
46. MagritteModel.kt version 1.0 Reference: https://github.com/xebia-france/magritte package fr.xebia.magritte.model data class MagritteModel(val id: String, val name: String, val description: String, val configuration: MagritteConfiguration)
47. proguard.pro version 1.0 -keep class fr.xebia.magritte.model.** {*;} -dontwarn okio.** -dontwarn javax.annotation.** -keepclasseswithmembers class * { @com.squareup.moshi.* <methods>; } -keep @com.squareup.moshi.JsonQualifier interface *
48. MagritteModel.kt version 2.0 Reference: https://github.com/xebia-france/magritte package fr.xebia.magritte.model import android.os.Parcelable import com.squareup.moshi.Json import kotlinx.android.parcel.Parcelize @Parcelize data class MagritteModel( @Json(name = "id") val id: String, @Json(name = "name") val name: String, @Json(name = "description") val description: String, @Json(name = "model_file_path") val modelFilePath: String, @Json(name = "model_type") val modelType: TFModelType, @Json(name = "configuration") val configuration: MagritteConfiguration ) : Parcelable
49. Reference: https://speakerdeck.com/takhion/unleash-the-secret-power-of-kotlin-metadata Kotlin Metadata informations are GONE!
50. proguard.pro version 2.0 Reference: https://github.com/square/moshi/issues/345 -keep class kotlin.Metadata { *; } -keepclassmembers class kotlin.Metadata { public <methods>; } -keep class fr.xebia.magritte.model.** {*;} -dontwarn okio.** -dontwarn javax.annotation.** -keepclasseswithmembers class * { @com.squareup.moshi.* <methods>; } -keep @com.squareup.moshi.JsonQualifier interface *
52. Reference: https://speakerdeck.com/takhion/unleash-the-secret-power-of-kotlin-metadata k -> kind mv -> meta data version bv -> byte code version d1 -> data 1 d2 -> data 2 xs -> extra string xi -> extra int
53. How to configure ProGuard rules for 3rd party libraries?
54. When to use -dontwarn? Example: Retrofit2 complains about can’t find annotation classes javax.annotation.Nullable (from JSR 305) Reference: https://medium.com/google-developers/troubleshooting-proguard-issues-on-android-bce9de4f8a74
55. Reference: https://github.com/square/retrofit/blob/master/retrofit/pom.xml#L33-L37 <scope>provided</scope>
56. proguard.pro -dontwarn okhttp3.** -dontwarn okio.** -dontwarn javax.annotation.** -dontwarn org.conscrypt.**
57. android.enableR8=true (What do we know about R8?)
58. Reference: https://twitter.com/JakeWharton/status/982382740019564544 Lars Bak who likes short code name?
59. What we know about R8? • R8 & D8 share the same code base • R8 is a ProGuard replacement for whole-program optimization, shrinking and minification. • R8 uses the same ProGuard keep rule format & command line arguments • R8 implements different optimisations than ProGuard • R8 supports Instant Run*(?) • Available from Android Studio 3.2 alpha 6 • Build faster than ProGuard & decrease dex file size by 34% for some app** *Reference: https://developer.android.com/studio/build/shrink-code.html#gradle-shrinker **Reference: http://androidbackstage.blogspot.fr/2018/01/episode-86-its-gr8.html
60. gradle.properties android.enableR8=true
61. Reference: https://twitter.com/JakeWharton/status/973291095152087041
62. R8 Benchmarks * Magritte: https://github.com/xebia-france/magritte ** Tested with Android Studio 3.2 alpha 11 Magritte* Release (ProGuard without optimization) Release (R8** without optimization) Release (ProGuard with optimization) Release (R8** with optimization) APK Size 22.1MB 23.2MB 23MB 23MB Dex Count 25758 23199 (-10%) ? Build Time 52s 57s
63. R8 Benchmarks * Magritte: https://github.com/xebia-france/magritte ** Tested with Android Studio 3.2 alpha 11 Magritte* Release (ProGuard without optimization) Release (R8** without optimization) Release (ProGuard with optimization) Release (R8** with optimization) APK Size 22.1MB 23.2MB 23MB 23MB Dex Count 25758 23199 (-10%) 21527 18039 (-16.2%) Build Time 52s 57s 2m16s 55s (2.5x faster)
64. R8 Benchmarks * Plaid: https://github.com/nickbutcher/plaid by Nick Butcher ** Tested with Android Studio 3.2 alpha 11 Plaid* Release (ProGuard with optimization) Release (R8** with optimization) APK Size 6.8MB 6.7MB Dex Count 19049 16460 (-14%) Build Time 1m49s 36s (2.75x faster)
65. $ takeaway ls
66. Takeways • Read the source Luke • Read the byte code too Luke • Exception log is your friend • Dex decompiler is also your friend • Commercialized alternative: DexGuard Reference: https://blog.codinghorror.com/learn-to-read-the-source-luke/
67. Takeways • Consumer ProGuard file for your AAR library • Test with ProGuard • ProGuard for Instant App • …
68. KEEP CALM AND ENABLE PROGUARD / R8
69. Enjoy AndroidMakers 2018!