Pro Yearly is on sale from $80 to $50! »

Write your own ProGuard

Write your own ProGuard

ProGuard: We all use it, but how much do we really know about it? Obviously, it's something that Google recommends that we use. And somewhere you read that it makes the code smaller. But what does it really do?

In this presentation, we'll answer that question together by developing a replacement for it. Like the real ProGuard, it will be able to shrink, optimize, and obfuscate code. The difference from ProGuard will be that we understand every step.

730df0227780e818df8ce1e19c9a6c48?s=128

Edward Dale

December 01, 2017
Tweet

Transcript

  1. Write your own ProGuard Edward Dale @scompt

  2. Write your own ProGuard Edward Dale @scompt R8

  3. ProGuard: Purpose ProGuard is the most popular optimizer for Java

    bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard
  4. ProGuard: Purpose ProGuard is the most popular optimizer for Java

    bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard
  5. ProGuard: Highlights • ProGuard is a command-line tool with an

    optional graphical user interface. • ProGuard is easy to configure. A few intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire run-time libraries in seconds. • ProGuard is the default tool in development environments like Oracle’s Wireless Toolkit, NetBeans, EclipseME, Intel’s TXE SDK and Google’s Android SDK.
  6. ProGuard: Highlights • ProGuard is easy to configure. A few

    intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire runtime libraries in seconds.
  7. ProGuard Steps Shrink Optimize Obfuscate Preverify

  8. ProGuard Steps Shrink Optimize Obfuscate Preverify

  9. Transform API Build Steps S O O P Compile Assemble

    Process .java .xml Dex .apk
  10. Build Steps Compile Assemble Transform API Process .java .xml Dex

    .apk
  11. Transform API The goal of this API is to simplify

    injecting custom class manipulations without having to deal with tasks, and to offer more flexibility on what is manipulated. http://tools.android.com/tech-docs/new-build-system/transform-api
  12. Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled

    Bytecode Resources Libraries Other Files Transform
  13. Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled

    Bytecode Resources Libraries Other Files Compiled Bytecode
  14. Transform API Examples JacocoTransform https://goo.gl/jNoA2w Instruments code for measuring coverage

    MultiDexTransform https://goo.gl/MxnuiX Gathers information about the project to configure MultiDex in a later stage ProGuardTransform https://goo.gl/h7CGb4 Runs ProGuard…
  15. Transform API 1 class MyPlugin implements Plugin<Project> { 2 @Override

    3 void apply(Project p) { 4 def t = new WriteYourOwnProGuardTransform(p) 5 p.android.registerTransform(t) 6 } 7 } 1 apply plugin: MyPlugin build.gradle MyPlugin.groovy
  16. 1 class WriteYourOwnProGuardTransform(val project: Project) 2 : Transform() { 3

    4 override fun isIncremental() = false 5 6 override fun getName() = "writeYourOwnProGuard" 7 8 @Throws(TransformException::class, 9 InterruptedException::class, 10 IOException::class) 11 override fun transform(invocation: TransformInvocation) { WriteYourOwnProGuardTransform.kotlin
  17. 1 override fun getInputTypes() = 2 setOf(QualifiedContent.DefaultContentType.CLASSES, 3 QualifiedContent.DefaultContentType.RESOURCES) 4

    5 override fun getScopes() = 6 setOf(QualifiedContent.Scope.PROJECT, 7 QualifiedContent.Scope.SUB_PROJECTS, 8 QualifiedContent.Scope.EXTERNAL_LIBRARIES) WriteYourOwnProGuardTransform.kotlin
  18. 1 public interface TransformInvocation { 2 Context getContext(); 3 4

    Collection<TransformInput> getInputs(); 5 6 Collection<TransformInput> getReferencedInputs(); 7 8 Collection<SecondaryInput> getSecondaryInputs(); 9 10 TransformOutputProvider getOutputProvider(); 11 12 boolean isIncremental(); 13 } 1 public interface TransformInput { 2 Collection<JarInput> getJarInputs(); 3 4 Collection<DirectoryInput> getDirectoryInputs(); 5 }
  19. 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/ 5 3cba64e049f7e363d1d5aa6ed4ca7a3f/

    6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}
  20. .jar .class 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/

    5 3cba64e049f7e363d1d5aa6ed4ca7a3f/ 6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}
  21. Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/

    JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/
  22. Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/

    JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/
  23. Transform API Build Steps S O O P Compile Assemble

    Process .java .xml Dex .apk
  24. ProGuard Steps Shrink Optimize Obfuscate Preverify

  25. Shrink • Removes all classes, methods, resources not reachable from

    an entry point (seeds) • Dynamically referenced classes/methods need to be "kept" using -keep or -keepclasseswithmembers
  26. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager

  27. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = []

  28. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = [FeedActivity,

    LoginActivity]
  29. Shrink 1. Find all classes in the program 2. Find

    all connections between classes in the program 3. Find all “seed” classes 4. Find all classes reachable from seeds 5. Remove all unreachable classes
  30. Shrink 1. Find all classes in the program 1 for

    (directoryInput in input.directoryInputs) { 2 val root = directoryInput.file.toPath() 3 Files.walkFileTree(root, /* Accumulate class name */) 1 val classPool = ClassPool.getDefault() 2 for (input in transformInvocation.inputs) { 3 for (jarInput in input.jarInputs) { 4 val stream = Files.newInputStream(jarInput.file.toPath()) 5 JarInputStream(stream).use { 6 var entry: JarEntry? = it.nextJarEntry 7 while (entry != null) { 8 val klass = classPool.makeClass(stream) 9 // Accumulate class name
  31. Shrink 2. Find all connections between classes in the program

    1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames
  32. Shrink 2. Find all connections between classes in the program

    1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames constPool
  33. “For each type it loads, a Java virtual machine must

    store a constant pool. A constant pool is an ordered set of constants used by the type, including literals (string, integer, and floating point constants) and symbolic references to types, fields, and methods. Entries in the constant pool are referenced by index, much like the elements of an array. Because it holds symbolic references to all types, fields, and methods used by a type, the constant pool plays a central role in the dynamic linking of Java programs” –Inside the Java Virtual Machine by Bill Venners
  34. Shrink 2. Find all connections between classes in the program

    1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames
  35. Shrink 3. Find all “seed” classes 1 val isSeedClass =

    klass.name.endsWith(“Activity")
  36. Shrink 4. Find all classes reachable from seeds 1 private

    fun buildGraph(deps: Multimap<String, String>) 2 : Graph<String, DefaultEdge> { 3 4 val graph = DefaultDirectedGraph() 5 deps.keySet() 6 .filterNot { it.startsWith("java.lang") } 7 .forEach { graph.addVertex(it) } 8 deps.values() 9 .filterNot { it.startsWith("java.lang") } 10 .forEach { graph.addVertex(it) } 11 deps.entries() 12 .filterNot { it.key.startsWith("java.lang") } 13 .filterNot { it.value.startsWith("java.lang") } 14 .filterNot { it.key == it.value } 15 .forEach { graph.addEdge(it.key, it.value) } 16 return graph 17 }
  37. Shrink 4. Find all classes reachable from seeds 1 graph.vertexSet()

    2 .filter { it.endsWith("Activity") } 3 .map { DepthFirstIterator(graph, it).asSequence().toSet() } 4 .flatten()
  38. Shrink 5. Remove all unreachable classes 1 override fun transform(invocation:

    TransformInvocation) { 2 val classNames = getAllClassNames(invocation) 3 val connections = findConnections(classNames) 4 val classGraph = buildGraph(connections) 5 val reachableClasses = findReachableClasses(classGraph) 6 7 // for-each JAR file 8 // remove unreachable classes 9 // repackage JAR file 10 // copy JAR file in output 11 12 // for-each class file 13 // if reachable copy to output
  39. None
  40. None
  41. None
  42. None
  43. ProGuard Steps Shrink Optimize Obfuscate Preverify

  44. Optimize • Performs lots of different bytecode-level optimizations to the

    code • For concrete example, see Jake Wharton’s recent “Sinking Your Teeth Into Bytecode” presentation https://goo.gl/8hnRzP
  45. Optimize • Marks methods as final, whenever possible. • Removes

    unused method parameters. • Propagates the values of method parameters from method invocations to the invoked methods. • Propagates the values of method return values from methods to their invocations. • Inlines short methods. • Inlines methods that are only called once.
  46. ProGuard Steps Shrink Optimize Obfuscate Preverify

  47. Obfuscate • Classes and class members receive new short random

    names, except for the ones listed by the various -keep options • Internal attributes that are useful for debugging are removed
  48. Obfuscate FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager

  49. Obfuscate FeedActivity A B C LoginActivity OldUserManager

  50. Obfuscate

  51. Resources • Example ProGuard-like transform – Edward Dale
 https://github.com/scompt/WriteYourOwnProGuard •

    “Sinking Your Teeth Into Bytecode” – Jake Wharton
 http://jakewharton.com/sinking-your-teeth-into-bytecode/ • Gephi – The Open Graph Viz Platform
 https://gephi.org/ • JGraphT
 http://jgrapht.org/
  52. Image Citations https://emojipedia.org https://freeletics.com https://gephi.org http://www.cuelogic.com http://knowyourmeme.com https://imgflip.com https://www.flaticon.com https://www.jgrapht.org

  53. Questions? Edward Dale @scompt