Write your own ProGuard

ProGuard: We all use it, but how much do we really know about it? Obviously, it's something that Google recommends that we use. And somewhere you read that it makes the code smaller. But what does it really do?

In this presentation, we'll answer that question together by developing a replacement for it. Like the real ProGuard, it will be able to shrink, optimize, and obfuscate code. The difference from ProGuard will be that we understand every step.

Transcript

1. Write your own ProGuard Edward Dale @scompt
2. Write your own ProGuard Edward Dale @scompt R8
3. ProGuard: Purpose ProGuard is the most popular optimizer for Java bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard
4. ProGuard: Purpose ProGuard is the most popular optimizer for Java bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard
5. ProGuard: Highlights • ProGuard is a command-line tool with an optional graphical user interface. • ProGuard is easy to configure. A few intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire run-time libraries in seconds. • ProGuard is the default tool in development environments like Oracle’s Wireless Toolkit, NetBeans, EclipseME, Intel’s TXE SDK and Google’s Android SDK.
6. ProGuard: Highlights • ProGuard is easy to configure. A few intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire runtime libraries in seconds.
7. ProGuard Steps Shrink Optimize Obfuscate Preverify
8. ProGuard Steps Shrink Optimize Obfuscate Preverify
9. Transform API Build Steps S O O P Compile Assemble Process .java .xml Dex .apk
10. Build Steps Compile Assemble Transform API Process .java .xml Dex .apk
11. Transform API The goal of this API is to simplify injecting custom class manipulations without having to deal with tasks, and to offer more flexibility on what is manipulated. http://tools.android.com/tech-docs/new-build-system/transform-api
12. Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled Bytecode Resources Libraries Other Files Transform
13. Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled Bytecode Resources Libraries Other Files Compiled Bytecode
14. Transform API Examples JacocoTransform https://goo.gl/jNoA2w Instruments code for measuring coverage MultiDexTransform https://goo.gl/MxnuiX Gathers information about the project to configure MultiDex in a later stage ProGuardTransform https://goo.gl/h7CGb4 Runs ProGuard…
15. Transform API 1 class MyPlugin implements Plugin<Project> { 2 @Override 3 void apply(Project p) { 4 def t = new WriteYourOwnProGuardTransform(p) 5 p.android.registerTransform(t) 6 } 7 } 1 apply plugin: MyPlugin build.gradle MyPlugin.groovy
16. 1 class WriteYourOwnProGuardTransform(val project: Project) 2 : Transform() { 3 4 override fun isIncremental() = false 5 6 override fun getName() = "writeYourOwnProGuard" 7 8 @Throws(TransformException::class, 9 InterruptedException::class, 10 IOException::class) 11 override fun transform(invocation: TransformInvocation) { WriteYourOwnProGuardTransform.kotlin
17. 1 override fun getInputTypes() = 2 setOf(QualifiedContent.DefaultContentType.CLASSES, 3 QualifiedContent.DefaultContentType.RESOURCES) 4 5 override fun getScopes() = 6 setOf(QualifiedContent.Scope.PROJECT, 7 QualifiedContent.Scope.SUB_PROJECTS, 8 QualifiedContent.Scope.EXTERNAL_LIBRARIES) WriteYourOwnProGuardTransform.kotlin
18. 1 public interface TransformInvocation { 2 Context getContext(); 3 4 Collection<TransformInput> getInputs(); 5 6 Collection<TransformInput> getReferencedInputs(); 7 8 Collection<SecondaryInput> getSecondaryInputs(); 9 10 TransformOutputProvider getOutputProvider(); 11 12 boolean isIncremental(); 13 } 1 public interface TransformInput { 2 Collection<JarInput> getJarInputs(); 3 4 Collection<DirectoryInput> getDirectoryInputs(); 5 }
19. 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/ 5 3cba64e049f7e363d1d5aa6ed4ca7a3f/ 6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}
20. .jar .class 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/ 5 3cba64e049f7e363d1d5aa6ed4ca7a3f/ 6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}
21. Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/ JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/
22. Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/ JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/
23. Transform API Build Steps S O O P Compile Assemble Process .java .xml Dex .apk
24. ProGuard Steps Shrink Optimize Obfuscate Preverify
25. Shrink • Removes all classes, methods, resources not reachable from an entry point (seeds) • Dynamically referenced classes/methods need to be "kept" using -keep or -keepclasseswithmembers
26. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager
27. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = []
28. Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = [FeedActivity, LoginActivity]
29. Shrink 1. Find all classes in the program 2. Find all connections between classes in the program 3. Find all “seed” classes 4. Find all classes reachable from seeds 5. Remove all unreachable classes
30. Shrink 1. Find all classes in the program 1 for (directoryInput in input.directoryInputs) { 2 val root = directoryInput.file.toPath() 3 Files.walkFileTree(root, /* Accumulate class name */) 1 val classPool = ClassPool.getDefault() 2 for (input in transformInvocation.inputs) { 3 for (jarInput in input.jarInputs) { 4 val stream = Files.newInputStream(jarInput.file.toPath()) 5 JarInputStream(stream).use { 6 var entry: JarEntry? = it.nextJarEntry 7 while (entry != null) { 8 val klass = classPool.makeClass(stream) 9 // Accumulate class name
31. Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames
32. Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames constPool
33. “For each type it loads, a Java virtual machine must store a constant pool. A constant pool is an ordered set of constants used by the type, including literals (string, integer, and floating point constants) and symbolic references to types, fields, and methods. Entries in the constant pool are referenced by index, much like the elements of an array. Because it holds symbolic references to all types, fields, and methods used by a type, the constant pool plays a central role in the dynamic linking of Java programs” –Inside the Java Virtual Machine by Bill Venners
34. Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames
35. Shrink 3. Find all “seed” classes 1 val isSeedClass = klass.name.endsWith(“Activity")
36. Shrink 4. Find all classes reachable from seeds 1 private fun buildGraph(deps: Multimap<String, String>) 2 : Graph<String, DefaultEdge> { 3 4 val graph = DefaultDirectedGraph() 5 deps.keySet() 6 .filterNot { it.startsWith("java.lang") } 7 .forEach { graph.addVertex(it) } 8 deps.values() 9 .filterNot { it.startsWith("java.lang") } 10 .forEach { graph.addVertex(it) } 11 deps.entries() 12 .filterNot { it.key.startsWith("java.lang") } 13 .filterNot { it.value.startsWith("java.lang") } 14 .filterNot { it.key == it.value } 15 .forEach { graph.addEdge(it.key, it.value) } 16 return graph 17 }
37. Shrink 4. Find all classes reachable from seeds 1 graph.vertexSet() 2 .filter { it.endsWith("Activity") } 3 .map { DepthFirstIterator(graph, it).asSequence().toSet() } 4 .flatten()
38. Shrink 5. Remove all unreachable classes 1 override fun transform(invocation: TransformInvocation) { 2 val classNames = getAllClassNames(invocation) 3 val connections = findConnections(classNames) 4 val classGraph = buildGraph(connections) 5 val reachableClasses = findReachableClasses(classGraph) 6 7 // for-each JAR file 8 // remove unreachable classes 9 // repackage JAR file 10 // copy JAR file in output 11 12 // for-each class file 13 // if reachable copy to output
43. ProGuard Steps Shrink Optimize Obfuscate Preverify
44. Optimize • Performs lots of different bytecode-level optimizations to the code • For concrete example, see Jake Wharton’s recent “Sinking Your Teeth Into Bytecode” presentation https://goo.gl/8hnRzP
45. Optimize • Marks methods as final, whenever possible. • Removes unused method parameters. • Propagates the values of method parameters from method invocations to the invoked methods. • Propagates the values of method return values from methods to their invocations. • Inlines short methods. • Inlines methods that are only called once.
46. ProGuard Steps Shrink Optimize Obfuscate Preverify
47. Obfuscate • Classes and class members receive new short random names, except for the ones listed by the various -keep options • Internal attributes that are useful for debugging are removed
48. Obfuscate FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager
49. Obfuscate FeedActivity A B C LoginActivity OldUserManager
50. Obfuscate
51. Resources • Example ProGuard-like transform – Edward Dale
 https://github.com/scompt/WriteYourOwnProGuard • “Sinking Your Teeth Into Bytecode” – Jake Wharton
 http://jakewharton.com/sinking-your-teeth-into-bytecode/ • Gephi – The Open Graph Viz Platform
 https://gephi.org/ • JGraphT
 http://jgrapht.org/
52. Image Citations https://emojipedia.org https://freeletics.com https://gephi.org http://www.cuelogic.com http://knowyourmeme.com https://imgflip.com https://www.flaticon.com https://www.jgrapht.org
53. Questions? Edward Dale @scompt