Slide 1

Slide 1 text

Write your own ProGuard Edward Dale @scompt

Slide 2

Slide 2 text

Write your own ProGuard Edward Dale @scompt R8

Slide 3

Slide 3 text

ProGuard: Purpose ProGuard is the most popular optimizer for Java bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard

Slide 4

Slide 4 text

ProGuard: Purpose ProGuard is the most popular optimizer for Java bytecode. It makes your Java and Android 
 applications up to 90% smaller and up to 
 20% faster. ProGuard also provides minimal 
 protection against reverse engineering by 
 obfuscating the names of classes, fields and 
 methods. https://www.guardsquare.com/en/proguard

Slide 5

Slide 5 text

ProGuard: Highlights • ProGuard is a command-line tool with an optional graphical user interface. • ProGuard is easy to configure. A few intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire run-time libraries in seconds. • ProGuard is the default tool in development environments like Oracle’s Wireless Toolkit, NetBeans, EclipseME, Intel’s TXE SDK and Google’s Android SDK.

Slide 6

Slide 6 text

ProGuard: Highlights • ProGuard is easy to configure. A few intuitive command line options or a simple configuration file is all it takes. All available options are detailed in the user manual. • ProGuard is fast. It processes small Android applications and entire runtime libraries in seconds.

Slide 7

Slide 7 text

ProGuard Steps Shrink Optimize Obfuscate Preverify

Slide 8

Slide 8 text

ProGuard Steps Shrink Optimize Obfuscate Preverify

Slide 9

Slide 9 text

Transform API Build Steps S O O P Compile Assemble Process .java .xml Dex .apk

Slide 10

Slide 10 text

Build Steps Compile Assemble Transform API Process .java .xml Dex .apk

Slide 11

Slide 11 text

Transform API The goal of this API is to simplify injecting custom class manipulations without having to deal with tasks, and to offer more flexibility on what is manipulated. http://tools.android.com/tech-docs/new-build-system/transform-api

Slide 12

Slide 12 text

Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled Bytecode Resources Libraries Other Files Transform

Slide 13

Slide 13 text

Transform API Compiled Bytecode Resources Libraries Other Files } {Compiled Bytecode Resources Libraries Other Files Compiled Bytecode

Slide 14

Slide 14 text

Transform API Examples JacocoTransform https://goo.gl/jNoA2w Instruments code for measuring coverage MultiDexTransform https://goo.gl/MxnuiX Gathers information about the project to configure MultiDex in a later stage ProGuardTransform https://goo.gl/h7CGb4 Runs ProGuard…

Slide 15

Slide 15 text

Transform API 1 class MyPlugin implements Plugin { 2 @Override 3 void apply(Project p) { 4 def t = new WriteYourOwnProGuardTransform(p) 5 p.android.registerTransform(t) 6 } 7 } 1 apply plugin: MyPlugin build.gradle MyPlugin.groovy

Slide 16

Slide 16 text

1 class WriteYourOwnProGuardTransform(val project: Project) 2 : Transform() { 3 4 override fun isIncremental() = false 5 6 override fun getName() = "writeYourOwnProGuard" 7 8 @Throws(TransformException::class, 9 InterruptedException::class, 10 IOException::class) 11 override fun transform(invocation: TransformInvocation) { WriteYourOwnProGuardTransform.kotlin

Slide 17

Slide 17 text

1 override fun getInputTypes() = 2 setOf(QualifiedContent.DefaultContentType.CLASSES, 3 QualifiedContent.DefaultContentType.RESOURCES) 4 5 override fun getScopes() = 6 setOf(QualifiedContent.Scope.PROJECT, 7 QualifiedContent.Scope.SUB_PROJECTS, 8 QualifiedContent.Scope.EXTERNAL_LIBRARIES) WriteYourOwnProGuardTransform.kotlin

Slide 18

Slide 18 text

1 public interface TransformInvocation { 2 Context getContext(); 3 4 Collection getInputs(); 5 6 Collection getReferencedInputs(); 7 8 Collection getSecondaryInputs(); 9 10 TransformOutputProvider getOutputProvider(); 11 12 boolean isIncremental(); 13 } 1 public interface TransformInput { 2 Collection getJarInputs(); 3 4 Collection getDirectoryInputs(); 5 }

Slide 19

Slide 19 text

1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/ 5 3cba64e049f7e363d1d5aa6ed4ca7a3f/ 6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}

Slide 20

Slide 20 text

.jar .class 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0, 2 file=/Users/scompt/.gradle/caches/ 3 transforms-1/files-1.1/ 4 appcompat-v7-26.1.0.aar/ 5 3cba64e049f7e363d1d5aa6ed4ca7a3f/ 6 jars/classes.jar, 7 contentTypes=CLASSES, 8 scopes=EXTERNAL_LIBRARIES, 9 status=NOTCHANGED} 1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH, 2 file=/Users/scompt/projects/ 3 WriteYourOwnProGuard/app/build/ 4 intermediates/classes/debug, 5 contentTypes=CLASSES, 6 scopes=PROJECT, 7 changedFiles={}}

Slide 21

Slide 21 text

Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/ JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/

Slide 22

Slide 22 text

Bytecode Manipulation Apache Commons BCEL (Byte Code Engineering Library) https://commons.apache.org/proper/commons-bcel/ JBoss Javassist http://jboss-javassist.github.io/javassist/ OW2 Consortium ASM http://asm.ow2.org/

Slide 23

Slide 23 text

Transform API Build Steps S O O P Compile Assemble Process .java .xml Dex .apk

Slide 24

Slide 24 text

ProGuard Steps Shrink Optimize Obfuscate Preverify

Slide 25

Slide 25 text

Shrink • Removes all classes, methods, resources not reachable from an entry point (seeds) • Dynamically referenced classes/methods need to be "kept" using -keep or -keepclasseswithmembers

Slide 26

Slide 26 text

Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager

Slide 27

Slide 27 text

Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = []

Slide 28

Slide 28 text

Shrink FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager seeds = [FeedActivity, LoginActivity]

Slide 29

Slide 29 text

Shrink 1. Find all classes in the program 2. Find all connections between classes in the program 3. Find all “seed” classes 4. Find all classes reachable from seeds 5. Remove all unreachable classes

Slide 30

Slide 30 text

Shrink 1. Find all classes in the program 1 for (directoryInput in input.directoryInputs) { 2 val root = directoryInput.file.toPath() 3 Files.walkFileTree(root, /* Accumulate class name */) 1 val classPool = ClassPool.getDefault() 2 for (input in transformInvocation.inputs) { 3 for (jarInput in input.jarInputs) { 4 val stream = Files.newInputStream(jarInput.file.toPath()) 5 JarInputStream(stream).use { 6 var entry: JarEntry? = it.nextJarEntry 7 while (entry != null) { 8 val klass = classPool.makeClass(stream) 9 // Accumulate class name

Slide 31

Slide 31 text

Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames

Slide 32

Slide 32 text

Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames constPool

Slide 33

Slide 33 text

“For each type it loads, a Java virtual machine must store a constant pool. A constant pool is an ordered set of constants used by the type, including literals (string, integer, and floating point constants) and symbolic references to types, fields, and methods. Entries in the constant pool are referenced by index, much like the elements of an array. Because it holds symbolic references to all types, fields, and methods used by a type, the constant pool plays a central role in the dynamic linking of Java programs” –Inside the Java Virtual Machine by Bill Venners

Slide 34

Slide 34 text

Shrink 2. Find all connections between classes in the program 1 val classPool = ClassPool.getDefault() 2 for (pathToClass in classPaths) { 3 FileInputStream(pathToClass.toFile()).use { 4 val klass = classPool.makeClassIfNew(it) 5 val sourceClassName = klass.name 6 val depClassNames = klass.classFile 7 .constPool 8 .classNames

Slide 35

Slide 35 text

Shrink 3. Find all “seed” classes 1 val isSeedClass = klass.name.endsWith(“Activity")

Slide 36

Slide 36 text

Shrink 4. Find all classes reachable from seeds 1 private fun buildGraph(deps: Multimap) 2 : Graph { 3 4 val graph = DefaultDirectedGraph() 5 deps.keySet() 6 .filterNot { it.startsWith("java.lang") } 7 .forEach { graph.addVertex(it) } 8 deps.values() 9 .filterNot { it.startsWith("java.lang") } 10 .forEach { graph.addVertex(it) } 11 deps.entries() 12 .filterNot { it.key.startsWith("java.lang") } 13 .filterNot { it.value.startsWith("java.lang") } 14 .filterNot { it.key == it.value } 15 .forEach { graph.addEdge(it.key, it.value) } 16 return graph 17 }

Slide 37

Slide 37 text

Shrink 4. Find all classes reachable from seeds 1 graph.vertexSet() 2 .filter { it.endsWith("Activity") } 3 .map { DepthFirstIterator(graph, it).asSequence().toSet() } 4 .flatten()

Slide 38

Slide 38 text

Shrink 5. Remove all unreachable classes 1 override fun transform(invocation: TransformInvocation) { 2 val classNames = getAllClassNames(invocation) 3 val connections = findConnections(classNames) 4 val classGraph = buildGraph(connections) 5 val reachableClasses = findReachableClasses(classGraph) 6 7 // for-each JAR file 8 // remove unreachable classes 9 // repackage JAR file 10 // copy JAR file in output 11 12 // for-each class file 13 // if reachable copy to output

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

ProGuard Steps Shrink Optimize Obfuscate Preverify

Slide 44

Slide 44 text

Optimize • Performs lots of different bytecode-level optimizations to the code • For concrete example, see Jake Wharton’s recent “Sinking Your Teeth Into Bytecode” presentation https://goo.gl/8hnRzP

Slide 45

Slide 45 text

Optimize • Marks methods as final, whenever possible. • Removes unused method parameters. • Propagates the values of method parameters from method invocations to the invoked methods. • Propagates the values of method return values from methods to their invocations. • Inlines short methods. • Inlines methods that are only called once.

Slide 46

Slide 46 text

ProGuard Steps Shrink Optimize Obfuscate Preverify

Slide 47

Slide 47 text

Obfuscate • Classes and class members receive new short random names, except for the ones listed by the various -keep options • Internal attributes that are useful for debugging are removed

Slide 48

Slide 48 text

Obfuscate FeedActivity FeedApi UserManager UserApi LoginActivity OldUserManager

Slide 49

Slide 49 text

Obfuscate FeedActivity A B C LoginActivity OldUserManager

Slide 50

Slide 50 text

Obfuscate

Slide 51

Slide 51 text

Resources • Example ProGuard-like transform – Edward Dale
 https://github.com/scompt/WriteYourOwnProGuard • “Sinking Your Teeth Into Bytecode” – Jake Wharton
 http://jakewharton.com/sinking-your-teeth-into-bytecode/ • Gephi – The Open Graph Viz Platform
 https://gephi.org/ • JGraphT
 http://jgrapht.org/

Slide 52

Slide 52 text

Image Citations https://emojipedia.org https://freeletics.com https://gephi.org http://www.cuelogic.com http://knowyourmeme.com https://imgflip.com https://www.flaticon.com https://www.jgrapht.org

Slide 53

Slide 53 text

Questions? Edward Dale @scompt