Write your own ProGuard

ProGuard: We all use it, but how much do we really know about it? Obviously, it's something that Google recommends that we use. And somewhere you read that it makes the code smaller. But what does it really do?

In this presentation, we'll answer that question together by developing a replacement for it. Like the real ProGuard, it will be able to shrink, optimize, and obfuscate code. The difference from ProGuard will be that we understand every step.

Transcript

1. Write your own
   ProGuard
   Edward Dale
   
   @scompt
   

   View Slide

2. Write your own
   ProGuard
   Edward Dale
   
   @scompt
   R8
   

   View Slide

3. ProGuard: Purpose
   ProGuard is the most popular optimizer for Java bytecode.
   It makes your Java and Android 

   applications up to 90% smaller and up to 

   20% faster. ProGuard also provides minimal 

   protection against reverse engineering by 

   obfuscating the names of classes, fields and 

   methods.
   
   https://www.guardsquare.com/en/proguard
   

   View Slide

4. ProGuard: Purpose
   ProGuard is the most popular optimizer for Java bytecode.
   It makes your Java and Android 

   applications up to 90% smaller and up to 

   20% faster. ProGuard also provides minimal 

   protection against reverse engineering by 

   obfuscating the names of classes, fields and 

   methods.
   
   https://www.guardsquare.com/en/proguard
   

   View Slide

5. ProGuard: Highlights
   • ProGuard is a command-line tool with an optional graphical
   user interface.
   
   • ProGuard is easy to configure. A few intuitive command line
   options or a simple configuration file is all it takes. All
   available options are detailed in the user manual.
   
   • ProGuard is fast. It processes small Android applications
   and entire run-time libraries in seconds.
   
   • ProGuard is the default tool in development environments
   like Oracle’s Wireless Toolkit, NetBeans, EclipseME, Intel’s
   TXE SDK and Google’s Android SDK.
   

   View Slide

6. ProGuard: Highlights
   • ProGuard is easy to
   configure. A few intuitive
   command line options or a
   simple configuration file is all
   it takes. All available options
   are detailed in the user
   manual.
   
   • ProGuard is fast. It processes
   small Android applications and
   entire runtime libraries in
   seconds.
   

   View Slide

7. ProGuard Steps
   Shrink Optimize Obfuscate Preverify
   

   View Slide

8. ProGuard Steps
   Shrink Optimize Obfuscate Preverify
   

   View Slide

9. Transform API
   Build Steps
   S O O P
   Compile
   Assemble
   Process
   .java
   .xml
   Dex
   .apk
   

   View Slide

10. Build Steps
    Compile
    Assemble
    Transform API
    Process
    .java
    .xml
    Dex
    .apk
    

    View Slide

11. Transform API
    The goal of this API is to simplify injecting custom class
    manipulations without having to deal with tasks, and to
    offer more flexibility on what is manipulated.
    http://tools.android.com/tech-docs/new-build-system/transform-api
    

    View Slide

12. Transform API
    Compiled Bytecode
    Resources
    Libraries
    Other Files
    } {Compiled Bytecode
    Resources
    Libraries
    Other Files
    Transform
    

    View Slide

13. Transform API
    Compiled Bytecode
    Resources
    Libraries
    Other Files
    } {Compiled Bytecode
    Resources
    Libraries
    Other Files
    Compiled Bytecode
    

    View Slide

14. Transform API Examples
    JacocoTransform
    https://goo.gl/jNoA2w
    Instruments code for measuring coverage
    MultiDexTransform
    https://goo.gl/MxnuiX
    Gathers information about the project to configure MultiDex in a later stage
    ProGuardTransform
    https://goo.gl/h7CGb4
    Runs ProGuard…
    

    View Slide

15. Transform API
    1 class MyPlugin implements Plugin {
    2 @Override
    3 void apply(Project p) {
    4 def t = new WriteYourOwnProGuardTransform(p)
    5 p.android.registerTransform(t)
    6 }
    7 }
    1 apply plugin: MyPlugin
    build.gradle
    MyPlugin.groovy
    

    View Slide

16. 1 class WriteYourOwnProGuardTransform(val project: Project)
    2 : Transform() {
    3
    4 override fun isIncremental() = false
    5
    6 override fun getName() = "writeYourOwnProGuard"
    7
    8 @Throws(TransformException::class,
    9 InterruptedException::class,
    10 IOException::class)
    11 override fun transform(invocation: TransformInvocation) {
    WriteYourOwnProGuardTransform.kotlin
    

    View Slide

17. 1 override fun getInputTypes() =
    2 setOf(QualifiedContent.DefaultContentType.CLASSES,
    3 QualifiedContent.DefaultContentType.RESOURCES)
    4
    5 override fun getScopes() =
    6 setOf(QualifiedContent.Scope.PROJECT,
    7 QualifiedContent.Scope.SUB_PROJECTS,
    8 QualifiedContent.Scope.EXTERNAL_LIBRARIES)
    WriteYourOwnProGuardTransform.kotlin
    

    View Slide

18. 1 public interface TransformInvocation {
    2 Context getContext();
    3
    4 Collection getInputs();
    5
    6 Collection getReferencedInputs();
    7
    8 Collection getSecondaryInputs();
    9
    10 TransformOutputProvider getOutputProvider();
    11
    12 boolean isIncremental();
    13 }
    1 public interface TransformInput {
    2 Collection getJarInputs();
    3
    4 Collection getDirectoryInputs();
    5 }
    

    View Slide

19. 1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0,
    2 file=/Users/scompt/.gradle/caches/
    3 transforms-1/files-1.1/
    4 appcompat-v7-26.1.0.aar/
    5 3cba64e049f7e363d1d5aa6ed4ca7a3f/
    6 jars/classes.jar,
    7 contentTypes=CLASSES,
    8 scopes=EXTERNAL_LIBRARIES,
    9 status=NOTCHANGED}
    1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH,
    2 file=/Users/scompt/projects/
    3 WriteYourOwnProGuard/app/build/
    4 intermediates/classes/debug,
    5 contentTypes=CLASSES,
    6 scopes=PROJECT,
    7 changedFiles={}}
    

    View Slide

20. .jar .class
    1 ImmutableJarInput{name=com.android.support:appcompat-v7:26.1.0,
    2 file=/Users/scompt/.gradle/caches/
    3 transforms-1/files-1.1/
    4 appcompat-v7-26.1.0.aar/
    5 3cba64e049f7e363d1d5aa6ed4ca7a3f/
    6 jars/classes.jar,
    7 contentTypes=CLASSES,
    8 scopes=EXTERNAL_LIBRARIES,
    9 status=NOTCHANGED}
    1 ImmutableDirectoryInput{name=aa2bd2bBIGHASH,
    2 file=/Users/scompt/projects/
    3 WriteYourOwnProGuard/app/build/
    4 intermediates/classes/debug,
    5 contentTypes=CLASSES,
    6 scopes=PROJECT,
    7 changedFiles={}}
    

    View Slide

21. Bytecode Manipulation
    Apache Commons BCEL (Byte Code Engineering Library)
    https://commons.apache.org/proper/commons-bcel/
    JBoss Javassist
    http://jboss-javassist.github.io/javassist/
    OW2 Consortium ASM
    http://asm.ow2.org/
    

    View Slide

22. Bytecode Manipulation
    Apache Commons BCEL (Byte Code Engineering Library)
    https://commons.apache.org/proper/commons-bcel/
    JBoss Javassist
    http://jboss-javassist.github.io/javassist/
    OW2 Consortium ASM
    http://asm.ow2.org/
    

    View Slide

23. Transform API
    Build Steps
    S O O P
    Compile
    Assemble
    Process
    .java
    .xml
    Dex
    .apk
    

    View Slide

24. ProGuard Steps
    Shrink Optimize Obfuscate Preverify
    

    View Slide

25. Shrink
    • Removes all classes, methods, resources not reachable from an
    entry point (seeds)
    
    • Dynamically referenced classes/methods need to be "kept" using
    -keep or -keepclasseswithmembers
    

    View Slide

26. Shrink
    FeedActivity
    FeedApi
    UserManager
    UserApi
    LoginActivity
    OldUserManager
    

    View Slide

27. Shrink
    FeedActivity
    FeedApi
    UserManager
    UserApi
    LoginActivity
    OldUserManager
    seeds = []
    

    View Slide

28. Shrink
    FeedActivity
    FeedApi
    UserManager
    UserApi
    LoginActivity
    OldUserManager
    seeds = [FeedActivity, LoginActivity]
    

    View Slide

29. Shrink
    1. Find all classes in the program
    
    2. Find all connections between classes in the program
    
    3. Find all “seed” classes
    
    4. Find all classes reachable from seeds
    
    5. Remove all unreachable classes
    

    View Slide

30. Shrink 1. Find all classes in the program
    1 for (directoryInput in input.directoryInputs) {
    2 val root = directoryInput.file.toPath()
    3 Files.walkFileTree(root, /* Accumulate class name */)
    1 val classPool = ClassPool.getDefault()
    2 for (input in transformInvocation.inputs) {
    3 for (jarInput in input.jarInputs) {
    4 val stream = Files.newInputStream(jarInput.file.toPath())
    5 JarInputStream(stream).use {
    6 var entry: JarEntry? = it.nextJarEntry
    7 while (entry != null) {
    8 val klass = classPool.makeClass(stream)
    9 // Accumulate class name
    

    View Slide

31. Shrink 2. Find all connections between classes in the program
    1 val classPool = ClassPool.getDefault()
    2 for (pathToClass in classPaths) {
    3 FileInputStream(pathToClass.toFile()).use {
    4 val klass = classPool.makeClassIfNew(it)
    5 val sourceClassName = klass.name
    6 val depClassNames = klass.classFile
    7 .constPool
    8 .classNames
    

    View Slide

32. Shrink 2. Find all connections between classes in the program
    1 val classPool = ClassPool.getDefault()
    2 for (pathToClass in classPaths) {
    3 FileInputStream(pathToClass.toFile()).use {
    4 val klass = classPool.makeClassIfNew(it)
    5 val sourceClassName = klass.name
    6 val depClassNames = klass.classFile
    7 .constPool
    8 .classNames
    constPool
    

    View Slide

33. “For each type it loads, a Java virtual machine must
    store a constant pool. A constant pool is an ordered
    set of constants used by the type, including literals
    (string, integer, and floating point constants) and
    symbolic references to types, fields, and methods.
    Entries in the constant pool are referenced by index,
    much like the elements of an array. Because it holds
    symbolic references to all types, fields, and methods
    used by a type, the constant pool plays a central role
    in the dynamic linking of Java programs”
    –Inside the Java Virtual Machine by Bill Venners
    

    View Slide

34. Shrink 2. Find all connections between classes in the program
    1 val classPool = ClassPool.getDefault()
    2 for (pathToClass in classPaths) {
    3 FileInputStream(pathToClass.toFile()).use {
    4 val klass = classPool.makeClassIfNew(it)
    5 val sourceClassName = klass.name
    6 val depClassNames = klass.classFile
    7 .constPool
    8 .classNames
    

    View Slide

35. Shrink 3. Find all “seed” classes
    1 val isSeedClass = klass.name.endsWith(“Activity")
    

    View Slide

36. Shrink 4. Find all classes reachable from seeds
    1 private fun buildGraph(deps: Multimap)
    2 : Graph {
    3
    4 val graph = DefaultDirectedGraph()
    5 deps.keySet()
    6 .filterNot { it.startsWith("java.lang") }
    7 .forEach { graph.addVertex(it) }
    8 deps.values()
    9 .filterNot { it.startsWith("java.lang") }
    10 .forEach { graph.addVertex(it) }
    11 deps.entries()
    12 .filterNot { it.key.startsWith("java.lang") }
    13 .filterNot { it.value.startsWith("java.lang") }
    14 .filterNot { it.key == it.value }
    15 .forEach { graph.addEdge(it.key, it.value) }
    16 return graph
    17 }
    

    View Slide

37. Shrink 4. Find all classes reachable from seeds
    1 graph.vertexSet()
    2 .filter { it.endsWith("Activity") }
    3 .map { DepthFirstIterator(graph, it).asSequence().toSet() }
    4 .flatten()
    

    View Slide

38. Shrink 5. Remove all unreachable classes
    1 override fun transform(invocation: TransformInvocation) {
    2 val classNames = getAllClassNames(invocation)
    3 val connections = findConnections(classNames)
    4 val classGraph = buildGraph(connections)
    5 val reachableClasses = findReachableClasses(classGraph)
    6
    7 // for-each JAR file
    8 // remove unreachable classes
    9 // repackage JAR file
    10 // copy JAR file in output
    11
    12 // for-each class file
    13 // if reachable copy to output
    

    View Slide

39. View Slide

40. View Slide

41. View Slide

42. View Slide

43. ProGuard Steps
    Shrink Optimize Obfuscate Preverify
    

    View Slide

44. Optimize
    • Performs lots of different bytecode-level optimizations to the code
    
    • For concrete example, see Jake Wharton’s recent “Sinking Your
    Teeth Into Bytecode” presentation https://goo.gl/8hnRzP
    

    View Slide

45. Optimize
    • Marks methods as final, whenever possible.
    
    • Removes unused method parameters.
    
    • Propagates the values of method parameters from method
    invocations to the invoked methods.
    
    • Propagates the values of method return values from methods to
    their invocations.
    
    • Inlines short methods.
    
    • Inlines methods that are only called once.
    

    View Slide

46. ProGuard Steps
    Shrink Optimize Obfuscate Preverify
    

    View Slide

47. Obfuscate
    • Classes and class members receive new short random names,
    except for the ones listed by the various -keep options
    
    • Internal attributes that are useful for debugging are removed
    

    View Slide

48. Obfuscate
    FeedActivity
    FeedApi
    UserManager
    UserApi
    LoginActivity
    OldUserManager
    

    View Slide

49. Obfuscate
    FeedActivity
    A
    B
    C
    LoginActivity
    OldUserManager
    

    View Slide

50. Obfuscate
    

    View Slide

51. Resources
    • Example ProGuard-like transform – Edward Dale

    https://github.com/scompt/WriteYourOwnProGuard
    
    • “Sinking Your Teeth Into Bytecode” – Jake Wharton

    http://jakewharton.com/sinking-your-teeth-into-bytecode/
    
    • Gephi – The Open Graph Viz Platform

    https://gephi.org/
    
    • JGraphT

    http://jgrapht.org/
    

    View Slide

52. Image Citations
    https://emojipedia.org
    https://freeletics.com
    https://gephi.org
    http://www.cuelogic.com
    http://knowyourmeme.com https://imgflip.com
    https://www.flaticon.com
    https://www.jgrapht.org
    

    View Slide

53. Questions?
    Edward Dale
    
    @scompt
    

    View Slide