Slide 1

Slide 1 text

(Android) Transformers - bytecode in disguise! Doug Stevenson @CodingDoug

Slide 2

Slide 2 text

@CodingDoug

Slide 3

Slide 3 text

@CodingDoug

Slide 4

Slide 4 text

@CodingDoug

Slide 5

Slide 5 text

@CodingDoug

Slide 6

Slide 6 text

@CodingDoug

Slide 7

Slide 7 text

@CodingDoug

Slide 8

Slide 8 text

@CodingDoug How can you build a no-code HTTP data collector SDK? ● Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache) ● No API hooks to collect HTTP transaction data ● Options: ○ Modify app source code? ○ JVM “agent”? ○ Modify bytecode of compiled JVM classes, before deployment

Slide 9

Slide 9 text

@CodingDoug How can you build a no-code HTTP data collector SDK? ● Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache) ● No API hooks to collect HTTP transaction data ● Options: ○ Modify app source code? (infeasible) ○ JVM “agent”? (impossible on Android) ○ Modify bytecode of compiled JVM classes, before deployment

Slide 10

Slide 10 text

@CodingDoug val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close() Task: instrument bytecode that compiled from source like this:

Slide 11

Slide 11 text

@CodingDoug Java/Kotlin Compiler classes / jars DX / D8 dex file(s) APK aapt2

Slide 12

Slide 12 text

@CodingDoug classes / jars DX / D8 But where exactly are the classes? And how exactly do I hook the build process? bytecode stuff goes here

Slide 13

Slide 13 text

@CodingDoug DX / D8

Slide 14

Slide 14 text

@CodingDoug DX / D8 private val inputJars: List = [ /path/to/jar ] private val inputClasses: List = [ /path/to/classes ] Read private config at runtime (reflection)

Slide 15

Slide 15 text

@CodingDoug DX / D8 private val inputJars: List = [ /path/to/jar ] [ /my/modified/jar ] private val inputClasses: List = [ /path/to/classes ] [ /my/modified/classes ] Don’t ever do this. Messing with private members via reflection is wrong! And inject my own changes Read private config at runtime (reflection)

Slide 16

Slide 16 text

@CodingDoug DX / D8 private val inputJars: List = [ /path/to/jar ] [ /my/modified/jar ] private val inputClasses: List = [ /path/to/classes ] [ /my/modified/classes ]

Slide 17

Slide 17 text

@CodingDoug classes / jars DX / D8 Proguard Transform Your Transform Another Transform

Slide 18

Slide 18 text

@CodingDoug Gradle plugin Uses Transform API Does bytecode stuff Companion SDK Runtime support Sample app Modified by transform Open sourced sample project components

Slide 19

Slide 19 text

@CodingDoug project-root transformer-app transformer-libs transformer-module transformer-plugin

Slide 20

Slide 20 text

The Transform Gradle plugin

Slide 21

Slide 21 text

@CodingDoug Transform Gradle plugin - build.gradle apply plugin: 'java-gradle-plugin' apply plugin: 'kotlin' apply plugin: 'maven' repositories { google() jcenter() } dependencies { compileOnly gradleApi() compileOnly 'com.android.tools.build:gradle:3.4.2' implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.3.41' implementation 'commons-io:commons-io:2.6' implementation ‘org.ow2.asm:asm:7.1' }

Slide 22

Slide 22 text

@CodingDoug implementation-class=com.hyperaware.transformer.plugin.MyPlugin Define the plugin entry point in META-INF with a properties file

Slide 23

Slide 23 text

@CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }

Slide 24

Slide 24 text

@CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }

Slide 25

Slide 25 text

@CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }

Slide 26

Slide 26 text

@CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set { return typeClasses } // more ... }

Slide 27

Slide 27 text

@CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set { return typeClasses } // more ... }

Slide 28

Slide 28 text

@CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set { return typeClasses } // more ... }

Slide 29

Slide 29 text

@CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform is interested in classes from all parts of the app private val scopes = setOf( QualifiedContent.Scope.PROJECT, QualifiedContent.Scope.SUB_PROJECTS, QualifiedContent.Scope.EXTERNAL_LIBRARIES ) override fun getScopes(): MutableSet { return scopes.toMutableSet() } // more ... }

Slide 30

Slide 30 text

@CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform is interested in classes from all parts of the app private val scopes = setOf( QualifiedContent.Scope.PROJECT, QualifiedContent.Scope.SUB_PROJECTS, QualifiedContent.Scope.EXTERNAL_LIBRARIES ) override fun getScopes(): MutableSet { return scopes.toMutableSet() } // more ... }

Slide 31

Slide 31 text

@CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform can handle incremental builds override fun isIncremental(): Boolean { return true } // more ... }

Slide 32

Slide 32 text

@CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... override fun transform(transformInvocation: TransformInvocation) { // Find the Gradle extension that contains configuration for this Transform val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension() val appExtension = project.extensions.findByName("android") as AppExtension val ignores = listOf( Regex("com/hyperaware/transformer/.*") ) val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores, ext) MyTransformImpl(config).doIt() } }

Slide 33

Slide 33 text

@CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... override fun transform(transformInvocation: TransformInvocation) { // Find the Gradle extension that contains configuration for this Transform val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension() val appExtension = project.extensions.findByName("android") as AppExtension val ignores = listOf( Regex("com/hyperaware/transformer/.*") ) val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores, ext) MyTransformImpl(config).doIt() } }

Slide 34

Slide 34 text

@CodingDoug TransformInvocation interface - defines inputs and ouputs public interface TransformInvocation { @NonNull Context getContext(); @NonNull Collection getInputs(); @NonNull Collection getReferencedInputs(); @NonNull Collection getSecondaryInputs(); @Nullable TransformOutputProvider getOutputProvider(); boolean isIncremental(); }

Slide 35

Slide 35 text

@CodingDoug Transformer Gradle plugin - MyTransformImpl class MyTransformImpl(config: TransformConfig) { private val logger = Logging.getLogger(MyTransformImpl::class.java) private val transformInvocation = config.transformInvocation private val androidClasspath = config.androidClasspath private val ignorePaths = config.ignorePaths private val outputProvider = transformInvocation.outputProvider private val instrumentationConfig = InstrumentationConfig( buildRuntimeClasspath(transformInvocation), config.pluginConfig.logVisits, config.pluginConfig.logInstrumentation ) private val instrumenter = ClassInstrumenter(instrumentationConfig) fun doIt() { logger.debug(instrumentationConfig.toString()) logger.debug("isIncremental: ${transformInvocation.isIncremental}") for (ti in transformInvocation.inputs) { instrumentDirectoryInputs(ti.directoryInputs) instrumentJarInputs(ti.jarInputs) } } /** * Builds the runtime classpath of the project. This combines all the * various TransformInput file locations in addition to the targeted * Android platform jar into a single collection that's suitable to be a * classpath for the entire app. */ private fun buildRuntimeClasspath(transformInvocation: TransformInvocation): List { val allTransformInputs = transformInvocation.inputs + transformInvocation.referencedInputs val allJarsAndDirs = allTransformInputs.map { ti -> (ti.directoryInputs + ti.jarInputs).map { i -> i.file } } val allClassesAtRuntime = androidClasspath + allJarsAndDirs.flatten() return allClassesAtRuntime.map { file -> file.toURI().toURL() } } private fun instrumentDirectoryInputs(directoryInputs: Collection) { // A DirectoryInput is a tree of class files that simply gets // copied to the output directory. // for (di in directoryInputs) { // Build a unique name for the output dir based on the path // of the input dir. // logger.debug("TransformInput dir $di") val outDir = outputProvider.getContentLocation(di.name, di.contentTypes, di.scopes, Format.DIRECTORY) logger.debug(" Directory input ${di.file}") logger.debug(" Directory output $outDir") if (transformInvocation.isIncremental) { // Incremental builds will specify which individual class files changed. for (changedFile in di.changedFiles) { when (changedFile.value) { private fun instrumentJarInputs(jarInputs: Collection) { // A JarInput is a jar file that just gets copied to a destination // output jar. // for (ji in jarInputs) { // Build a unique name for the output file based on the path // of the input jar. // logger.debug("TransformInput jar $ji") val outDir = outputProvider.getContentLocation(ji.name, ji.contentTypes, ji.scopes, Format.DIRECTORY) logger.debug(" Jar input ${ji.file}") logger.debug(" Dir output $outDir") val doTransform = !transformInvocation.isIncremental || ji.status == Status.ADDED || ji.status == Status.CHANGED if (doTransform) { ensureDirectoryExists(outDir) FileUtils.cleanDirectory(outDir) val inJar = JarFile(ji.file) var count = 0 for (entry in inJar.entries()) { val outFile = File(outDir, entry.name) if (!entry.isDirectory) { ensureDirectoryExists(outFile.parentFile) inJar.getInputStream(entry).use { inputStream -> IOUtils.buffer(FileOutputStream(outFile)).use { outputStream -> if (isInstrumentableClassFile(entry.name)) { try { processClassStream(entry.name, inputStream, outputStream) } catch (e: Exception) { logger.error("Can't process class ${entry.name}", e) throw e } } else { copyStream(inputStream, outputStream) } } } count++ } } logger.debug(" Entries copied: $count") } else if (ji.status == Status.REMOVED) { logger.debug(" REMOVED") if (outDir.exists()) { FileUtils.forceDelete(outDir) } } } } Status.ADDED, Status.CHANGED -> { val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key) val destFile = File(outDir, relativeFile) changedFile.key.inputStream().use { inputStream -> destFile.outputStream().use { outputStream -> if (isInstrumentableClassFile(relativeFile)) { processClassStream(relativeFile, inputStream, outputStream) } else { copyStream(inputStream, outputStream) } } } } Status.REMOVED -> { val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key) val destFile = File(outDir, relativeFile) FileUtils.forceDelete(destFile) } Status.NOTCHANGED, null -> { } } } logger.debug(" Files processed: ${di.changedFiles.size}") } else { ensureDirectoryExists(outDir) FileUtils.cleanDirectory(outDir) logger.debug(" Copying ${di.file} to $outDir") var count = 0 for (file in FileUtils.iterateFiles(di.file, null, true)) { val relativeFile = normalizedRelativeFilePath(di.file, file) val destFile = File(outDir, relativeFile) ensureDirectoryExists(destFile.parentFile) IOUtils.buffer(file.inputStream()).use { inputStream -> IOUtils.buffer(destFile.outputStream()).use { outputStream -> if (isInstrumentableClassFile(relativeFile)) { try { processClassStream(relativeFile, inputStream, outputStream) } catch (e: Exception) { logger.error("Can't process class $file", e) throw e } } else { copyStream(inputStream, outputStream) } } } count++ } logger.debug(" Files processed: $count") } } } private fun ensureDirectoryExists(dir: File) { if (! ((dir.isDirectory && dir.canWrite()) || dir.mkdirs())) { throw IOException("Can't write or create ${dir.path}") } } // Builds a relative path from a given file and its parent. // For file /a/b/c and parent /a, returns "b/c". private fun normalizedRelativeFilePath(parent: File, file: File): String { val parts = mutableListOf() var current = file while (current != parent) { parts.add(current.name) current = current.parentFile } return parts.asReversed().joinToString("/") } // Checks the (relative) path of a given class file and returns true if // it's assumed to be instrumentable. The path must end with .class and // also not match any of the regular expressions in ignorePaths. private fun isInstrumentableClassFile(path: String): Boolean { return if (ignorePaths.any { it.matches(path) }) { logger.debug("Ignoring class $path") false } else { path.toLowerCase().endsWith(".class") } } private fun copyStream(inputStream: InputStream, outputStream: OutputStream) { IOUtils.copy(inputStream, outputStream) } private fun processClassStream(name: String, inputStream: InputStream, outputStream: OutputStream) { val classBytes = IOUtils.toByteArray(inputStream) val bytesToWrite = try { val instrBytes = instrumenter.instrument(classBytes) instrBytes } catch (e: Exception) { // If instrumentation fails, just write the original bytes logger.error("Failed to instrument $name, using original contents", e) classBytes } IOUtils.write(bytesToWrite, outputStream) } }

Slide 36

Slide 36 text

@CodingDoug TransformInvocation processing pseudocode foreach TransformInput object “ti” foreach input in “ti” (jars and directories) if input requires processing output = ti.outputProvider.getContentLocation(...) foreach file in the jar or directory if it’s a class file read the bytecode // honestly, the only interesting part use ASM to make bytecode changes as needed write bytecode to output else copy the file to output

Slide 37

Slide 37 text

@CodingDoug ASM asm.ow2.io

Slide 38

Slide 38 text

@CodingDoug ASM uses the visitor pattern to deliver class contents

Slide 39

Slide 39 text

@CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents

Slide 40

Slide 40 text

@CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents

Slide 41

Slide 41 text

@CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents

Slide 42

Slide 42 text

@CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor MethodVisitor - visitParameter(name) - visitAnnotation(desc): AnnotationVisitor - visitCode() - visitLocalVariable(name) - visitMethodInsn(opcode) - visitEnd() ASM uses the visitor pattern to deliver class contents

Slide 43

Slide 43 text

Let’s visit some bytecode!

Slide 44

Slide 44 text

@CodingDoug But first, a little bit about the JVM Constant Pool ● One per-class ● Pre-computed values, references, types ● Indexed by position #5 = Utf8 onCreate #6 = Utf8 (Landroid/os/Bundle;)V #7 = Utf8 Lorg/jetbrains/annotations/Nullable; #8 = NameAndType #5:#6 // onCreate:(Landroid/os/Bundle;)V #9 = Methodref #4.#8 // android/support/v7/app/AppCompatActivi #10 = Integer 2131296284 #11 = Utf8 setContentView #12 = Utf8 (I)V #13 = NameAndType #11:#12 // setContentView:(I)V

Slide 45

Slide 45 text

@CodingDoug But first, a little bit about the JVM Local variables ● Similar to registers for a typical CPU ● Indexed by position,
 0 = “this” for instance methods ● Typically map 1:1 with Java/Kotlin locals LocalVariableTable: Start Length Slot Name Signature 38 13 4 input Ljava/io/InputStream; 26 25 3 conn Ljava/net/URLConnection; 15 36 2 array [B 10 41 1 url Ljava/net/URL; 0 51 0 this Lcom/hyperaware/transformers/MyRunnable;

Slide 46

Slide 46 text

@CodingDoug But first, a little bit about the JVM Operand Stack ● Push values onto it (constants, locals, fields) ● Execute operations against pushed values (e.g. math, call method),
 those values are popped, result of operation is pushed ● Similar to Reverse Polish Notation (aka postfix notation) // KOTLIN: val c = a + b 0: iload_1 // push local int variable 1 “a” 1: iload_2 // push local int variable 2 “b” 2: iadd // add them, pop them, push the result 3: istore_3 // store the result in local variable 3 “c”, pop it

Slide 47

Slide 47 text

@CodingDoug But first, a little bit about the JVM Stack Frame ● Created on the current thread’s stack for each method call ● Contains local variables & operand stack
 (max sizes are pre-determined) ● Popped off the stack when the method returns

Slide 48

Slide 48 text

@CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }

Slide 49

Slide 49 text

@CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }

Slide 50

Slide 50 text

@CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }

Slide 51

Slide 51 text

@CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }

Slide 52

Slide 52 text

@CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }

Slide 53

Slide 53 text

@CodingDoug class bytes ASM ClassReader InstrumentationVisitor
 (a ClassVisitor) ASM ClassWriter
 (a ClassVisitor) modified
 class bytes invokes visitor methods invokes visitor methods decorates

Slide 54

Slide 54 text

@CodingDoug InstrumentationVisitor skeleton class InstrumentationVisitor( classVisitor ClassVisitor, // actually a ClassWriter private val config: InstrumentationConfig ) : ClassVisitor(ASM_API_VERSION, classVisitor) { }

Slide 55

Slide 55 text

@CodingDoug InstrumentationVisitor skeleton class InstrumentationVisitor( classVisitor ClassVisitor, // actually a ClassWriter private val config: InstrumentationConfig ) : ClassVisitor(ASM_API_VERSION, classVisitor) { override fun visit( version: Int, access: Int, className: String, signature: String?, superName: String, interfaces: Array? ) override fun visitMethod( access: Int, methodName: String, methodDesc: String, signature: String?, exceptions: Array? ): MethodVisitor }

Slide 56

Slide 56 text

@CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array? // declared exceptions thrown ): MethodVisitor { }

Slide 57

Slide 57 text

@CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array? // declared exceptions thrown ): MethodVisitor { // Get a MethodVisitor using the ClassVisitor we’re decorating val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions) }

Slide 58

Slide 58 text

@CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array? // declared exceptions thrown ): MethodVisitor { // Get a MethodVisitor using the ClassVisitor we’re decorating val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions) // Wrap it in a custom MethodVisitor subclass return MyMethodVisitor(ASM_API_VERSION, mv, access, methodName, methodDesc) }

Slide 59

Slide 59 text

@CodingDoug class MyMethodVisitor private inner class MyMethodVisitor( api: Int, // class API version (ASM constant) mv: MethodVisitor, // original decorated MethodVisitor access: Int, // access flags (e.g. public, final...) methodName: String, // name of the declared method methodDesc: String // signature of the declared method ): AdviceAdapter(api, mv, access, methodName, methodDesc) { }

Slide 60

Slide 60 text

@CodingDoug class MyMethodVisitor private inner class MyMethodVisitor( api: Int, // class API version (ASM constant) mv: MethodVisitor, // original decorated MethodVisitor access: Int, // access flags (e.g. public, final...) methodName: String, // name of the declared method methodDesc: String // signature of the declared method ): AdviceAdapter(api, mv, access, methodName, methodDesc) { override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) // is this from an interface? }

Slide 61

Slide 61 text

@CodingDoug So, what are we trying to do, again? Automatically measure all HTTP requests with code that looks like this: fun makeRequest() { val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close() }

Slide 62

Slide 62 text

@CodingDoug javap -c verbose SomeClass.class (1) public final void makeRequest(); descriptor: ()V flags: ACC_PUBLIC, ACC_FINAL Code: stack=3, locals=5, args_size=1 LocalVariableTable: Start Length Slot Name Signature 38 13 4 input Ljava/io/InputStream; 26 25 3 conn Ljava/net/URLConnection; 15 36 2 array [B 10 41 1 url Ljava/net/URL; 0 51 0 this Lcom/hyperaware/transformers/MyRunnable2; val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 63

Slide 63 text

@CodingDoug javap -c verbose SomeClass.class (2) // KOTLIN: val url = URL("https://www.google.com") 0: new #10 // class java/net/URL 3: dup 4: ldc #12 // String https://www.google.com 6: invokespecial #16 // Method java/net/URL."":(Ljava/lang/String;)V 9: astore_1 // KOTLIN: val array = ByteArray(15) 10: bipush 15 12: newarray byte 14: astore_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 64

Slide 64 text

@CodingDoug javap -c verbose SomeClass.class (3) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”) 19: dup 20: ldc #67 // String url.openConnection() 22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 25: astore_3 // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”) 30: dup 31: ldc #75 // String conn.getInputStream() 33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 36: astore 4 38: aload 4 40: aload_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 65

Slide 65 text

@CodingDoug javap -c verbose SomeClass.class (4) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”) 19: dup 20: ldc #67 // String url.openConnection() 22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 25: astore_3 // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”) 30: dup 31: ldc #75 // String conn.getInputStream() 33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 36: astore 4 38: aload 4 40: aload_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 66

Slide 66 text

@CodingDoug javap -c verbose SomeClass.class (5) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 // URL is now pushed on the stack 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // URL is popped, and URLConnection is now pushed 25: astore_3 // URLConnection is stored in local var 3 and popped // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; 36: astore 4 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 67

Slide 67 text

@CodingDoug Before: val conn = url.openConnection() After: val conn = URLConnectionInstrumentation.openConnection(url) Given: 1) this static method is available at runtime, and 2) InstrumentedHttpsURLConnection subclasses HttpsURLConnection: @JvmStatic fun openConnection(url: URL): HttpsURLConnection { return InstrumentedHttpsURLConnection(url, url.openConnection()) }

Slide 68

Slide 68 text

@CodingDoug More generally, before: val res: R = obj.fn(a, b, c) After: val res: R = UtilityClass.fn(obj, a, b, c) Provided: class UtilityClass { companion object { @JvmStatic fun fn(obj, a, b, c): R { return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R } } }

Slide 69

Slide 69 text

@CodingDoug More generally, before: val res: R = obj.fn(a, b, c) After: val res: R = UtilityClass.fn(obj, a, b, c) Provided: class UtilityClass { companion object { @JvmStatic fun fn(obj, a, b, c): R { return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R } } }

Slide 70

Slide 70 text

@CodingDoug Bytecode change - one weird trick Kotlin: val res: R = obj.fn(a, b, c) Bytecode: invokevirtual #x // #x -> pkg/of/obj.fn() Kotlin: val res: R = UtilityClass.fn(obj, a, b, c) Bytecode: invokestatic #y // #y -> pkg/of/UtilityClass.fn() 3 c 2 b 1 a 0 obj before 0 res after 1. Pop 4 items off the stack, obj and args a, b, c 2. Make a new frame with obj, a, b, c as locals 3. Call fn instance method on obj 4. Push returned res object on the stack 1. Pop the top 4 items off the stack, obj and args a, b, c 2. Make a new frame with obj, a, b, c as locals 3. Call the UtilityClass.fn static method 4. Push returned res object on the stack

Slide 71

Slide 71 text

@CodingDoug Only one change to bytecode is necessary here: replace invokevirtual #x with invokestatic #y (while the companion library static method does the real work)

Slide 72

Slide 72 text

@CodingDoug One weird trick, applied // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 // URL is now pushed on the stack 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; 16: invokestatic #41 // Method UrlConnectionInstrumentation.openConnection:(Ljava/net/ URL;)Ljava/net/URLConnection; // URL is popped, and URLConnection is now pushed 25: astore_3 // URLConnection is stored in local var 3 and popped // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; 36: astore 4 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()

Slide 73

Slide 73 text

How do you do it? Let’s revisit the visitor

Slide 74

Slide 74 text

@CodingDoug Revisit the ASM visitor private inner class MyMethodVisitor( api: Int, mv: MethodVisitor, access: Int, methodName: String, methodDesc: String ): AdviceAdapter(api, mv, access, methodName, methodDesc) { override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) // is this from an interface? }

Slide 75

Slide 75 text

@CodingDoug Implement that weird trick with ASM override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) { // is this from an interface? if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") { // This is the method call we’re looking for! Make the switch } else { // Don’t touch a thing } }

Slide 76

Slide 76 text

@CodingDoug Implement that weird trick with ASM override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) { // is this from an interface? if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") { // This is the method call we’re looking for! Make the switch super.visitMethodInsn( Opcodes.INVOKESTATIC, // static method call "com/company/utility/UrlConnectionInstrumentation", // utility class “openConnection", // static method "(Ljava/net/URL;)Ljava/net/URLConnection;", // method signature false) // is this from an interface? } else { // Don’t touch a thing super.visitMethodInsn(opcode, owner, name, desc, itf) } }

Slide 77

Slide 77 text

@CodingDoug URLConnectionInstrumentation static method class UrlConnectionInstrumentation { companion object { @JvmStatic fun openConnection(url: URL): URLConnection { return when (val conn = url.openConnection()) { is HttpsURLConnection -> InstrumentedHttpsURLConnection(url, conn) is HttpURLConnection -> InstrumentedHttpURLConnection(url, conn) else -> conn } } } }

Slide 78

Slide 78 text

@CodingDoug Decorator: InstrumentedURLConnection (sample) private class InstrumentedHttpsURLConnection(url: URL, private val urlConn: HttpsURLConnection) : HttpsURLConnection { override fun connect() { // Could add code here to mark the start of the transaction urlConn.connect() } override fun getContent(): Any { // Could measure the time it takes for this to complete return urlConn.content } override fun getInputStream(): InputStream { // Also decorate and measure other objects exposed by the API return InstrumentedInputStream(urlConn.inputStream) } // ... and many more overrides call through to urlConn }

Slide 79

Slide 79 text

@CodingDoug In the end… ● Use ASM to replace interesting method calls with static method calls ○ invokevirtual #x -> invokestatic #y ● Implement static methods to call through to original method, decorate the result ○ Runtime support provided by companion library ● Decorators: ○ Override all methods ○ Always call through to original objects ○ Internally measure HTTP transaction behavior

Slide 80

Slide 80 text

@CodingDoug Tips for transforms ● Transforms can be slow for large apps, so: ○ Be sure to implement incremental builds optimally ○ Allow the developer to disable it easily ○ Don’t process classes unnecessarily ● Don’t transform the companion library classes! ● Write integration tests against transformed classes ● ./gradlew -t install

Slide 81

Slide 81 text

Thank you! @CodingDoug firebase.google.com Get the code: bit.ly/android-transformers-repo