(Android) Transformers - bytecode in disguise!

If you’ve used Firebase Performance Monitoring, you know that it automatically discovers and reports performance metrics for all the HTTP transactions in your app. But have you ever wondered how it does that? Android doesn’t provide any APIs to listen in on an arbitrary connection (that would be a security problem!), and Firebase supports URLConnection, Apache, and OKHTTP APIs. The secret here is bytecode manipulation at build time using the Transform API provided by the Android build tools. Join this session to find out how the Firebase Performance Monitoring Gradle plugin intercepts and measures HTTP transactions at runtime with some assistance from bytecode manipulation at build time.

Transcript

1. (Android) Transformers - bytecode in disguise! Doug Stevenson @CodingDoug
2. @CodingDoug
3. @CodingDoug
4. @CodingDoug
5. @CodingDoug
6. @CodingDoug
7. @CodingDoug
8. @CodingDoug How can you build a no-code HTTP data collector SDK? • Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache) • No API hooks to collect HTTP transaction data • Options: ◦ Modify app source code? ◦ JVM “agent”? ◦ Modify bytecode of compiled JVM classes, before deployment
9. @CodingDoug How can you build a no-code HTTP data collector SDK? • Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache) • No API hooks to collect HTTP transaction data • Options: ◦ Modify app source code? (infeasible) ◦ JVM “agent”? (impossible on Android) ◦ Modify bytecode of compiled JVM classes, before deployment
10. @CodingDoug val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close() Task: instrument bytecode that compiled from source like this:
11. @CodingDoug Java/Kotlin Compiler classes / jars DX / D8 dex file(s) APK aapt2
12. @CodingDoug classes / jars DX / D8 But where exactly are the classes? And how exactly do I hook the build process? bytecode stuff goes here
13. @CodingDoug DX / D8
14. @CodingDoug DX / D8 private val inputJars: List<File> = [ /path/to/jar ] private val inputClasses: List<File> = [ /path/to/classes ] Read private config at runtime (reflection)
15. @CodingDoug DX / D8 private val inputJars: List<File> = [ /path/to/jar ] [ /my/modified/jar ] private val inputClasses: List<File> = [ /path/to/classes ] [ /my/modified/classes ] Don’t ever do this. Messing with private members via reflection is wrong! And inject my own changes Read private config at runtime (reflection)
16. @CodingDoug DX / D8 private val inputJars: List<File> = [ /path/to/jar ] [ /my/modified/jar ] private val inputClasses: List<File> = [ /path/to/classes ] [ /my/modified/classes ]
17. @CodingDoug classes / jars DX / D8 Proguard Transform Your Transform Another Transform
18. @CodingDoug Gradle plugin Uses Transform API Does bytecode stuff Companion SDK Runtime support Sample app Modified by transform Open sourced sample project components
19. @CodingDoug project-root transformer-app transformer-libs transformer-module transformer-plugin
20. The Transform Gradle plugin
21. @CodingDoug Transform Gradle plugin - build.gradle apply plugin: 'java-gradle-plugin' apply plugin: 'kotlin' apply plugin: 'maven' repositories { google() jcenter() } dependencies { compileOnly gradleApi() compileOnly 'com.android.tools.build:gradle:3.4.2' implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.3.41' implementation 'commons-io:commons-io:2.6' implementation ‘org.ow2.asm:asm:7.1' }
22. @CodingDoug implementation-class=com.hyperaware.transformer.plugin.MyPlugin Define the plugin entry point in META-INF with a properties file
23. @CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin<Project> { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }
24. @CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin<Project> { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }
25. @CodingDoug Transform Gradle plugin - MyPlugin class class MyPlugin : Plugin<Project> { override fun apply(project: Project) { project.logger.log(LogLevel.INFO, "MyPlugin applied") // Check to see if this is an android project val ext = project.extensions.findByName("android") if (ext != null && ext is AppExtension) { project.logger.log(LogLevel.INFO, "Registering transform") // Register our class transform ext.registerTransform(MyTransform(project)) // Add an extension for gradle configuration project.extensions.create("transform", MyExtension::class.java) } else { throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android app projects") } } }
26. @CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set<QualifiedContent.ContentType> { return typeClasses } // more ... }
27. @CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set<QualifiedContent.ContentType> { return typeClasses } // more ... }
28. @CodingDoug Transform Gradle plugin - MyTransform class class MyTransform(private val project: Project) : Transform() { override fun getName(): String { return MyTransform::class.java.simpleName } // This transform is interested in classes only (and not resources) private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES) override fun getInputTypes(): Set<QualifiedContent.ContentType> { return typeClasses } // more ... }
29. @CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform is interested in classes from all parts of the app private val scopes = setOf( QualifiedContent.Scope.PROJECT, QualifiedContent.Scope.SUB_PROJECTS, QualifiedContent.Scope.EXTERNAL_LIBRARIES ) override fun getScopes(): MutableSet<in QualifiedContent.Scope> { return scopes.toMutableSet() } // more ... }
30. @CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform is interested in classes from all parts of the app private val scopes = setOf( QualifiedContent.Scope.PROJECT, QualifiedContent.Scope.SUB_PROJECTS, QualifiedContent.Scope.EXTERNAL_LIBRARIES ) override fun getScopes(): MutableSet<in QualifiedContent.Scope> { return scopes.toMutableSet() } // more ... }
31. @CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... // This transform can handle incremental builds override fun isIncremental(): Boolean { return true } // more ... }
32. @CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... override fun transform(transformInvocation: TransformInvocation) { // Find the Gradle extension that contains configuration for this Transform val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension() val appExtension = project.extensions.findByName("android") as AppExtension val ignores = listOf( Regex("com/hyperaware/transformer/.*") ) val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores, ext) MyTransformImpl(config).doIt() } }
33. @CodingDoug Transform Gradle plugin - MyTransform class MyTransform(private val project: Project) : Transform() { // ... override fun transform(transformInvocation: TransformInvocation) { // Find the Gradle extension that contains configuration for this Transform val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension() val appExtension = project.extensions.findByName("android") as AppExtension val ignores = listOf( Regex("com/hyperaware/transformer/.*") ) val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores, ext) MyTransformImpl(config).doIt() } }
34. @CodingDoug TransformInvocation interface - defines inputs and ouputs public interface TransformInvocation { @NonNull Context getContext(); @NonNull Collection<TransformInput> getInputs(); @NonNull Collection<TransformInput> getReferencedInputs(); @NonNull Collection<SecondaryInput> getSecondaryInputs(); @Nullable TransformOutputProvider getOutputProvider(); boolean isIncremental(); }
35. @CodingDoug Transformer Gradle plugin - MyTransformImpl class MyTransformImpl(config: TransformConfig) { private val logger = Logging.getLogger(MyTransformImpl::class.java) private val transformInvocation = config.transformInvocation private val androidClasspath = config.androidClasspath private val ignorePaths = config.ignorePaths private val outputProvider = transformInvocation.outputProvider private val instrumentationConfig = InstrumentationConfig( buildRuntimeClasspath(transformInvocation), config.pluginConfig.logVisits, config.pluginConfig.logInstrumentation ) private val instrumenter = ClassInstrumenter(instrumentationConfig) fun doIt() { logger.debug(instrumentationConfig.toString()) logger.debug("isIncremental: ${transformInvocation.isIncremental}") for (ti in transformInvocation.inputs) { instrumentDirectoryInputs(ti.directoryInputs) instrumentJarInputs(ti.jarInputs) } } /** * Builds the runtime classpath of the project. This combines all the * various TransformInput file locations in addition to the targeted * Android platform jar into a single collection that's suitable to be a * classpath for the entire app. */ private fun buildRuntimeClasspath(transformInvocation: TransformInvocation): List<URL> { val allTransformInputs = transformInvocation.inputs + transformInvocation.referencedInputs val allJarsAndDirs = allTransformInputs.map { ti -> (ti.directoryInputs + ti.jarInputs).map { i -> i.file } } val allClassesAtRuntime = androidClasspath + allJarsAndDirs.flatten() return allClassesAtRuntime.map { file -> file.toURI().toURL() } } private fun instrumentDirectoryInputs(directoryInputs: Collection<DirectoryInput>) { // A DirectoryInput is a tree of class files that simply gets // copied to the output directory. // for (di in directoryInputs) { // Build a unique name for the output dir based on the path // of the input dir. // logger.debug("TransformInput dir $di") val outDir = outputProvider.getContentLocation(di.name, di.contentTypes, di.scopes, Format.DIRECTORY) logger.debug(" Directory input ${di.file}") logger.debug(" Directory output $outDir") if (transformInvocation.isIncremental) { // Incremental builds will specify which individual class files changed. for (changedFile in di.changedFiles) { when (changedFile.value) { private fun instrumentJarInputs(jarInputs: Collection<JarInput>) { // A JarInput is a jar file that just gets copied to a destination // output jar. // for (ji in jarInputs) { // Build a unique name for the output file based on the path // of the input jar. // logger.debug("TransformInput jar $ji") val outDir = outputProvider.getContentLocation(ji.name, ji.contentTypes, ji.scopes, Format.DIRECTORY) logger.debug(" Jar input ${ji.file}") logger.debug(" Dir output $outDir") val doTransform = !transformInvocation.isIncremental || ji.status == Status.ADDED || ji.status == Status.CHANGED if (doTransform) { ensureDirectoryExists(outDir) FileUtils.cleanDirectory(outDir) val inJar = JarFile(ji.file) var count = 0 for (entry in inJar.entries()) { val outFile = File(outDir, entry.name) if (!entry.isDirectory) { ensureDirectoryExists(outFile.parentFile) inJar.getInputStream(entry).use { inputStream -> IOUtils.buffer(FileOutputStream(outFile)).use { outputStream -> if (isInstrumentableClassFile(entry.name)) { try { processClassStream(entry.name, inputStream, outputStream) } catch (e: Exception) { logger.error("Can't process class ${entry.name}", e) throw e } } else { copyStream(inputStream, outputStream) } } } count++ } } logger.debug(" Entries copied: $count") } else if (ji.status == Status.REMOVED) { logger.debug(" REMOVED") if (outDir.exists()) { FileUtils.forceDelete(outDir) } } } } Status.ADDED, Status.CHANGED -> { val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key) val destFile = File(outDir, relativeFile) changedFile.key.inputStream().use { inputStream -> destFile.outputStream().use { outputStream -> if (isInstrumentableClassFile(relativeFile)) { processClassStream(relativeFile, inputStream, outputStream) } else { copyStream(inputStream, outputStream) } } } } Status.REMOVED -> { val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key) val destFile = File(outDir, relativeFile) FileUtils.forceDelete(destFile) } Status.NOTCHANGED, null -> { } } } logger.debug(" Files processed: ${di.changedFiles.size}") } else { ensureDirectoryExists(outDir) FileUtils.cleanDirectory(outDir) logger.debug(" Copying ${di.file} to $outDir") var count = 0 for (file in FileUtils.iterateFiles(di.file, null, true)) { val relativeFile = normalizedRelativeFilePath(di.file, file) val destFile = File(outDir, relativeFile) ensureDirectoryExists(destFile.parentFile) IOUtils.buffer(file.inputStream()).use { inputStream -> IOUtils.buffer(destFile.outputStream()).use { outputStream -> if (isInstrumentableClassFile(relativeFile)) { try { processClassStream(relativeFile, inputStream, outputStream) } catch (e: Exception) { logger.error("Can't process class $file", e) throw e } } else { copyStream(inputStream, outputStream) } } } count++ } logger.debug(" Files processed: $count") } } } private fun ensureDirectoryExists(dir: File) { if (! ((dir.isDirectory && dir.canWrite()) || dir.mkdirs())) { throw IOException("Can't write or create ${dir.path}") } } // Builds a relative path from a given file and its parent. // For file /a/b/c and parent /a, returns "b/c". private fun normalizedRelativeFilePath(parent: File, file: File): String { val parts = mutableListOf<String>() var current = file while (current != parent) { parts.add(current.name) current = current.parentFile } return parts.asReversed().joinToString("/") } // Checks the (relative) path of a given class file and returns true if // it's assumed to be instrumentable. The path must end with .class and // also not match any of the regular expressions in ignorePaths. private fun isInstrumentableClassFile(path: String): Boolean { return if (ignorePaths.any { it.matches(path) }) { logger.debug("Ignoring class $path") false } else { path.toLowerCase().endsWith(".class") } } private fun copyStream(inputStream: InputStream, outputStream: OutputStream) { IOUtils.copy(inputStream, outputStream) } private fun processClassStream(name: String, inputStream: InputStream, outputStream: OutputStream) { val classBytes = IOUtils.toByteArray(inputStream) val bytesToWrite = try { val instrBytes = instrumenter.instrument(classBytes) instrBytes } catch (e: Exception) { // If instrumentation fails, just write the original bytes logger.error("Failed to instrument $name, using original contents", e) classBytes } IOUtils.write(bytesToWrite, outputStream) } }
36. @CodingDoug TransformInvocation processing pseudocode foreach TransformInput object “ti” foreach input in “ti” (jars and directories) if input requires processing output = ti.outputProvider.getContentLocation(...) foreach file in the jar or directory if it’s a class file read the bytecode // honestly, the only interesting part use ASM to make bytecode changes as needed write bytecode to output else copy the file to output
37. @CodingDoug ASM asm.ow2.io
38. @CodingDoug ASM uses the visitor pattern to deliver class contents
39. @CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents
40. @CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents
41. @CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor ASM uses the visitor pattern to deliver class contents
42. @CodingDoug ClassVisitor - visit() - visitOuterClass(name) - visitInnerClass(name) - visitAnnotation(desc): AnnotationVisitor - visitField(name, desc): FieldVisitor - visitMethod(name, desc): MethodVisitor MethodVisitor - visitParameter(name) - visitAnnotation(desc): AnnotationVisitor - visitCode() - visitLocalVariable(name) - visitMethodInsn(opcode) - visitEnd() ASM uses the visitor pattern to deliver class contents
43. Let’s visit some bytecode!
44. @CodingDoug But first, a little bit about the JVM Constant Pool • One per-class • Pre-computed values, references, types • Indexed by position #5 = Utf8 onCreate #6 = Utf8 (Landroid/os/Bundle;)V #7 = Utf8 Lorg/jetbrains/annotations/Nullable; #8 = NameAndType #5:#6 // onCreate:(Landroid/os/Bundle;)V #9 = Methodref #4.#8 // android/support/v7/app/AppCompatActivi #10 = Integer 2131296284 #11 = Utf8 setContentView #12 = Utf8 (I)V #13 = NameAndType #11:#12 // setContentView:(I)V
45. @CodingDoug But first, a little bit about the JVM Local variables • Similar to registers for a typical CPU • Indexed by position,
 0 = “this” for instance methods • Typically map 1:1 with Java/Kotlin locals LocalVariableTable: Start Length Slot Name Signature 38 13 4 input Ljava/io/InputStream; 26 25 3 conn Ljava/net/URLConnection; 15 36 2 array [B 10 41 1 url Ljava/net/URL; 0 51 0 this Lcom/hyperaware/transformers/MyRunnable;
46. @CodingDoug But first, a little bit about the JVM Operand Stack • Push values onto it (constants, locals, fields) • Execute operations against pushed values (e.g. math, call method),
 those values are popped, result of operation is pushed • Similar to Reverse Polish Notation (aka postfix notation) // KOTLIN: val c = a + b 0: iload_1 // push local int variable 1 “a” 1: iload_2 // push local int variable 2 “b” 2: iadd // add them, pop them, push the result 3: istore_3 // store the result in local variable 3 “c”, pop it
47. @CodingDoug But first, a little bit about the JVM Stack Frame • Created on the current thread’s stack for each method call • Contains local variables & operand stack
 (max sizes are pre-determined) • Popped off the stack when the method returns
48. @CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }
49. @CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }
50. @CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }
51. @CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }
52. @CodingDoug ClassInstrumenter - kick off bytecode processing class ClassInstrumenter(private val config: InstrumentationConfig) { private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray()) fun instrument(input: ByteArray): ByteArray { val cr = ClassReader(input) // Custom ClassWriter needs to specify a ClassLoader that knows // about all classes in the app. val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) { override fun getClassLoader(): ClassLoader = cl } // Our InstrumentationVisitor wraps the ClassWriter to intercept and // change bytecode as class elements are being visited. val iv = InstrumentationVisitor(cw, config) cr.accept(iv, ClassReader.SKIP_FRAMES) return cw.toByteArray() } }
53. @CodingDoug class bytes ASM ClassReader InstrumentationVisitor
 (a ClassVisitor) ASM ClassWriter
 (a ClassVisitor) modified
 class bytes invokes visitor methods invokes visitor methods decorates
54. @CodingDoug InstrumentationVisitor skeleton class InstrumentationVisitor( classVisitor ClassVisitor, // actually a ClassWriter private val config: InstrumentationConfig ) : ClassVisitor(ASM_API_VERSION, classVisitor) { }
55. @CodingDoug InstrumentationVisitor skeleton class InstrumentationVisitor( classVisitor ClassVisitor, // actually a ClassWriter private val config: InstrumentationConfig ) : ClassVisitor(ASM_API_VERSION, classVisitor) { override fun visit( version: Int, access: Int, className: String, signature: String?, superName: String, interfaces: Array<String>? ) override fun visitMethod( access: Int, methodName: String, methodDesc: String, signature: String?, exceptions: Array<String>? ): MethodVisitor }
56. @CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array<String>? // declared exceptions thrown ): MethodVisitor { }
57. @CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array<String>? // declared exceptions thrown ): MethodVisitor { // Get a MethodVisitor using the ClassVisitor we’re decorating val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions) }
58. @CodingDoug InstrumentationVisitor.visitMethod() override fun visitMethod( access: Int, // public / private / final / etc methodName: String, // e.g. "openConnection" methodDesc: String, // e.g. "()Ljava/net/URLConnection; signature: String?, // for any generics exceptions: Array<String>? // declared exceptions thrown ): MethodVisitor { // Get a MethodVisitor using the ClassVisitor we’re decorating val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions) // Wrap it in a custom MethodVisitor subclass return MyMethodVisitor(ASM_API_VERSION, mv, access, methodName, methodDesc) }
59. @CodingDoug class MyMethodVisitor private inner class MyMethodVisitor( api: Int, // class API version (ASM constant) mv: MethodVisitor, // original decorated MethodVisitor access: Int, // access flags (e.g. public, final...) methodName: String, // name of the declared method methodDesc: String // signature of the declared method ): AdviceAdapter(api, mv, access, methodName, methodDesc) { }
60. @CodingDoug class MyMethodVisitor private inner class MyMethodVisitor( api: Int, // class API version (ASM constant) mv: MethodVisitor, // original decorated MethodVisitor access: Int, // access flags (e.g. public, final...) methodName: String, // name of the declared method methodDesc: String // signature of the declared method ): AdviceAdapter(api, mv, access, methodName, methodDesc) { override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) // is this from an interface? }
61. @CodingDoug So, what are we trying to do, again? Automatically measure all HTTP requests with code that looks like this: fun makeRequest() { val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close() }
62. @CodingDoug javap -c verbose SomeClass.class (1) public final void makeRequest(); descriptor: ()V flags: ACC_PUBLIC, ACC_FINAL Code: stack=3, locals=5, args_size=1 LocalVariableTable: Start Length Slot Name Signature 38 13 4 input Ljava/io/InputStream; 26 25 3 conn Ljava/net/URLConnection; 15 36 2 array [B 10 41 1 url Ljava/net/URL; 0 51 0 this Lcom/hyperaware/transformers/MyRunnable2; val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
63. @CodingDoug javap -c verbose SomeClass.class (2) // KOTLIN: val url = URL("https://www.google.com") 0: new #10 // class java/net/URL 3: dup 4: ldc #12 // String https://www.google.com 6: invokespecial #16 // Method java/net/URL."<init>":(Ljava/lang/String;)V 9: astore_1 // KOTLIN: val array = ByteArray(15) 10: bipush 15 12: newarray byte 14: astore_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
64. @CodingDoug javap -c verbose SomeClass.class (3) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”) 19: dup 20: ldc #67 // String url.openConnection() 22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 25: astore_3 // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”) 30: dup 31: ldc #75 // String conn.getInputStream() 33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 36: astore 4 38: aload 4 40: aload_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
65. @CodingDoug javap -c verbose SomeClass.class (4) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”) 19: dup 20: ldc #67 // String url.openConnection() 22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 25: astore_3 // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”) 30: dup 31: ldc #75 // String conn.getInputStream() 33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull: (Ljava/lang/Object;Ljava/lang/String;)V 36: astore 4 38: aload 4 40: aload_2 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
66. @CodingDoug javap -c verbose SomeClass.class (5) // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 // URL is now pushed on the stack 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; // URL is popped, and URLConnection is now pushed 25: astore_3 // URLConnection is stored in local var 3 and popped // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; 36: astore 4 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
67. @CodingDoug Before: val conn = url.openConnection() After: val conn = URLConnectionInstrumentation.openConnection(url) Given: 1) this static method is available at runtime, and 2) InstrumentedHttpsURLConnection subclasses HttpsURLConnection: @JvmStatic fun openConnection(url: URL): HttpsURLConnection { return InstrumentedHttpsURLConnection(url, url.openConnection()) }
68. @CodingDoug More generally, before: val res: R = obj.fn(a, b, c) After: val res: R = UtilityClass.fn(obj, a, b, c) Provided: class UtilityClass { companion object { @JvmStatic fun fn(obj, a, b, c): R { return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R } } }
69. @CodingDoug More generally, before: val res: R = obj.fn(a, b, c) After: val res: R = UtilityClass.fn(obj, a, b, c) Provided: class UtilityClass { companion object { @JvmStatic fun fn(obj, a, b, c): R { return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R } } }
70. @CodingDoug Bytecode change - one weird trick Kotlin: val res: R = obj.fn(a, b, c) Bytecode: invokevirtual #x // #x -> pkg/of/obj.fn() Kotlin: val res: R = UtilityClass.fn(obj, a, b, c) Bytecode: invokestatic #y // #y -> pkg/of/UtilityClass.fn() 3 c 2 b 1 a 0 obj before 0 res after 1. Pop 4 items off the stack, obj and args a, b, c 2. Make a new frame with obj, a, b, c as locals 3. Call fn instance method on obj 4. Push returned res object on the stack 1. Pop the top 4 items off the stack, obj and args a, b, c 2. Make a new frame with obj, a, b, c as locals 3. Call the UtilityClass.fn static method 4. Push returned res object on the stack
71. @CodingDoug Only one change to bytecode is necessary here: replace invokevirtual #x with invokestatic #y (while the companion library static method does the real work)
72. @CodingDoug One weird trick, applied // KOTLIN: val conn: URLConnection = url.openConnection() 15: aload_1 // URL is now pushed on the stack 16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection; 16: invokestatic #41 // Method UrlConnectionInstrumentation.openConnection:(Ljava/net/ URL;)Ljava/net/URLConnection; // URL is popped, and URLConnection is now pushed 25: astore_3 // URLConnection is stored in local var 3 and popped // KOTLIN: val input: InputStream = conn.getInputStream() 26: aload_3 27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream; 36: astore 4 val url = URL("https://www.google.com") val array = ByteArray(15) val conn: URLConnection = url.openConnection() val input: InputStream = conn.getInputStream() input.read(array) input.close()
73. How do you do it? Let’s revisit the visitor
74. @CodingDoug Revisit the ASM visitor private inner class MyMethodVisitor( api: Int, mv: MethodVisitor, access: Int, methodName: String, methodDesc: String ): AdviceAdapter(api, mv, access, methodName, methodDesc) { override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) // is this from an interface? }
75. @CodingDoug Implement that weird trick with ASM override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) { // is this from an interface? if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") { // This is the method call we’re looking for! Make the switch } else { // Don’t touch a thing } }
76. @CodingDoug Implement that weird trick with ASM override fun visitMethodInsn( opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic) owner: String, // containing object name: String, // name of the method desc: String, // signature itf: Boolean) { // is this from an interface? if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") { // This is the method call we’re looking for! Make the switch super.visitMethodInsn( Opcodes.INVOKESTATIC, // static method call "com/company/utility/UrlConnectionInstrumentation", // utility class “openConnection", // static method "(Ljava/net/URL;)Ljava/net/URLConnection;", // method signature false) // is this from an interface? } else { // Don’t touch a thing super.visitMethodInsn(opcode, owner, name, desc, itf) } }
77. @CodingDoug URLConnectionInstrumentation static method class UrlConnectionInstrumentation { companion object { @JvmStatic fun openConnection(url: URL): URLConnection { return when (val conn = url.openConnection()) { is HttpsURLConnection -> InstrumentedHttpsURLConnection(url, conn) is HttpURLConnection -> InstrumentedHttpURLConnection(url, conn) else -> conn } } } }
78. @CodingDoug Decorator: InstrumentedURLConnection (sample) private class InstrumentedHttpsURLConnection(url: URL, private val urlConn: HttpsURLConnection) : HttpsURLConnection { override fun connect() { // Could add code here to mark the start of the transaction urlConn.connect() } override fun getContent(): Any { // Could measure the time it takes for this to complete return urlConn.content } override fun getInputStream(): InputStream { // Also decorate and measure other objects exposed by the API return InstrumentedInputStream(urlConn.inputStream) } // ... and many more overrides call through to urlConn }
79. @CodingDoug In the end… • Use ASM to replace interesting method calls with static method calls ◦ invokevirtual #x -> invokestatic #y • Implement static methods to call through to original method, decorate the result ◦ Runtime support provided by companion library • Decorators: ◦ Override all methods ◦ Always call through to original objects ◦ Internally measure HTTP transaction behavior
80. @CodingDoug Tips for transforms • Transforms can be slow for large apps, so: ◦ Be sure to implement incremental builds optimally ◦ Allow the developer to disable it easily ◦ Don’t process classes unnecessarily • Don’t transform the companion library classes! • Write integration tests against transformed classes • ./gradlew -t install
81. Thank you! @CodingDoug firebase.google.com Get the code: bit.ly/android-transformers-repo