Upgrade to Pro — share decks privately, control downloads, hide ads and more …

(Android) Transformers - bytecode in disguise!

(Android) Transformers - bytecode in disguise!

If you’ve used Firebase Performance Monitoring, you know that it automatically discovers and reports performance metrics for all the HTTP transactions in your app. But have you ever wondered how it does that? Android doesn’t provide any APIs to listen in on an arbitrary connection (that would be a security problem!), and Firebase supports URLConnection, Apache, and OKHTTP APIs. The secret here is bytecode manipulation at build time using the Transform API provided by the Android build tools. Join this session to find out how the Firebase Performance Monitoring Gradle plugin intercepts and measures HTTP transactions at runtime with some assistance from bytecode manipulation at build time.

Doug Stevenson

April 23, 2019
Tweet

More Decks by Doug Stevenson

Other Decks in Technology

Transcript

  1. (Android) Transformers -
    bytecode in disguise!
    Doug Stevenson
    @CodingDoug

    View full-size slide

  2. @CodingDoug
    How can you build a no-code HTTP data collector SDK?
    ● Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache)
    ● No API hooks to collect HTTP transaction data
    ● Options:
    ○ Modify app source code?
    ○ JVM “agent”?
    ○ Modify bytecode of compiled JVM classes, before deployment

    View full-size slide

  3. @CodingDoug
    How can you build a no-code HTTP data collector SDK?
    ● Several common ways to perform HTTP requests (OKHTTP, URLConnection, Apache)
    ● No API hooks to collect HTTP transaction data
    ● Options:
    ○ Modify app source code? (infeasible)
    ○ JVM “agent”? (impossible on Android)
    ○ Modify bytecode of compiled JVM classes, before deployment

    View full-size slide

  4. @CodingDoug
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()
    Task: instrument bytecode that compiled from source like this:

    View full-size slide

  5. @CodingDoug
    Java/Kotlin
    Compiler
    classes / jars
    DX / D8
    dex file(s)
    APK
    aapt2

    View full-size slide

  6. @CodingDoug
    classes / jars
    DX / D8
    But where exactly
    are the classes?
    And how exactly do I
    hook the build process?
    bytecode stuff
    goes here

    View full-size slide

  7. @CodingDoug
    DX / D8

    View full-size slide

  8. @CodingDoug
    DX / D8
    private val inputJars: List =
    [ /path/to/jar ]
    private val inputClasses: List =
    [ /path/to/classes ]
    Read private
    config at runtime
    (reflection)

    View full-size slide

  9. @CodingDoug
    DX / D8
    private val inputJars: List =
    [ /path/to/jar ] [ /my/modified/jar ]
    private val inputClasses: List =
    [ /path/to/classes ] [ /my/modified/classes ]
    Don’t ever do this.
    Messing with private members via reflection is wrong!
    And inject my
    own changes
    Read private
    config at runtime
    (reflection)

    View full-size slide

  10. @CodingDoug
    DX / D8
    private val inputJars: List =
    [ /path/to/jar ] [ /my/modified/jar ]
    private val inputClasses: List =
    [ /path/to/classes ] [ /my/modified/classes ]

    View full-size slide

  11. @CodingDoug
    classes / jars
    DX / D8
    Proguard Transform
    Your Transform
    Another Transform

    View full-size slide

  12. @CodingDoug
    Gradle plugin
    Uses Transform API
    Does bytecode stuff
    Companion SDK
    Runtime support
    Sample app
    Modified by transform
    Open sourced sample project components

    View full-size slide

  13. @CodingDoug
    project-root
    transformer-app
    transformer-libs
    transformer-module
    transformer-plugin

    View full-size slide

  14. The Transform Gradle plugin

    View full-size slide

  15. @CodingDoug
    Transform Gradle plugin - build.gradle
    apply plugin: 'java-gradle-plugin'
    apply plugin: 'kotlin'
    apply plugin: 'maven'
    repositories {
    google()
    jcenter()
    }
    dependencies {
    compileOnly gradleApi()
    compileOnly 'com.android.tools.build:gradle:3.4.2'
    implementation 'org.jetbrains.kotlin:kotlin-stdlib:1.3.41'
    implementation 'commons-io:commons-io:2.6'
    implementation ‘org.ow2.asm:asm:7.1'
    }

    View full-size slide

  16. @CodingDoug
    implementation-class=com.hyperaware.transformer.plugin.MyPlugin
    Define the plugin entry point in META-INF

    with a properties file

    View full-size slide

  17. @CodingDoug
    Transform Gradle plugin - MyPlugin class
    class MyPlugin : Plugin {
    override fun apply(project: Project) {
    project.logger.log(LogLevel.INFO, "MyPlugin applied")
    // Check to see if this is an android project
    val ext = project.extensions.findByName("android")
    if (ext != null && ext is AppExtension) {
    project.logger.log(LogLevel.INFO, "Registering transform")
    // Register our class transform
    ext.registerTransform(MyTransform(project))
    // Add an extension for gradle configuration
    project.extensions.create("transform", MyExtension::class.java)
    }
    else {
    throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android
    app projects")
    }
    }
    }

    View full-size slide

  18. @CodingDoug
    Transform Gradle plugin - MyPlugin class
    class MyPlugin : Plugin {
    override fun apply(project: Project) {
    project.logger.log(LogLevel.INFO, "MyPlugin applied")
    // Check to see if this is an android project
    val ext = project.extensions.findByName("android")
    if (ext != null && ext is AppExtension) {
    project.logger.log(LogLevel.INFO, "Registering transform")
    // Register our class transform
    ext.registerTransform(MyTransform(project))
    // Add an extension for gradle configuration
    project.extensions.create("transform", MyExtension::class.java)
    }
    else {
    throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android
    app projects")
    }
    }
    }

    View full-size slide

  19. @CodingDoug
    Transform Gradle plugin - MyPlugin class
    class MyPlugin : Plugin {
    override fun apply(project: Project) {
    project.logger.log(LogLevel.INFO, "MyPlugin applied")
    // Check to see if this is an android project
    val ext = project.extensions.findByName("android")
    if (ext != null && ext is AppExtension) {
    project.logger.log(LogLevel.INFO, "Registering transform")
    // Register our class transform
    ext.registerTransform(MyTransform(project))
    // Add an extension for gradle configuration
    project.extensions.create("transform", MyExtension::class.java)
    }
    else {
    throw Exception("${MyPlugin::class.java.name} plugin may only be applied to Android
    app projects")
    }
    }
    }

    View full-size slide

  20. @CodingDoug
    Transform Gradle plugin - MyTransform class
    class MyTransform(private val project: Project) : Transform() {
    override fun getName(): String {
    return MyTransform::class.java.simpleName
    }
    // This transform is interested in classes only (and not resources)
    private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES)
    override fun getInputTypes(): Set {
    return typeClasses
    }
    // more ...
    }

    View full-size slide

  21. @CodingDoug
    Transform Gradle plugin - MyTransform class
    class MyTransform(private val project: Project) : Transform() {
    override fun getName(): String {
    return MyTransform::class.java.simpleName
    }
    // This transform is interested in classes only (and not resources)
    private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES)
    override fun getInputTypes(): Set {
    return typeClasses
    }
    // more ...
    }

    View full-size slide

  22. @CodingDoug
    Transform Gradle plugin - MyTransform class
    class MyTransform(private val project: Project) : Transform() {
    override fun getName(): String {
    return MyTransform::class.java.simpleName
    }
    // This transform is interested in classes only (and not resources)
    private val typeClasses = setOf(QualifiedContent.DefaultContentType.CLASSES)
    override fun getInputTypes(): Set {
    return typeClasses
    }
    // more ...
    }

    View full-size slide

  23. @CodingDoug
    Transform Gradle plugin - MyTransform
    class MyTransform(private val project: Project) : Transform() {
    // ...
    // This transform is interested in classes from all parts of the app
    private val scopes = setOf(
    QualifiedContent.Scope.PROJECT,
    QualifiedContent.Scope.SUB_PROJECTS,
    QualifiedContent.Scope.EXTERNAL_LIBRARIES
    )
    override fun getScopes(): MutableSet {
    return scopes.toMutableSet()
    }
    // more ...
    }

    View full-size slide

  24. @CodingDoug
    Transform Gradle plugin - MyTransform
    class MyTransform(private val project: Project) : Transform() {
    // ...
    // This transform is interested in classes from all parts of the app
    private val scopes = setOf(
    QualifiedContent.Scope.PROJECT,
    QualifiedContent.Scope.SUB_PROJECTS,
    QualifiedContent.Scope.EXTERNAL_LIBRARIES
    )
    override fun getScopes(): MutableSet {
    return scopes.toMutableSet()
    }
    // more ...
    }

    View full-size slide

  25. @CodingDoug
    Transform Gradle plugin - MyTransform
    class MyTransform(private val project: Project) : Transform() {
    // ...
    // This transform can handle incremental builds
    override fun isIncremental(): Boolean {
    return true
    }
    // more ...
    }

    View full-size slide

  26. @CodingDoug
    Transform Gradle plugin - MyTransform
    class MyTransform(private val project: Project) : Transform() {
    // ...
    override fun transform(transformInvocation: TransformInvocation) {
    // Find the Gradle extension that contains configuration for this Transform
    val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension()
    val appExtension = project.extensions.findByName("android") as AppExtension
    val ignores = listOf(
    Regex("com/hyperaware/transformer/.*")
    )
    val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores,
    ext)
    MyTransformImpl(config).doIt()
    }
    }

    View full-size slide

  27. @CodingDoug
    Transform Gradle plugin - MyTransform
    class MyTransform(private val project: Project) : Transform() {
    // ...
    override fun transform(transformInvocation: TransformInvocation) {
    // Find the Gradle extension that contains configuration for this Transform
    val ext = project.extensions.findByType(MyExtension::class.java) ?: MyExtension()
    val appExtension = project.extensions.findByName("android") as AppExtension
    val ignores = listOf(
    Regex("com/hyperaware/transformer/.*")
    )
    val config = TransformConfig(transformInvocation, appExtension.bootClasspath, ignores,
    ext)
    MyTransformImpl(config).doIt()
    }
    }

    View full-size slide

  28. @CodingDoug
    TransformInvocation interface - defines inputs and ouputs
    public interface TransformInvocation {
    @NonNull
    Context getContext();
    @NonNull
    Collection getInputs();
    @NonNull
    Collection getReferencedInputs();
    @NonNull
    Collection getSecondaryInputs();
    @Nullable
    TransformOutputProvider getOutputProvider();
    boolean isIncremental();
    }

    View full-size slide

  29. @CodingDoug
    Transformer Gradle plugin - MyTransformImpl
    class MyTransformImpl(config: TransformConfig) {
    private val logger = Logging.getLogger(MyTransformImpl::class.java)
    private val transformInvocation = config.transformInvocation
    private val androidClasspath = config.androidClasspath
    private val ignorePaths = config.ignorePaths
    private val outputProvider = transformInvocation.outputProvider
    private val instrumentationConfig = InstrumentationConfig(
    buildRuntimeClasspath(transformInvocation),
    config.pluginConfig.logVisits,
    config.pluginConfig.logInstrumentation
    )
    private val instrumenter = ClassInstrumenter(instrumentationConfig)
    fun doIt() {
    logger.debug(instrumentationConfig.toString())
    logger.debug("isIncremental: ${transformInvocation.isIncremental}")
    for (ti in transformInvocation.inputs) {
    instrumentDirectoryInputs(ti.directoryInputs)
    instrumentJarInputs(ti.jarInputs)
    }
    }
    /**
    * Builds the runtime classpath of the project. This combines all the
    * various TransformInput file locations in addition to the targeted
    * Android platform jar into a single collection that's suitable to be a
    * classpath for the entire app.
    */
    private fun buildRuntimeClasspath(transformInvocation: TransformInvocation): List {
    val allTransformInputs = transformInvocation.inputs + transformInvocation.referencedInputs
    val allJarsAndDirs = allTransformInputs.map { ti ->
    (ti.directoryInputs + ti.jarInputs).map { i -> i.file }
    }
    val allClassesAtRuntime = androidClasspath + allJarsAndDirs.flatten()
    return allClassesAtRuntime.map { file -> file.toURI().toURL() }
    }
    private fun instrumentDirectoryInputs(directoryInputs: Collection) {
    // A DirectoryInput is a tree of class files that simply gets
    // copied to the output directory.
    //
    for (di in directoryInputs) {
    // Build a unique name for the output dir based on the path
    // of the input dir.
    //
    logger.debug("TransformInput dir $di")
    val outDir = outputProvider.getContentLocation(di.name, di.contentTypes, di.scopes,
    Format.DIRECTORY)
    logger.debug(" Directory input ${di.file}")
    logger.debug(" Directory output $outDir")
    if (transformInvocation.isIncremental) {
    // Incremental builds will specify which individual class files changed.
    for (changedFile in di.changedFiles) {
    when (changedFile.value) {
    private fun instrumentJarInputs(jarInputs: Collection) {
    // A JarInput is a jar file that just gets copied to a destination
    // output jar.
    //
    for (ji in jarInputs) {
    // Build a unique name for the output file based on the path
    // of the input jar.
    //
    logger.debug("TransformInput jar $ji")
    val outDir = outputProvider.getContentLocation(ji.name, ji.contentTypes, ji.scopes,
    Format.DIRECTORY)
    logger.debug(" Jar input ${ji.file}")
    logger.debug(" Dir output $outDir")
    val doTransform = !transformInvocation.isIncremental || ji.status == Status.ADDED || ji.status
    == Status.CHANGED
    if (doTransform) {
    ensureDirectoryExists(outDir)
    FileUtils.cleanDirectory(outDir)
    val inJar = JarFile(ji.file)
    var count = 0
    for (entry in inJar.entries()) {
    val outFile = File(outDir, entry.name)
    if (!entry.isDirectory) {
    ensureDirectoryExists(outFile.parentFile)
    inJar.getInputStream(entry).use { inputStream ->
    IOUtils.buffer(FileOutputStream(outFile)).use { outputStream ->
    if (isInstrumentableClassFile(entry.name)) {
    try {
    processClassStream(entry.name, inputStream, outputStream)
    }
    catch (e: Exception) {
    logger.error("Can't process class ${entry.name}", e)
    throw e
    }
    }
    else {
    copyStream(inputStream, outputStream)
    }
    }
    }
    count++
    }
    }
    logger.debug(" Entries copied: $count")
    }
    else if (ji.status == Status.REMOVED) {
    logger.debug(" REMOVED")
    if (outDir.exists()) {
    FileUtils.forceDelete(outDir)
    }
    }
    }
    }
    Status.ADDED, Status.CHANGED -> {
    val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key)
    val destFile = File(outDir, relativeFile)
    changedFile.key.inputStream().use { inputStream ->
    destFile.outputStream().use { outputStream ->
    if (isInstrumentableClassFile(relativeFile)) {
    processClassStream(relativeFile, inputStream, outputStream)
    }
    else {
    copyStream(inputStream, outputStream)
    }
    }
    }
    }
    Status.REMOVED -> {
    val relativeFile = normalizedRelativeFilePath(di.file, changedFile.key)
    val destFile = File(outDir, relativeFile)
    FileUtils.forceDelete(destFile)
    }
    Status.NOTCHANGED, null -> {
    }
    }
    }
    logger.debug(" Files processed: ${di.changedFiles.size}")
    }
    else {
    ensureDirectoryExists(outDir)
    FileUtils.cleanDirectory(outDir)
    logger.debug(" Copying ${di.file} to $outDir")
    var count = 0
    for (file in FileUtils.iterateFiles(di.file, null, true)) {
    val relativeFile = normalizedRelativeFilePath(di.file, file)
    val destFile = File(outDir, relativeFile)
    ensureDirectoryExists(destFile.parentFile)
    IOUtils.buffer(file.inputStream()).use { inputStream ->
    IOUtils.buffer(destFile.outputStream()).use { outputStream ->
    if (isInstrumentableClassFile(relativeFile)) {
    try {
    processClassStream(relativeFile, inputStream, outputStream)
    }
    catch (e: Exception) {
    logger.error("Can't process class $file", e)
    throw e
    }
    }
    else {
    copyStream(inputStream, outputStream)
    }
    }
    }
    count++
    }
    logger.debug(" Files processed: $count")
    }
    }
    }
    private fun ensureDirectoryExists(dir: File) {
    if (! ((dir.isDirectory && dir.canWrite()) || dir.mkdirs())) {
    throw IOException("Can't write or create ${dir.path}")
    }
    }
    // Builds a relative path from a given file and its parent.
    // For file /a/b/c and parent /a, returns "b/c".
    private fun normalizedRelativeFilePath(parent: File, file: File): String {
    val parts = mutableListOf()
    var current = file
    while (current != parent) {
    parts.add(current.name)
    current = current.parentFile
    }
    return parts.asReversed().joinToString("/")
    }
    // Checks the (relative) path of a given class file and returns true if
    // it's assumed to be instrumentable. The path must end with .class and
    // also not match any of the regular expressions in ignorePaths.
    private fun isInstrumentableClassFile(path: String): Boolean {
    return if (ignorePaths.any { it.matches(path) }) {
    logger.debug("Ignoring class $path")
    false
    }
    else {
    path.toLowerCase().endsWith(".class")
    }
    }
    private fun copyStream(inputStream: InputStream, outputStream: OutputStream) {
    IOUtils.copy(inputStream, outputStream)
    }
    private fun processClassStream(name: String, inputStream: InputStream, outputStream: OutputStream) {
    val classBytes = IOUtils.toByteArray(inputStream)
    val bytesToWrite = try {
    val instrBytes = instrumenter.instrument(classBytes)
    instrBytes
    }
    catch (e: Exception) {
    // If instrumentation fails, just write the original bytes
    logger.error("Failed to instrument $name, using original contents", e)
    classBytes
    }
    IOUtils.write(bytesToWrite, outputStream)
    }
    }

    View full-size slide

  30. @CodingDoug
    TransformInvocation processing pseudocode
    foreach TransformInput object “ti”
    foreach input in “ti” (jars and directories)
    if input requires processing
    output = ti.outputProvider.getContentLocation(...)
    foreach file in the jar or directory
    if it’s a class file
    read the bytecode
    // honestly, the only interesting part
    use ASM to make bytecode changes as needed
    write bytecode to output
    else
    copy the file to output

    View full-size slide

  31. @CodingDoug
    ASM

    asm.ow2.io

    View full-size slide

  32. @CodingDoug
    ASM uses the visitor pattern to deliver class contents

    View full-size slide

  33. @CodingDoug
    ClassVisitor
    - visit()
    - visitOuterClass(name)
    - visitInnerClass(name)
    - visitAnnotation(desc): AnnotationVisitor
    - visitField(name, desc): FieldVisitor
    - visitMethod(name, desc): MethodVisitor
    ASM uses the visitor pattern to deliver class contents

    View full-size slide

  34. @CodingDoug
    ClassVisitor
    - visit()
    - visitOuterClass(name)
    - visitInnerClass(name)
    - visitAnnotation(desc): AnnotationVisitor
    - visitField(name, desc): FieldVisitor
    - visitMethod(name, desc): MethodVisitor
    ASM uses the visitor pattern to deliver class contents

    View full-size slide

  35. @CodingDoug
    ClassVisitor
    - visit()
    - visitOuterClass(name)
    - visitInnerClass(name)
    - visitAnnotation(desc): AnnotationVisitor
    - visitField(name, desc): FieldVisitor
    - visitMethod(name, desc): MethodVisitor
    ASM uses the visitor pattern to deliver class contents

    View full-size slide

  36. @CodingDoug
    ClassVisitor
    - visit()
    - visitOuterClass(name)
    - visitInnerClass(name)
    - visitAnnotation(desc): AnnotationVisitor
    - visitField(name, desc): FieldVisitor
    - visitMethod(name, desc): MethodVisitor
    MethodVisitor
    - visitParameter(name)
    - visitAnnotation(desc): AnnotationVisitor
    - visitCode()
    - visitLocalVariable(name)
    - visitMethodInsn(opcode)
    - visitEnd()
    ASM uses the visitor pattern to deliver class contents

    View full-size slide

  37. Let’s visit some bytecode!

    View full-size slide

  38. @CodingDoug
    But first, a little bit about the JVM
    Constant Pool
    ● One per-class
    ● Pre-computed values, references, types
    ● Indexed by position
    #5 = Utf8 onCreate
    #6 = Utf8 (Landroid/os/Bundle;)V
    #7 = Utf8 Lorg/jetbrains/annotations/Nullable;
    #8 = NameAndType #5:#6 // onCreate:(Landroid/os/Bundle;)V
    #9 = Methodref #4.#8 // android/support/v7/app/AppCompatActivi
    #10 = Integer 2131296284
    #11 = Utf8 setContentView
    #12 = Utf8 (I)V
    #13 = NameAndType #11:#12 // setContentView:(I)V

    View full-size slide

  39. @CodingDoug
    But first, a little bit about the JVM
    Local variables
    ● Similar to registers for a typical CPU
    ● Indexed by position,

    0 = “this” for instance methods
    ● Typically map 1:1 with Java/Kotlin locals
    LocalVariableTable:
    Start Length Slot Name Signature
    38 13 4 input Ljava/io/InputStream;
    26 25 3 conn Ljava/net/URLConnection;
    15 36 2 array [B
    10 41 1 url Ljava/net/URL;
    0 51 0 this Lcom/hyperaware/transformers/MyRunnable;

    View full-size slide

  40. @CodingDoug
    But first, a little bit about the JVM
    Operand Stack
    ● Push values onto it (constants, locals, fields)
    ● Execute operations against pushed values (e.g. math, call method),

    those values are popped, result of operation is pushed
    ● Similar to Reverse Polish Notation (aka postfix notation)
    // KOTLIN: val c = a + b
    0: iload_1 // push local int variable 1 “a”
    1: iload_2 // push local int variable 2 “b”
    2: iadd // add them, pop them, push the result
    3: istore_3 // store the result in local variable 3 “c”, pop it

    View full-size slide

  41. @CodingDoug
    But first, a little bit about the JVM
    Stack Frame
    ● Created on the current thread’s stack for each method call
    ● Contains local variables & operand stack

    (max sizes are pre-determined)
    ● Popped off the stack when the method returns

    View full-size slide

  42. @CodingDoug
    ClassInstrumenter - kick off bytecode processing
    class ClassInstrumenter(private val config: InstrumentationConfig) {
    private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray())
    fun instrument(input: ByteArray): ByteArray {
    val cr = ClassReader(input)
    // Custom ClassWriter needs to specify a ClassLoader that knows
    // about all classes in the app.
    val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) {
    override fun getClassLoader(): ClassLoader = cl
    }
    // Our InstrumentationVisitor wraps the ClassWriter to intercept and
    // change bytecode as class elements are being visited.
    val iv = InstrumentationVisitor(cw, config)
    cr.accept(iv, ClassReader.SKIP_FRAMES)
    return cw.toByteArray()
    }
    }

    View full-size slide

  43. @CodingDoug
    ClassInstrumenter - kick off bytecode processing
    class ClassInstrumenter(private val config: InstrumentationConfig) {
    private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray())
    fun instrument(input: ByteArray): ByteArray {
    val cr = ClassReader(input)
    // Custom ClassWriter needs to specify a ClassLoader that knows
    // about all classes in the app.
    val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) {
    override fun getClassLoader(): ClassLoader = cl
    }
    // Our InstrumentationVisitor wraps the ClassWriter to intercept and
    // change bytecode as class elements are being visited.
    val iv = InstrumentationVisitor(cw, config)
    cr.accept(iv, ClassReader.SKIP_FRAMES)
    return cw.toByteArray()
    }
    }

    View full-size slide

  44. @CodingDoug
    ClassInstrumenter - kick off bytecode processing
    class ClassInstrumenter(private val config: InstrumentationConfig) {
    private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray())
    fun instrument(input: ByteArray): ByteArray {
    val cr = ClassReader(input)
    // Custom ClassWriter needs to specify a ClassLoader that knows
    // about all classes in the app.
    val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) {
    override fun getClassLoader(): ClassLoader = cl
    }
    // Our InstrumentationVisitor wraps the ClassWriter to intercept and
    // change bytecode as class elements are being visited.
    val iv = InstrumentationVisitor(cw, config)
    cr.accept(iv, ClassReader.SKIP_FRAMES)
    return cw.toByteArray()
    }
    }

    View full-size slide

  45. @CodingDoug
    ClassInstrumenter - kick off bytecode processing
    class ClassInstrumenter(private val config: InstrumentationConfig) {
    private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray())
    fun instrument(input: ByteArray): ByteArray {
    val cr = ClassReader(input)
    // Custom ClassWriter needs to specify a ClassLoader that knows
    // about all classes in the app.
    val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) {
    override fun getClassLoader(): ClassLoader = cl
    }
    // Our InstrumentationVisitor wraps the ClassWriter to intercept and
    // change bytecode as class elements are being visited.
    val iv = InstrumentationVisitor(cw, config)
    cr.accept(iv, ClassReader.SKIP_FRAMES)
    return cw.toByteArray()
    }
    }

    View full-size slide

  46. @CodingDoug
    ClassInstrumenter - kick off bytecode processing
    class ClassInstrumenter(private val config: InstrumentationConfig) {
    private val cl = URLClassLoader(config.runtimeClasspath.toTypedArray())
    fun instrument(input: ByteArray): ByteArray {
    val cr = ClassReader(input)
    // Custom ClassWriter needs to specify a ClassLoader that knows
    // about all classes in the app.
    val cw = object : ClassWriter(ClassWriter.COMPUTE_MAXS or ClassWriter.COMPUTE_FRAMES) {
    override fun getClassLoader(): ClassLoader = cl
    }
    // Our InstrumentationVisitor wraps the ClassWriter to intercept and
    // change bytecode as class elements are being visited.
    val iv = InstrumentationVisitor(cw, config)
    cr.accept(iv, ClassReader.SKIP_FRAMES)
    return cw.toByteArray()
    }
    }

    View full-size slide

  47. @CodingDoug
    class bytes ASM ClassReader
    InstrumentationVisitor

    (a ClassVisitor)
    ASM ClassWriter

    (a ClassVisitor)
    modified

    class bytes
    invokes visitor methods
    invokes visitor methods
    decorates

    View full-size slide

  48. @CodingDoug
    InstrumentationVisitor skeleton
    class InstrumentationVisitor(
    classVisitor ClassVisitor, // actually a ClassWriter
    private val config: InstrumentationConfig
    ) : ClassVisitor(ASM_API_VERSION, classVisitor) {
    }

    View full-size slide

  49. @CodingDoug
    InstrumentationVisitor skeleton
    class InstrumentationVisitor(
    classVisitor ClassVisitor, // actually a ClassWriter
    private val config: InstrumentationConfig
    ) : ClassVisitor(ASM_API_VERSION, classVisitor) {
    override fun visit(
    version: Int,
    access: Int,
    className: String,
    signature: String?,
    superName: String,
    interfaces: Array?
    )
    override fun visitMethod(
    access: Int,
    methodName: String,
    methodDesc: String,
    signature: String?,
    exceptions: Array?
    ): MethodVisitor
    }

    View full-size slide

  50. @CodingDoug
    InstrumentationVisitor.visitMethod()
    override fun visitMethod(
    access: Int, // public / private / final / etc
    methodName: String, // e.g. "openConnection"
    methodDesc: String, // e.g. "()Ljava/net/URLConnection;
    signature: String?, // for any generics
    exceptions: Array? // declared exceptions thrown
    ): MethodVisitor {
    }

    View full-size slide

  51. @CodingDoug
    InstrumentationVisitor.visitMethod()
    override fun visitMethod(
    access: Int, // public / private / final / etc
    methodName: String, // e.g. "openConnection"
    methodDesc: String, // e.g. "()Ljava/net/URLConnection;
    signature: String?, // for any generics
    exceptions: Array? // declared exceptions thrown
    ): MethodVisitor {
    // Get a MethodVisitor using the ClassVisitor we’re decorating
    val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions)
    }

    View full-size slide

  52. @CodingDoug
    InstrumentationVisitor.visitMethod()
    override fun visitMethod(
    access: Int, // public / private / final / etc
    methodName: String, // e.g. "openConnection"
    methodDesc: String, // e.g. "()Ljava/net/URLConnection;
    signature: String?, // for any generics
    exceptions: Array? // declared exceptions thrown
    ): MethodVisitor {
    // Get a MethodVisitor using the ClassVisitor we’re decorating
    val mv = super.visitMethod(access, methodName, methodDesc, signature, exceptions)
    // Wrap it in a custom MethodVisitor subclass
    return MyMethodVisitor(ASM_API_VERSION, mv, access, methodName, methodDesc)
    }

    View full-size slide

  53. @CodingDoug
    class MyMethodVisitor
    private inner class MyMethodVisitor(
    api: Int, // class API version (ASM constant)
    mv: MethodVisitor, // original decorated MethodVisitor
    access: Int, // access flags (e.g. public, final...)
    methodName: String, // name of the declared method
    methodDesc: String // signature of the declared method
    ): AdviceAdapter(api, mv, access, methodName, methodDesc) {
    }

    View full-size slide

  54. @CodingDoug
    class MyMethodVisitor
    private inner class MyMethodVisitor(
    api: Int, // class API version (ASM constant)
    mv: MethodVisitor, // original decorated MethodVisitor
    access: Int, // access flags (e.g. public, final...)
    methodName: String, // name of the declared method
    methodDesc: String // signature of the declared method
    ): AdviceAdapter(api, mv, access, methodName, methodDesc) {
    override fun visitMethodInsn(
    opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic)
    owner: String, // containing object
    name: String, // name of the method
    desc: String, // signature
    itf: Boolean) // is this from an interface?
    }

    View full-size slide

  55. @CodingDoug
    So, what are we trying to do, again?
    Automatically measure all HTTP requests with code that looks like this:
    fun makeRequest() {
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()
    }

    View full-size slide

  56. @CodingDoug
    javap -c verbose SomeClass.class (1)
    public final void makeRequest();
    descriptor: ()V
    flags: ACC_PUBLIC, ACC_FINAL
    Code:
    stack=3, locals=5, args_size=1
    LocalVariableTable:
    Start Length Slot Name Signature
    38 13 4 input Ljava/io/InputStream;
    26 25 3 conn Ljava/net/URLConnection;
    15 36 2 array [B
    10 41 1 url Ljava/net/URL;
    0 51 0 this Lcom/hyperaware/transformers/MyRunnable2;
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  57. @CodingDoug
    javap -c verbose SomeClass.class (2)
    // KOTLIN: val url = URL("https://www.google.com")
    0: new #10 // class java/net/URL
    3: dup
    4: ldc #12 // String https://www.google.com
    6: invokespecial #16 // Method java/net/URL."":(Ljava/lang/String;)V
    9: astore_1
    // KOTLIN: val array = ByteArray(15)
    10: bipush 15
    12: newarray byte
    14: astore_2
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  58. @CodingDoug
    javap -c verbose SomeClass.class (3)
    // KOTLIN: val conn: URLConnection = url.openConnection()
    15: aload_1
    16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection;
    // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”)
    19: dup
    20: ldc #67 // String url.openConnection()
    22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull:
    (Ljava/lang/Object;Ljava/lang/String;)V
    25: astore_3
    // KOTLIN: val input: InputStream = conn.getInputStream()
    26: aload_3
    27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream;
    // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”)
    30: dup
    31: ldc #75 // String conn.getInputStream()
    33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull:
    (Ljava/lang/Object;Ljava/lang/String;)V
    36: astore 4
    38: aload 4
    40: aload_2
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  59. @CodingDoug
    javap -c verbose SomeClass.class (4)
    // KOTLIN: val conn: URLConnection = url.openConnection()
    15: aload_1
    16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection;
    // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(conn, “url.openConnection()”)
    19: dup
    20: ldc #67 // String url.openConnection()
    22: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull:
    (Ljava/lang/Object;Ljava/lang/String;)V
    25: astore_3
    // KOTLIN: val input: InputStream = conn.getInputStream()
    26: aload_3
    27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream;
    // HIDDEN KOTLIN: Intrinsics.checkExpressionValueIsNotNull(input, “conn.getInputStream()”)
    30: dup
    31: ldc #75 // String conn.getInputStream()
    33: invokestatic #73 // Method kotlin/jvm/internal/Intrinsics.checkExpressionValueIsNotNull:
    (Ljava/lang/Object;Ljava/lang/String;)V
    36: astore 4
    38: aload 4
    40: aload_2
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  60. @CodingDoug
    javap -c verbose SomeClass.class (5)
    // KOTLIN: val conn: URLConnection = url.openConnection()
    15: aload_1
    // URL is now pushed on the stack
    16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection;
    // URL is popped, and URLConnection is now pushed
    25: astore_3
    // URLConnection is stored in local var 3 and popped
    // KOTLIN: val input: InputStream = conn.getInputStream()
    26: aload_3
    27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream;
    36: astore 4
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  61. @CodingDoug
    Before:
    val conn = url.openConnection()
    After:
    val conn = URLConnectionInstrumentation.openConnection(url)
    Given: 1) this static method is available at runtime, and 2) InstrumentedHttpsURLConnection subclasses
    HttpsURLConnection:
    @JvmStatic fun openConnection(url: URL): HttpsURLConnection {
    return InstrumentedHttpsURLConnection(url, url.openConnection())
    }

    View full-size slide

  62. @CodingDoug
    More generally, before:
    val res: R = obj.fn(a, b, c)
    After:
    val res: R = UtilityClass.fn(obj, a, b, c)
    Provided:
    class UtilityClass {
    companion object {
    @JvmStatic fun fn(obj, a, b, c): R {
    return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R
    }
    }
    }

    View full-size slide

  63. @CodingDoug
    More generally, before:
    val res: R = obj.fn(a, b, c)
    After:
    val res: R = UtilityClass.fn(obj, a, b, c)
    Provided:
    class UtilityClass {
    companion object {
    @JvmStatic fun fn(obj, a, b, c): R {
    return DecoratedResult(obj, obj.fn(a, b, c)) // DecoratedResult extends R
    }
    }
    }

    View full-size slide

  64. @CodingDoug
    Bytecode change - one weird trick
    Kotlin: val res: R = obj.fn(a, b, c)
    Bytecode: invokevirtual #x // #x -> pkg/of/obj.fn()
    Kotlin: val res: R = UtilityClass.fn(obj, a, b, c)
    Bytecode: invokestatic #y // #y -> pkg/of/UtilityClass.fn()
    3 c
    2 b
    1 a
    0 obj
    before
    0 res
    after
    1. Pop 4 items off the stack, obj and args a, b, c
    2. Make a new frame with obj, a, b, c as locals
    3. Call fn instance method on obj
    4. Push returned res object on the stack
    1. Pop the top 4 items off the stack, obj and args a, b, c
    2. Make a new frame with obj, a, b, c as locals
    3. Call the UtilityClass.fn static method
    4. Push returned res object on the stack

    View full-size slide

  65. @CodingDoug
    Only one change to bytecode is necessary here:

    replace invokevirtual #x with invokestatic #y

    (while the companion library static method does the real work)

    View full-size slide

  66. @CodingDoug
    One weird trick, applied
    // KOTLIN: val conn: URLConnection = url.openConnection()
    15: aload_1
    // URL is now pushed on the stack
    16: invokevirtual #20 // Method java/net/URL.openConnection:()Ljava/net/URLConnection;
    16: invokestatic #41 // Method UrlConnectionInstrumentation.openConnection:(Ljava/net/
    URL;)Ljava/net/URLConnection;
    // URL is popped, and URLConnection is now pushed
    25: astore_3
    // URLConnection is stored in local var 3 and popped
    // KOTLIN: val input: InputStream = conn.getInputStream()
    26: aload_3
    27: invokevirtual #26 // Method java/net/URLConnection.getInputStream:()Ljava/io/InputStream;
    36: astore 4
    val url = URL("https://www.google.com")
    val array = ByteArray(15)
    val conn: URLConnection = url.openConnection()
    val input: InputStream = conn.getInputStream()
    input.read(array)
    input.close()

    View full-size slide

  67. How do you do it?

    Let’s revisit the visitor

    View full-size slide

  68. @CodingDoug
    Revisit the ASM visitor
    private inner class MyMethodVisitor(
    api: Int,
    mv: MethodVisitor,
    access: Int,
    methodName: String,
    methodDesc: String
    ): AdviceAdapter(api, mv, access, methodName, methodDesc) {
    override fun visitMethodInsn(
    opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic)
    owner: String, // containing object
    name: String, // name of the method
    desc: String, // signature
    itf: Boolean) // is this from an interface?
    }

    View full-size slide

  69. @CodingDoug
    Implement that weird trick with ASM
    override fun visitMethodInsn(
    opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic)
    owner: String, // containing object
    name: String, // name of the method
    desc: String, // signature
    itf: Boolean) { // is this from an interface?
    if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") {
    // This is the method call we’re looking for! Make the switch
    }
    else {
    // Don’t touch a thing
    }
    }

    View full-size slide

  70. @CodingDoug
    Implement that weird trick with ASM
    override fun visitMethodInsn(
    opcode: Int, // type of method call this is (e.g. invokevirtual, invokestatic)
    owner: String, // containing object
    name: String, // name of the method
    desc: String, // signature
    itf: Boolean) { // is this from an interface?
    if (owner == "java/net/URL" && name == "openConnection" && desc == "()Ljava/net/URLConnection;") {
    // This is the method call we’re looking for! Make the switch
    super.visitMethodInsn(
    Opcodes.INVOKESTATIC, // static method call
    "com/company/utility/UrlConnectionInstrumentation", // utility class
    “openConnection", // static method
    "(Ljava/net/URL;)Ljava/net/URLConnection;", // method signature
    false) // is this from an interface?
    }
    else {
    // Don’t touch a thing
    super.visitMethodInsn(opcode, owner, name, desc, itf)
    }
    }

    View full-size slide

  71. @CodingDoug
    URLConnectionInstrumentation static method
    class UrlConnectionInstrumentation {
    companion object {
    @JvmStatic
    fun openConnection(url: URL): URLConnection {
    return when (val conn = url.openConnection()) {
    is HttpsURLConnection ->
    InstrumentedHttpsURLConnection(url, conn)
    is HttpURLConnection ->
    InstrumentedHttpURLConnection(url, conn)
    else ->
    conn
    }
    }
    }
    }

    View full-size slide

  72. @CodingDoug
    Decorator: InstrumentedURLConnection (sample)
    private class InstrumentedHttpsURLConnection(url: URL, private val urlConn: HttpsURLConnection)
    : HttpsURLConnection {
    override fun connect() {
    // Could add code here to mark the start of the transaction
    urlConn.connect()
    }
    override fun getContent(): Any {
    // Could measure the time it takes for this to complete
    return urlConn.content
    }
    override fun getInputStream(): InputStream {
    // Also decorate and measure other objects exposed by the API
    return InstrumentedInputStream(urlConn.inputStream)
    }
    // ... and many more overrides call through to urlConn
    }

    View full-size slide

  73. @CodingDoug
    In the end…
    ● Use ASM to replace interesting method calls with static method calls
    ○ invokevirtual #x -> invokestatic #y
    ● Implement static methods to call through to original method, decorate the result
    ○ Runtime support provided by companion library
    ● Decorators:
    ○ Override all methods
    ○ Always call through to original objects
    ○ Internally measure HTTP transaction behavior

    View full-size slide

  74. @CodingDoug
    Tips for transforms
    ● Transforms can be slow for large apps, so:
    ○ Be sure to implement incremental builds optimally
    ○ Allow the developer to disable it easily
    ○ Don’t process classes unnecessarily
    ● Don’t transform the companion library classes!
    ● Write integration tests against transformed classes
    ● ./gradlew -t install

    View full-size slide

  75. Thank you!
    @CodingDoug
    firebase.google.com
    Get the code:
    bit.ly/android-transformers-repo

    View full-size slide