Mix-It 2012 - Manipulation de bytecode : démocratisons la magie noire !

Mix-It 2012 - Manipulation de bytecode : démocratisons la magie noire !

2a4bd42b6a3db5858ff2250e236f9631?s=128

Frédéric Le Mouël

May 02, 2012
Tweet

Transcript

  1. 1 Manipulation de bytecode : démocratisons la magie noire !

    Julien Ponge @jponge Frédéric Le Mouël @flemouel
  2. 2 Gare de Lyon, restaurant "le Train Bleu" Carnet de

    voyage, Roger O’Reilly
  3. 3 @jponge @flemouel

  4. 4 Nous faisons du middleware. dynamique adaptatif reloading multi-tenancy resource-control

    dynamic aspects (...) (...) pervasif
  5. 5 Conteneurs (Java EE, Spring) Android liblayout Play! Framework JRebel

    Hibernate Langages sur JVM Terracotta FindBugs 4
  6. 6 java.lang.reflect ... parfois il faut “plus”

  7. 7 JVM+Bytecode ASM AspectJ Byteman JooFlux ... en fait c’est

    facile ! écrire et transformer du bytecode faire simplement des modifications statiques courantes avec des aspects aspects dynamiques, application à l’amélioration des couvertures de tests aspects dynamiques via invokedynamic
  8. JVM + Bytecode crash course 8

  9. 9 “JVM Bytecode for Dummies” — Charles Nutter @headius Clair,

    complet et didactique ! 8
  10. 10 java/lang/String Nom de classe

  11. 11 V void Z boolean C char B byte S

    short I int F float J long D double Types primitifs
  12. 12 Ljava/lang/String; Type objet : L;

  13. 13 [I [Ljava/lang/String; Tableaux : [

  14. 14 (I,I)I int foo(int, int) (Ljava/lang/String;)V void foo(String)

  15. 15 Constant pool (nombres, String, types) Variables locales Machine à

    pile OpCodes invocation de méthode manipulation de pile get / set champs arithmétique allocation sauts lock (...) 0 java/lang/String 1 “Foo” 2 666 ... ... 0 this (sauf static) 1 arg1 2 arg2 3 int i ... ...
  16. 16 public class HelloWorld { public static void main(String[] args)

    { int i = 1; int j = 2; int k = 100; System.out.println(">>> " + (i + j + k)); } }
  17. 17 $ javap -c HelloWorld Compiled from "HelloWorld.java" public class

    HelloWorld extends java.lang.Object{ public HelloWorld(); Code: 0: aload_0 1: invokespecial #1; //Method java/lang/Object."<init>":()V 4: return (...)
  18. 17 $ javap -c HelloWorld Compiled from "HelloWorld.java" public class

    HelloWorld extends java.lang.Object{ public HelloWorld(); Code: 0: aload_0 1: invokespecial #1; //Method java/lang/Object."<init>":()V 4: return (...) invocation private et constructeur + invoke{static, interface, virtual, dynamic} décompilateur constructeur + <cinit>, <clinit> this dans le constant pool
  19. 18 public static void main(java.lang.String[]); Code: 0: iconst_1 1: istore_1

    2: iconst_2 3: istore_2 4: bipush 100 6: istore_3 7: getstatic #2; //Field java/lang/System.out:Ljava/io/PrintStream; 10: new #3; //class java/lang/StringBuilder 13: dup 14: invokespecial #4; //Method java/lang/StringBuilder."<init>":()V 17: ldc #5; //String >>> 19: invokevirtual #6; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuil 22: iload_1 23: iload_2 24: iadd 25: iload_3 26: iadd 27: invokevirtual #7; //Method java/lang/StringBuilder.append:(I)Ljava/lang/StringBuilder; 30: invokevirtual #8; //Method java/lang/StringBuilder.toString:()Ljava/lang/String; 33: invokevirtual #9; //Method java/io/PrintStream.println:(Ljava/lang/String;)V 36: return ? ? ?
  20. 19 1 iconst_1 0 args 1 2 3 variables

  21. 20 istore_1 0 args 1 1 2 3 variables

  22. 21 2 iconst_2 0 args 1 1 2 3 variables

  23. 22 istore_2 0 args 1 1 2 2 3 variables

  24. 23 100 bipush 100 0 args 1 1 2 2

    3 variables
  25. 24 istore_3 0 args 1 1 2 2 3 100

    variables
  26. 25 out getstatic System.out 0 args 1 1 2 2

    3 100 variables
  27. 26 builder out new StringBuilder 0 args 1 1 2

    2 3 100 variables
  28. 27 builder builder out dup 0 args 1 1 2

    2 3 100 variables
  29. 28 builder out invokespecial StringBuilder.<init> 0 args 1 1 2

    2 3 100 variables
  30. 29 “>>> “ builder out ldc “>>> ” 0 args

    1 1 2 2 3 100 variables
  31. 30 builder out invokevirtual append(String) 0 args 1 1 2

    2 3 100 variables
  32. 31 1 builder out iload_1 0 args 1 1 2

    2 3 100 variables
  33. 32 2 1 builder out iload_2 0 args 1 1

    2 2 3 100 variables
  34. 33 3 builder out iadd 0 args 1 1 2

    2 3 100 variables
  35. 34 100 3 builder out iload_3 0 args 1 1

    2 2 3 100 variables
  36. 35 103 builder out iadd 0 args 1 1 2

    2 3 100 variables
  37. 36 builder out invokevirtual append(int) 0 args 1 1 2

    2 3 100 variables
  38. 37 “>>> 103” out invokevirtual toString() 0 args 1 1

    2 2 3 100 variables
  39. 38 invokevirtual println(String) 0 args 1 1 2 2 3

    100 variables
  40. 39 return 0 args 1 1 2 2 3 100

    variables
  41. ASM 40

  42. 41 package my; public class Main { public static void

    main(String[] args) { for (int i = 0; i < 10; i++) { System.out.println("Hello Devoxx France!"); } } } Générons ceci sans javac !
  43. 42 i: int [start, end] start: i = 0 loop:

    println(System.out, “Hello Devoxx France!”) incr(i, 1) jump(loop) if (i < 10) end: return goto isn’t harmful!
  44. 43 ClassWriter writer = new ClassWriter(COMPUTE_MAXS | COMPUTE_FRAMES); writer.visit(V1_6, ACC_PUBLIC,

    "my/Main", null, "java/lang/Object", null); writer.visitSource("Main.java", null); MethodVisitor mv = writer.visitMethod(ACC_STATIC | ACC_PUBLIC, "main", "([Ljava/lang/String;)V", null, null); mv.visitCode(); Classe + main(String...)
  45. 44 Label start = new Label(); Label loop = new

    Label(); Label end = new Label(); mv.visitLabel(start); final int COUNTER_VAR = 1; mv.visitLocalVariable("counter", "I", null, start, end, COUNTER_VAR); mv.visitInsn(ICONST_0); mv.visitVarInsn(ISTORE, COUNTER_VAR); i = 0
  46. 45 mv.visitLabel(loop); mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;"); mv.visitLdcInsn("Hello Devoxx France!"); mv.visitMethodInsn(INVOKEVIRTUAL,

    "java/io/PrintStream", "println", "(Ljava/lang/String;)V"); println()
  47. 46 mv.visitIincInsn(COUNTER_VAR, 1); mv.visitVarInsn(ILOAD, COUNTER_VAR); mv.visitLdcInsn(10); mv.visitJumpInsn(IF_ICMPLT, loop); mv.visitLabel(end); mv.visitInsn(RETURN);

    mv.visitMaxs(666, 666); mv.visitEnd(); writer.visitEnd(); i++, (i < 10)?, goto
  48. 47 File outDir = new File("gen/my"); outDir.mkdirs(); FileOutputStream out =

    new FileOutputStream(new File(outDir, "Main.class")); out.write(writer.toByteArray()); out.close(); .class !
  49. 48 (réalisé sans trucages)

  50. 49 profiling de chaque méthode utilisée par un programme

  51. Manifest-Version: 1.0 Premain-Class: profiler.ProfilerTransformer Can-Redefine-Classes: true Can-Retransform-Classes: true Can-Set-Native-Method-Prefix: false

    java -javaagent:profiler.jar some.Main 49
  52. 51 public class ProfilerTransformer implements ClassFileTransformer { public static void

    premain(String agentArgs, Instrumentation inst) { inst.addTransformer(new ProfilerTransformer()); } public byte[] transform(ClassLoader classLoader, String s, Class<?> aClass, ProtectionDomain protectionDomain, byte[] bytes) throws IllegalClassFormatException { ClassReader reader = new ClassReader(bytes); ClassWriter writer = new ClassWriter(reader, COMPUTE_FRAMES | COMPUTE_MAXS); ProfilerClassVisitor profiler = new ProfilerClassVisitor(ASM4, writer); reader.accept(profiler, 0); return writer.toByteArray(); } }
  53. 52 @Override public void visitCode() { super.visitCode(); startVarId = newLocal(Type.getType("J"));

    mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime", "()J"); mv.visitVarInsn(LSTORE, startVarId); }
  54. 53 @Override public void visitInsn(int opcode) { if ((opcode >=

    IRETURN && opcode <= RETURN) || opcode == ATHROW) { (...) mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime", "()J"); mv.visitVarInsn(LLOAD, startVarId); mv.visitInsn(LSUB); mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(J)V"); } super.visitInsn(opcode); }
  55. 54 ➟ adapters, inliner, optimizers, ... ➟ modèle objet ➟

    analyse statique de code ➟ tracing, retro-engineering, ... org.objectweb.asm.commons org.objectweb.asm.tree org.objectweb.asm.tree.analysis org.objectweb.asm.util
  56. 55 “Ok, et pour les cas plus simples ?”

  57. 56

  58. 57 WTF Layer WTF Layer WTF Layer Security, validation, logging,

    transactions, ...
  59. 58 public class UserController { // (...) public void update(UserProfile

    profile) throws ControllerException { validator.check(profile); if (logger.isLevelEnabled(DEBUG)) { logger.debug("Updating: " + profile); } dao.save(profile); redirect("/home"); } // (...) } “cross-cutting concerns” “business logic”
  60. 59 Pointcuts Advices Inter-type declarations Où ? Quoi ? +

    méthodes + attributs + hiérarchie + vérifications + ordre
  61. 60 (demo) * * en cas de plantage, ne jettez

    pas de tomates, merci.
  62. 61 aspect ArgsTracer { before(): target(HelloWorld) && call(void run(String...)) {

    System.out.println("Calling run()"); } } Que se passe t-il avec un before ?
  63. 62 --- HelloWorld.bytecode.original 2012-04-11 22:36:22.000000000 +0200 +++ HelloWorld.bytecode.weaved 2012-04-11 22:45:51.000000000

    +0200 @@ -19,8 +19,16 @@ 3: dup 4: invokespecial #34; //Method "<init>":()V 7: aload_0 - 8: invokevirtual #35; //Method run:([Ljava/lang/String;)V - 11: return + 8: astore_1 + 9: astore_2 + 10: invokestatic #44; //Method ArgsTracer.aspectOf:()LArgsTracer; + 13: aload_2 + 14: aload_1 + 15: invokevirtual #48; //Method ArgsTracer.ajc$before$ArgsTracer$1$66cd2654:(LHelloWorld; [Ljava/lang/String;)V + 18: aload_2 + 19: aload_1 + 20: invokevirtual #35; //Method run:([Ljava/lang/String;)V + 23: return }
  64. 63 aspect MakePersonComparable { declare parents: Person implements Comparable<Person>; public

    int Person.compareTo(Person other) { return getName().compareTo(other.getName()); } } Que se passe t-il avec des déclarations inter-types ?
  65. 64 $ javap -c -private Person Compiled from "Person.java" public

    class Person extends java.lang.Object implements java.lang.Comparable{ (...) public int compareTo(java.lang.Object); Code: 0: aload_0 1: aload_1 2: checkcast #1; //class Person 5: invokevirtual #50; //Method compareTo:(LPerson;)I 8: ireturn public int compareTo(Person); Code: 0: aload_0 1: aload_1 2: invokestatic #60; //Method MakePersonComparable.ajc$interMethod $MakePersonComparable$Person$compareTo:(LPerson;LPerson;)I 5: ireturn }
  66. 65 $ javap -c -private MakePersonComparable Compiled from "MakePersonComparable.aj" public

    class MakePersonComparable extends java.lang.Object{ (...) public static int ajc$interMethod$MakePersonComparable$Person$compareTo(Person, Person); Code: 0: aload_0 1: invokevirtual #44; //Method Person.getName:()Ljava/lang/String; 4: aload_1 5: invokevirtual #44; //Method Person.getName:()Ljava/lang/String; 8: invokevirtual #49; //Method java/lang/String.compareTo:(Ljava/lang/String;)I 11: ireturn (...) }
  67. None
  68. None
  69. 68 AspectJ est puissant et expressif AspectJ est pratique pour

    des modifications statiques AspectJ est à utiliser avec modération ↪ Spring Roo
  70. 69

  71. 70 Règles dynamiques ! tests : injection de fautes traces

    d’exécution suivi de ressources (...)
  72. 71 Event Condition Action “100+ instances ?” “Jetter une exception

    !” new java.io.FileInputStream(...);
  73. 72 Byteman agent rule1.btm rule2.btm rule3.btm JVM void redefineClasses(ClassDefinition... definitions)

    throws ClassNotFoundException, UnmodifiableClassException classes bmsubmit bminstall java -javagent:lib/byteman.jar=...
  74. 73 RULE calling print in HelloWorld CLASS HelloWorld METHOD print(String)

    IF true DO traceln("calling print() with str = " + $1) ENDRULE
  75. 74 java.lang.String ^java.lang.Object ^java.util.List java.util.List Types CLASS INTERFACE CLASS INTERFACE

  76. 75 AT ENTRY EXIT THROW (count) AT, AFTER INVOKE <method>

    (count) SYNCHRONIZE (count) AT, AFTER LINE <number> READ <field> (count) READ $<var> (count) WRITE <field> (count) WRITE $<var> (count) Spécifier une cible
  77. 76 debug(msg) traceln(msg) traceOpen(id, filename) traceln(id, msg) traceClose(id) traceStack() traceStack(id)

    killThread() killJVM() killJVM(code) Waiters Rendez-vous Joiners Countdowns Flags Counters Timers Helpers actions + conditions + les vôtres
  78. 77 (demo) * * en cas de plantage, ne jettez

    pas de tomates, merci.
  79. None
  80. foo.txt BlobStore.java LICENSE COPYING (...) work/index work/e6a92ec2fe5fba022c31c32c97ea455cee4b2736 work/a9a1282b23431aab89c9a4647fd88569eb9a389d work/6a55a0ac5df9e1edce33995a8ac59f17a04a64c9 78

  81. 80 maximum testable humainement ≃

  82. File.renameTo() File.renameTo() Lecture flux + File.exists() + File.delete() Lecture flux

    + File.exists() + File.delete() Lecture flux + corruption GZIP Lecture flux + corruption GZIP 81 Difficile à reproduire ...
  83. 82 @RunWith(BMUnitRunner.class) public class BlobStoreTest { ... }

  84. 83 @BMScript(value = "create_but_fail_to_mkdir", dir = BYTEMAN_SCRIPTS) @Test(expected = BlobStoreException.class)

    public void create_but_fail_to_mkdir() throws IOException { new BlobStore(new File(temporaryFolder.getRoot(), "missing")); } RULE inject a fault into BlobStore to fail on mkdirs() CLASS blob.store.BlobStore METHOD ensureValidWorkingDirectory(File) AFTER INVOKE java.io.File.mkdirs() IF true DO $! = false ENDRULE
  85. RULE create a countdown when entering BlobStore#rewriteIndex() CLASS blob.store.BlobStore METHOD

    rewriteIndex AT ENTRY IF true DO createCountDown("KillIndex", 1) ENDRULE RULE inject a fault into BlobStore#rewriteIndex() to fail appending to the index CLASS com.google.common.io.Files METHOD append AT ENTRY IF countDown("KillIndex") DO throw new java.io.IOException("Simulating a I/O error") ENDRULE private void rewriteIndex() { if (!indexFile.delete()) { throw new BlobStoreException("Could not delete " + indexFile); } try { for (String key : index.keySet()) { append(indexLineFor(key, index.get(key)), indexFile, UTF_8); } } catch (IOException e) { throw new BlobStoreException(e); } } #fail à la 2ème itération
  86. 85 versus

  87. Méconnu Magique Simple Utile ! 86

  88. 87 JooFlux (research work in progress)

  89. 88 invokedynamic (Java 7+) pour les langages dynamiques lambda Java

    8, Nashorn (JS) → talk de Rémi Forax
  90. (...) iconst_1 iconst_2 invokedynamic add(I,I)I + bootstrap (...) fonction typée,

    mais cible résolue à l’exécution renvoie un java.lang.invoke.CallSite via un invokestatic ou autre 88
  91. 90 CallSite method handle chaîne de combinateurs méthode cible cible

    (modifiable) asType() asCollector() dropArguments() guardWithTest() (...) (JIT-friendly)
  92. 91 JooFlux invokeXYZ 㱺 invokedynamic agent JVM agent JMX remplacement

    de méthode rechargement de méthode ajout d’aspect dynamique impact minimal coté JIT
  93. 92 (demo) * * en cas de plantage, ne jettez

    pas de tomates, merci.
  94. 93 Avons-nous démocratisé la magie noire ? dites-le nous !

  95. 94 @jponge @flemouel