Manipulation de bytecode au LavaJUG

Transcript

1. 1 Julien Ponge @jponge Frédéric Le Mouël @flemouel Lava JUG
2. 2 @jponge @flemouel
3. 3 Objectifs JVM, bytecode, JIT Il n’y a pas que le GC dans la vie Golo, langage dynamique Pourquoi, quoi, comment ? Partie 1 Partie 2
4. Manipulation de bytecode : démocratisons la magie noire ! 4 Julien Ponge @jponge Frédéric Le Mouël @flemouel
5. 5 Nous faisons du middleware. dynamique pervasif adaptatif reloading multi-tenancy resource-control dynamic aspects (...) (...)
6. 6 Conteneurs (Java EE, Spring) Android liblayout Play! Framework JRebel Hibernate Langages sur JVM Terracotta FindBugs
7. 7 java.lang.reflect ... parfois il faut “plus”
8. 8 JVM+Bytecode ASM AspectJ Byteman JooFlux ... en fait c’est facile ! écrire et transformer du bytecode faire simplement des modifications statiques courantes avec des aspects aspects dynamiques, application à l’amélioration des couvertures de tests aspects dynamiques via invokedynamic
9. JVM + Bytecode crash course 9
10. 10 “JVM Bytecode for Dummies” — Charles Nutter @headius Clair, complet et didactique !
11. 11 java/lang/String Nom de classe
12. 12 V void Z boolean C char B byte S short I int F float J long D double Types primitifs
13. 13 Ljava/lang/String; Type objet : L;
14. 14 [I [Ljava/lang/String; Tableaux : [
15. 15 (II)I int foo(int, int) (Ljava/lang/String;)V void foo(String)
16. 16 Constant pool (nombres, String, types) Variables locales Machine à pile OpCodes invocation de méthode manipulation de pile get / set champs arithmétique allocation sauts lock (...) 0 java/lang/String 1 “Foo” 2 666 ... ... 0 this (sauf static) 1 arg1 2 arg2 3 int i ... ...
17. 17 public class HelloWorld { public static void main(String[] args) { int i = 1; int j = 2; int k = 100; System.out.println(">>> " + (i + j + k)); } }
18. 18 $ javap -c HelloWorld Compiled from "HelloWorld.java" public class HelloWorld extends java.lang.Object{ public HelloWorld(); Code: 0: aload_0 1: invokespecial #1; //Method java/lang/Object."<init>":()V 4: return (...)
19. 18 $ javap -c HelloWorld Compiled from "HelloWorld.java" public class HelloWorld extends java.lang.Object{ public HelloWorld(); Code: 0: aload_0 1: invokespecial #1; //Method java/lang/Object."<init>":()V 4: return (...) invocation private et constructeur + invoke{static, interface, virtual, dynamic} décompilateur constructeur + <cinit>, <clinit> this dans le constant pool
20. 19 public static void main(java.lang.String[]); Code: 0: iconst_1 1: istore_1 2: iconst_2 3: istore_2 4: bipush 100 6: istore_3 7: getstatic #2; //Field java/lang/System.out:Ljava/io/PrintStream; 10: new #3; //class java/lang/StringBuilder 13: dup 14: invokespecial #4; //Method java/lang/StringBuilder."<init>":()V 17: ldc #5; //String >>> 19: invokevirtual #6; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuil 22: iload_1 23: iload_2 24: iadd 25: iload_3 26: iadd 27: invokevirtual #7; //Method java/lang/StringBuilder.append:(I)Ljava/lang/StringBuilder; 30: invokevirtual #8; //Method java/lang/StringBuilder.toString:()Ljava/lang/String; 33: invokevirtual #9; //Method java/io/PrintStream.println:(Ljava/lang/String;)V 36: return ? ? ?
21. 20 1 iconst_1 0 args 1 2 3 variables
22. 21 istore_1 0 args 1 1 2 3 variables
23. 22 2 iconst_2 0 args 1 1 2 3 variables
24. 23 istore_2 0 args 1 1 2 2 3 variables
25. 24 100 bipush 100 0 args 1 1 2 2 3 variables
26. 25 istore_3 0 args 1 1 2 2 3 100 variables
27. 26 out getstatic System.out 0 args 1 1 2 2 3 100 variables
28. 27 builder out new StringBuilder 0 args 1 1 2 2 3 100 variables
29. 28 builder builder out dup 0 args 1 1 2 2 3 100 variables
30. 29 builder out invokespecial StringBuilder.<init> 0 args 1 1 2 2 3 100 variables
31. 30 “>>> “ builder out ldc “>>> ” 0 args 1 1 2 2 3 100 variables
32. 31 builder out invokevirtual append(String) 0 args 1 1 2 2 3 100 variables
33. 32 1 builder out iload_1 0 args 1 1 2 2 3 100 variables
34. 33 2 1 builder out iload_2 0 args 1 1 2 2 3 100 variables
35. 34 3 builder out iadd 0 args 1 1 2 2 3 100 variables
36. 35 100 3 builder out iload_3 0 args 1 1 2 2 3 100 variables
37. 36 103 builder out iadd 0 args 1 1 2 2 3 100 variables
38. 37 builder out invokevirtual append(int) 0 args 1 1 2 2 3 100 variables
39. 38 “>>> 103” out invokevirtual toString() 0 args 1 1 2 2 3 100 variables
40. 39 invokevirtual println(String) 0 args 1 1 2 2 3 100 variables
41. 40 return 0 args 1 1 2 2 3 100 variables
42. ASM 41
43. 42 package my; public class Main { public static void main(String[] args) { for (int i = 0; i < 10; i++) { System.out.println("Hello Devoxx France!"); } } } Générons ceci sans javac !
44. 43 i: int [start, end] start: i = 0 loop: println(System.out, “Hello Devoxx France!”) incr(i, 1) jump(loop) if (i < 10) end: return goto isn’t harmful!
45. 44 ClassWriter writer = new ClassWriter(COMPUTE_MAXS | COMPUTE_FRAMES); writer.visit(V1_6, ACC_PUBLIC, "my/Main", null, "java/lang/Object", null); writer.visitSource("Main.java", null); MethodVisitor mv = writer.visitMethod(ACC_STATIC | ACC_PUBLIC, "main", "([Ljava/lang/String;)V", null, null); mv.visitCode(); Classe + main(String...)
46. 45 Label start = new Label(); Label loop = new Label(); Label end = new Label(); mv.visitLabel(start); final int COUNTER_VAR = 1; mv.visitLocalVariable("counter", "I", null, start, end, COUNTER_VAR); mv.visitInsn(ICONST_0); mv.visitVarInsn(ISTORE, COUNTER_VAR); i = 0
47. 46 mv.visitLabel(loop); mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;"); mv.visitLdcInsn("Hello Devoxx France!"); mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V"); println()
48. 47 mv.visitIincInsn(COUNTER_VAR, 1); mv.visitVarInsn(ILOAD, COUNTER_VAR); mv.visitLdcInsn(10); mv.visitJumpInsn(IF_ICMPLT, loop); mv.visitLabel(end); mv.visitInsn(RETURN); mv.visitMaxs(666, 666); mv.visitEnd(); writer.visitEnd(); i++, (i < 10)?, goto
49. 48 File outDir = new File("gen/my"); outDir.mkdirs(); FileOutputStream out = new FileOutputStream(new File(outDir, "Main.class")); out.write(writer.toByteArray()); out.close(); .class !
50. 49 (réalisé sans trucages)
51. 50 profiling de chaque méthode utilisée par un programme
52. Manifest-Version: 1.0 Premain-Class: profiler.ProfilerTransformer Can-Redefine-Classes: true Can-Retransform-Classes: true Can-Set-Native-Method-Prefix: false java -javaagent:profiler.jar some.Main
53. 52 public class ProfilerTransformer implements ClassFileTransformer { public static void premain(String agentArgs, Instrumentation inst) { inst.addTransformer(new ProfilerTransformer()); } public byte[] transform(ClassLoader classLoader, String s, Class<?> aClass, ProtectionDomain protectionDomain, byte[] bytes) throws IllegalClassFormatException { ClassReader reader = new ClassReader(bytes); ClassWriter writer = new ClassWriter(reader, COMPUTE_FRAMES | COMPUTE_MAXS); ProfilerClassVisitor profiler = new ProfilerClassVisitor(ASM4, writer); reader.accept(profiler, 0); return writer.toByteArray(); } }
54. 53 @Override public void visitCode() { super.visitCode(); startVarId = newLocal(Type.getType("J")); mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime", "()J"); mv.visitVarInsn(LSTORE, startVarId); }
55. 54 @Override public void visitInsn(int opcode) { if ((opcode >= IRETURN && opcode <= RETURN) || opcode == ATHROW) { (...) mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime", "()J"); mv.visitVarInsn(LLOAD, startVarId); mv.visitInsn(LSUB); mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(J)V"); } super.visitInsn(opcode); }
56. 55 ➟ adapters, inliner, optimizers, ... ➟ modèle objet ➟ analyse statique de code ➟ tracing, retro-engineering, ... org.objectweb.asm.commons org.objectweb.asm.tree org.objectweb.asm.tree.analysis org.objectweb.asm.util
57. 56 “Ok, et pour les cas plus simples ?”
59. 58 WTF Layer WTF Layer WTF Layer Security, validation, logging, transactions, ...
60. 59 public class UserController { // (...) public void update(UserProfile profile) throws ControllerException { validator.check(profile); if (logger.isLevelEnabled(DEBUG)) { logger.debug("Updating: " + profile); } dao.save(profile); redirect("/home"); } // (...) } “cross-cutting concerns” “business logic”
61. 60 Pointcuts Advices Inter-type declarations Où ? Quoi ? + méthodes + attributs + hiérarchie + vérifications + ordre
62. 61 (demo) * * en cas de plantage, ne jettez pas de tomates, merci.
63. 62 aspect ArgsTracer { before(): target(HelloWorld) && call(void run(String...)) { System.out.println("Calling run()"); } } Que se passe t-il avec un before ?
64. 63 --- HelloWorld.bytecode.original 2012-04-11 22:36:22.000000000 +0200 +++ HelloWorld.bytecode.weaved 2012-04-11 22:45:51.000000000 +0200 @@ -19,8 +19,16 @@ 3: dup 4: invokespecial #34; //Method "<init>":()V 7: aload_0 - 8: invokevirtual #35; //Method run:([Ljava/lang/String;)V - 11: return + 8: astore_1 + 9: astore_2 + 10: invokestatic #44; //Method ArgsTracer.aspectOf:()LArgsTracer; + 13: aload_2 + 14: aload_1 + 15: invokevirtual #48; //Method ArgsTracer.ajc$before$ArgsTracer$1$66cd2654:(LHelloWorld; [Ljava/lang/String;)V + 18: aload_2 + 19: aload_1 + 20: invokevirtual #35; //Method run:([Ljava/lang/String;)V + 23: return }
65. 64 aspect MakePersonComparable { declare parents: Person implements Comparable<Person>; public int Person.compareTo(Person other) { return getName().compareTo(other.getName()); } } Que se passe t-il avec des déclarations inter-types ?
66. 65 $ javap -c -private Person Compiled from "Person.java" public class Person extends java.lang.Object implements java.lang.Comparable{ (...) public int compareTo(java.lang.Object); Code: 0: aload_0 1: aload_1 2: checkcast #1; //class Person 5: invokevirtual #50; //Method compareTo:(LPerson;)I 8: ireturn public int compareTo(Person); Code: 0: aload_0 1: aload_1 2: invokestatic #60; //Method MakePersonComparable.ajc$interMethod $MakePersonComparable$Person$compareTo:(LPerson;LPerson;)I 5: ireturn }
67. 66 $ javap -c -private MakePersonComparable Compiled from "MakePersonComparable.aj" public class MakePersonComparable extends java.lang.Object{ (...) public static int ajc$interMethod$MakePersonComparable$Person$compareTo(Person, Person); Code: 0: aload_0 1: invokevirtual #44; //Method Person.getName:()Ljava/lang/String; 4: aload_1 5: invokevirtual #44; //Method Person.getName:()Ljava/lang/String; 8: invokevirtual #49; //Method java/lang/String.compareTo:(Ljava/lang/String;)I 11: ireturn (...) }
70. 69 AspectJ est puissant et expressif AspectJ est pratique pour des modifications statiques AspectJ est à utiliser avec modération ↪ Spring Roo
72. 71 Règles dynamiques ! tests : injection de fautes traces d’exécution suivi de ressources (...)
73. 72 Event Condition Action “100+ instances ?” “Jetter une exception !” new java.io.FileInputStream(...);
74. 73 Byteman agent rule1.btm rule2.btm rule3.btm JVM void redefineClasses(ClassDefinition... definitions) throws ClassNotFoundException, UnmodifiableClassException classes bmsubmit bminstall java -javagent:lib/byteman.jar=...
75. 74 RULE calling print in HelloWorld CLASS HelloWorld METHOD print(String) IF true DO traceln("calling print() with str = " + $1) ENDRULE
76. 75 java.lang.String ^java.lang.Object ^java.util.List java.util.List Types CLASS INTERFACE CLASS INTERFACE
77. 76 AT ENTRY EXIT THROW (count) AT, AFTER INVOKE <method> (count) SYNCHRONIZE (count) AT, AFTER LINE <number> READ <field> (count) READ $<var> (count) WRITE <field> (count) WRITE $<var> (count) Spécifier une cible
78. 77 debug(msg) traceln(msg) traceOpen(id, filename) traceln(id, msg) traceClose(id) traceStack() traceStack(id) killThread() killJVM() killJVM(code) Waiters Rendez-vous Joiners Countdowns Flags Counters Timers Helpers actions + conditions + les vôtres
79. 78 (demo) * * en cas de plantage, ne jettez pas de tomates, merci.
81. foo.txt BlobStore.java LICENSE COPYING (...) work/index work/e6a92ec2fe5fba022c31c32c97ea455cee4b2736 work/a9a1282b23431aab89c9a4647fd88569eb9a389d work/6a55a0ac5df9e1edce33995a8ac59f17a04a64c9
82. 81 maximum testable humainement ≃
83. File.renameTo() File.renameTo() Lecture flux + File.exists() + File.delete() Lecture flux + File.exists() + File.delete() Lecture flux + corruption GZIP Lecture flux + corruption GZIP 82 Difficile à reproduire ...
84. 83 @RunWith(BMUnitRunner.class) public class BlobStoreTest { ... }
85. 84 @BMScript(value = "create_but_fail_to_mkdir", dir = BYTEMAN_SCRIPTS) @Test(expected = BlobStoreException.class) public void create_but_fail_to_mkdir() throws IOException { new BlobStore(new File(temporaryFolder.getRoot(), "missing")); } RULE inject a fault into BlobStore to fail on mkdirs() CLASS blob.store.BlobStore METHOD ensureValidWorkingDirectory(File) AFTER INVOKE java.io.File.mkdirs() IF true DO $! = false ENDRULE
86. RULE create a countdown when entering BlobStore#rewriteIndex() CLASS blob.store.BlobStore METHOD rewriteIndex AT ENTRY IF true DO createCountDown("KillIndex", 1) ENDRULE RULE inject a fault into BlobStore#rewriteIndex() to fail appending to the index CLASS com.google.common.io.Files METHOD append AT ENTRY IF countDown("KillIndex") DO throw new java.io.IOException("Simulating a I/O error") ENDRULE private void rewriteIndex() { if (!indexFile.delete()) { throw new BlobStoreException("Could not delete " + indexFile); } try { for (String key : index.keySet()) { append(indexLineFor(key, index.get(key)), indexFile, UTF_8); } } catch (IOException e) { throw new BlobStoreException(e); } } #fail à la 2ème itération
87. 86 versus
88. Méconnu Magique Simple Utile ! 87
89. 88 JooFlux
90. 89 (demo) * * en cas de plantage, ne jettez pas de tomates, merci.
91. Pas de langage d’aspects Pas de pré/post compilation Un agent Java JVM non-modifiée JooFlux Dév appli Runtime
92. invokedynamic 101
93. invokestatic public static int add(int a, int b) invokevirtual public int add(int a, int b) invokeinterface int add(int a, int b) invokespecial private int add(int a, int b)
94. public Object call(Object receiver, Object[] args) throws Throwable { return CallSiteArray.defaultCall(this, receiver, args); } public Object call(Object receiver) throws Throwable { return call(receiver, CallSiteArray.NOPARAM); } public Object call(Object receiver, Object arg1) throws Throwable { return call(receiver, ArrayUtil.createArray(arg1)); } public Object call(Object receiver, Object arg1, Object arg2) throws Throwable { return call(receiver, ArrayUtil.createArray(arg1, arg2)); } public Object call(Object receiver, Object arg1, Object arg2, Object arg3) throws Throwable { return call(receiver, ArrayUtil.createArray(arg1, arg2, arg3)); } public Object call(Object receiver, Object arg1, Object arg2, Object arg3, Object arg4) throws Throwable { return call(receiver, ArrayUtil.createArray(arg1, arg2, arg3, arg4)); } “Tous les chemins mènent à Rome”
95. public static Object invoke(Object object, String methodName, Object[] parameters) { try { Class[] classTypes = new Class[parameters.length]; for (int i = 0; i < classTypes.length; i++) { classTypes[i] = parameters[i].getClass(); } Method method = object.getClass().getMethod(methodName, classTypes); return method.invoke(object, parameters); } catch (Throwable t) { return InvokerHelper.invokeMethod(object, methodName, parameters); } } Difficile pour le JIT ! CallSites génériques Réflexivité
96. invokedynamic nom symbolique + signature + bootstrap CallSite Constant Mutable Volatile MethodHandle direct ou combinateurs branchements adaptation filtres (...) 7
98. Implémentation
99. 1 2 3 4 Chargement des classes Interception Management Runtime
100. (...) ldc #3 ldc #5 invokedynamic foo (II)I bootstraper (...) Call site JMX agent JooFlux bootstraper Modified bytecode Registry Target method combinators Load time action Run time action Original bytecode JVM agent 1 2 3 4
101. Réécriture de bytecode ASM + Agent Bootstrap JooFlux Casse la vérification de bytecode ! 1
102. sémantique champ volatile impact mesuré négligeable VolatileCallSite 2
103. invokestatic invokespecial Method handle direct invokevirtual invokeinterface Method handle sur le premier receveur 3
104. invokeinterface Polymorphic inline-cache guardWithTest(Class1, obj) guardWithTest(Class2, obj) fallback(callsite, args[]) Class1#foo(II)I Class2#foo(II)I obj.foo(1, 2) 3
105. 4 public interface JooFluxManagementMXBean { public String getName(); public int getNumberOfRegisteredCallSites(); public Set<String> getRegisteredCallSiteKeys(); public String getCallSiteType(String target); public void changeCallSiteTarget(String methodType, String oldTarget, String newTarget); public void applyBeforeAspect(String callSitesKey, String aspectClass, String aspectMethod); public void applyAfterAspect(String callSitesKey, String aspectClass, String aspectMethod); }
106. (...) ldc #3 ldc #5 invokedynamic foo (II)I bootstraper (...) Call site JMX agent JooFlux bootstraper Modified bytecode Registry Target method combinators Load time action Run time action Original bytecode JVM agent 4 Registre : 0 impact perfs Enregistré à la 1ère interception Modification via API
107. Évaluation
108. Interception / redirection de méthode Injection d’aspect ‘vide’ Langages dynamiques Plates-formes AOP Micro Fibonacci(40) Macro SCImark Fork/Join Clojure Benchmarks X V V V
109. 0 2 4 6 8 Byteman Jooflux Avant Après Avant + Après Injection d’aspect ‘vide’ x 137 x 134 x 306 x 6 x 2 x 5 Microbench
110. Interception/redirection méthode 0 5 10 15 Clojure JRuby Groovy Rhino JS Jython Java+JooFlux Min Max x 1 x 48 x 18 x 15 x 2,2 x 10 x 3 x 6 x 1,2 x 2 Microbench
111. Interception/redirection méthode 0.9 1 1.1 SCIMark Fork/Join Clojure Min Max x 0,99 x 0,97 x 1,04 x 0,93 x 1,01 x 1,01 Macrobench
112. Conclusion
113. JVM généraliste Interception à grain fin Résultats initiaux encourageants Approche nouvelle, générique, sans langage dédié Bilan
114. Injection atomique multi-aspects Rollback / versioning Vérification formelle pré-injection Aspects contextuels / Internet of Things Contrôle de ressources avec isolation adaptative Perspectives
115. 114 https://github.com/dynamid/jooflux Questions / réponses Julien Ponge Frédéric Le Mouël http://dynamid.citi-lab.fr/