Jabel — как я потрошил компилятор Java

JDK 8 уже 5 лет, а многие проекты до сих пор используют эту версию байткода как целевую. Вот только сам язык Java развивается, появляются новые вкусные возможности синтаксиса (такие как "var", switch expressions, multi-line strings, улучшения для diamond оператора и абстрактных классов), а компилятор их позволяет использовать только с JDK той же версии.

Правда, если посмотреть на байткод, который генерируется для этих "фичей", он ничем не отличается от Java 8, что как бы намекает, что они являются чистым синтаксическим сахаром (в отличие от лямбд в Java 8, например). Так почему javac не позволяет их использовать для более старых целевых версий?

Этот вопрос мучал меня многие годы, и я решил полезть в код javac (а ведь он тоже на Java) и выяснить. И вы не поверите что я там нашёл! :)

Данный доклад расскажет, как устроены некоторые части javac, как работает добавление новых фичей, механизм plugin-ов, а так же что (и как) делает Jabel, чтобы позволить использовать синтаксис последних версий языка Java, при этом не нарушая Java 8 совместимость.

Transcript

1. Jabel – retrofitting Java Compiler by instrumenting it! Sergei Egorov @bsideup
2. About me • Staff Engineer at Pivotal’s Spring R&D, working on Project Reactor ⚛ • Java Champion • Testcontainers co-maintainer • Berlin Spring User Group co-organizer • Developer tools geek @bsideup
3. How compilers work?
4. Very easy! @bsideup
5. Source code @bsideup
6. Source code Scanner @bsideup
7. Source code Scanner Parser @bsideup
8. Source code Scanner Parser Tokens @bsideup
9. Source code Scanner Parser Semantic analyzer Tokens @bsideup
10. Source code Scanner Parser Semantic analyzer Tokens CST
 (concrete syntax tree) @bsideup
11. Front End Source code Scanner Parser Semantic analyzer Tokens CST
 (concrete syntax tree) @bsideup
12. Front End Source code Scanner Parser Semantic analyzer Optimizer Tokens CST
 (concrete syntax tree) @bsideup
13. Front End Source code Scanner Parser Semantic analyzer Optimizer Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
14. Middle End Front End Source code Scanner Parser Semantic analyzer Optimizer Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
15. Middle End Front End Source code Scanner Parser Semantic analyzer Optimizer code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
16. Middle End Back End Front End Source code Scanner Parser Semantic analyzer Optimizer code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
17. Middle End Back End Front End Source code Scanner Parser Semantic analyzer Optimizer Intermediate language code code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
18. @bsideup
19. @bsideup
20. Middle End Back End Front End Source code Scanner Parser Semantic analyzer Optimizer Intermediate language code code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup ?
21. @bsideup
22. Transpiller @bsideup
23. “Transpilers, or source-to-source compilers, are tools that read source code written in one programming language, and produce the equivalent code in another language that has a similar level of abstraction.” @bsideup
24. What about Lombok?
25. Project Lombok @Value public class ValueExample { String name; @Wither(AccessLevel.PACKAGE) @NonFinal int age; double score; protected String[] tags; @ToString(includeFieldNames = true) @Value(staticConstructor = “of") public static class Exercise<T> { String name; T value; } } @bsideup
26. Project Lombok @Value public class ValueExample { String name; @Wither(AccessLevel.PACKAGE) @NonFinal int age; double score; protected String[] tags; @ToString(includeFieldNames = true) @Value(staticConstructor = “of") public static class Exercise<T> { String name; T value; } } public final class ValueExample { private final String name; private int age; private final double score; protected final String[] tags; @java.beans.ConstructorProperties({"name", "age", "score", public ValueExample(String name, int age, double score, Str this.name = name; this.age = age; this.score = score; this.tags = tags; } public String getName() { return this.name; } public int getAge() { return this.age; } @bsideup
27. } public double getScore() { return this.score; } public String[] getTags() { return this.tags; } @java.lang.Override public boolean equals(Object o) { if (o == this) return true; if (!(o instanceof ValueExample)) return false; final ValueExample other = (ValueExample)o; final Object this$name = this.getName(); final Object other$name = other.getName(); if (this$name == null ? other$name != null : !this$name if (this.getAge() != other.getAge()) return false; if (Double.compare(this.getScore(), other.getScore()) ! if (!Arrays.deepEquals(this.getTags(), other.getTags()) return true; } @java.lang.Override public int hashCode() { final int PRIME = 59; int result = 1; final Object $name = this.getName(); result = result * PRIME + ($name == null ? 43 : $name.h result = result * PRIME + this.getAge(); final long $score = Double.doubleToLongBits(this.getSco result = result * PRIME + (int)($score >>> 32 ^ $score) result = result * PRIME + Arrays.deepHashCode(this.getT return result; } Project Lombok @bsideup @Value public class ValueExample { String name; @Wither(AccessLevel.PACKAGE) @NonFinal int age; double score; protected String[] tags; @ToString(includeFieldNames = true) @Value(staticConstructor = “of") public static class Exercise<T> { String name; T value; } } o_o
28. int result = 1; final Object $name = this.getName(); result = result * PRIME + ($name == null ? 43 : $name.h result = result * PRIME + this.getAge(); final long $score = Double.doubleToLongBits(this.getSco result = result * PRIME + (int)($score >>> 32 ^ $score) result = result * PRIME + Arrays.deepHashCode(this.getT return result; } @java.lang.Override public String toString() { return "ValueExample(name=" + getName() + ", age=" + ge Arrays.deepToString(getTags()) + ")"; } ValueExample withAge(int age) { return this.age == age ? this : new ValueExample(name, } public static final class Exercise<T> { private final String name; private final T value; private Exercise(String name, T value) { this.name = name; this.value = value; } public static <T> Exercise<T> of(String name, T value) return new Exercise<T>(name, value); } public String getName() { return this.name; } public T getValue() { Project Lombok @bsideup @Value public class ValueExample { String name; @Wither(AccessLevel.PACKAGE) @NonFinal int age; double score; protected String[] tags; @ToString(includeFieldNames = true) @Value(staticConstructor = “of") public static class Exercise<T> { String name; T value; } } o_O
29. } public T getValue() { return this.value; } @java.lang.Override public boolean equals(Object o) { if (o == this) return true; if (!(o instanceof ValueExample.Exercise)) return f final Exercise<?> other = (Exercise<?>)o; final Object this$name = this.getName(); final Object other$name = other.getName(); if (this$name == null ? other$name != null : !this$ final Object this$value = this.getValue(); final Object other$value = other.getValue(); if (this$value == null ? other$value != null : !thi return true; } @java.lang.Override public int hashCode() { final int PRIME = 59; int result = 1; final Object $name = this.getName(); result = result * PRIME + ($name == null ? 43 : $na final Object $value = this.getValue(); result = result * PRIME + ($value == null ? 43 : $v return result; } @java.lang.Override public String toString() { return "ValueExample.Exercise(name=" + getName() + } } } Project Lombok @bsideup @Value public class ValueExample { String name; @Wither(AccessLevel.PACKAGE) @NonFinal int age; double score; protected String[] tags; @ToString(includeFieldNames = true) @Value(staticConstructor = “of") public static class Exercise<T> { String name; T value; } } O_O
30. Is it a transpiler?
31. It could be, but…
32. Middle End Back End Front End Source code Scanner Parser Semantic analyzer Optimizer Intermediate language code code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
33. Middle End Back End Front End Source code Scanner Parser Semantic analyzer Optimizer Intermediate language code code generator Tokens CST
 (concrete syntax tree) AST
 (abstract syntax tree) @bsideup
34. The Hitchhiker's Guide to javac
35. @bsideup https://openjdk.java.net/groups/compiler/doc/hhgtjavac/index.html
36. @bsideup
37. @bsideup
38. @bsideup
39. @bsideup
40. Don’t confuse with…
41. @bsideup
42. Middle End Optimizer @bsideup
43. Why?
44. @bsideup
47. Lower example var result = switch (args.length) { case 1 -> { yield "one"; } case 2, 3 -> "two or three"; default -> "idk"; }; @bsideup
48. Lower example var result = switch (args.length) { case 1 -> { yield "one"; } case 2, 3 -> "two or three"; default -> "idk"; }; String innerPublic = null; switch (args.length) { case 1: { innerPublic = "one"; break; } case 2: case 3: { innerPublic = "two or three"; break; } default: { innerPublic = "idk"; break; } } final String result = innerPublic; @bsideup
49. What can it “lower”? • Switch expressions • “var” • Try-with-resources • Pattern matching in “instanceof” • Even records!* @bsideup
50. @bsideup
51. How it decides when to lower?
52. It doesn’t!
53. But…
54. /** * Models a feature of the Java programming language. Each feature can be associated with a * minimum source level, a maximum source level and a diagnostic fragment describing the feature, * which is used to generate error messages of the kind {@code feature XYZ not supported in source N}. */ public enum Feature { DIAMOND(JDK7, Fragments.FeatureDiamond, DiagKind.NORMAL), MODULES(JDK9, Fragments.FeatureModules, DiagKind.PLURAL), EFFECTIVELY_FINAL_VARIABLES_IN_TRY_WITH_RESOURCES(JDK9, Fragments.FeatureVarInTryWithResources, DiagKind.PLURAL), DEPRECATION_ON_IMPORT(MIN, JDK8), POLY(JDK8), LAMBDA(JDK8, Fragments.FeatureLambda, DiagKind.PLURAL), METHOD_REFERENCES(JDK8, Fragments.FeatureMethodReferences, DiagKind.PLURAL), DEFAULT_METHODS(JDK8, Fragments.FeatureDefaultMethods, DiagKind.PLURAL), STATIC_INTERFACE_METHODS(JDK8, Fragments.FeatureStaticIntfMethods, DiagKind.PLURAL), STATIC_INTERFACE_METHODS_INVOKE(JDK8, Fragments.FeatureStaticIntfMethodInvoke, DiagKind.PLURAL), STRICT_METHOD_CLASH_CHECK(JDK8), EFFECTIVELY_FINAL_IN_INNER_CLASSES(JDK8), TYPE_ANNOTATIONS(JDK8, Fragments.FeatureTypeAnnotations, DiagKind.PLURAL), ANNOTATIONS_AFTER_TYPE_PARAMS(JDK8, Fragments.FeatureAnnotationsAfterTypeParams, DiagKind.PLURAL), REPEATED_ANNOTATIONS(JDK8, Fragments.FeatureRepeatableAnnotations, DiagKind.PLURAL), INTERSECTION_TYPES_IN_CAST(JDK8, Fragments.FeatureIntersectionTypesInCast, DiagKind.PLURAL), GRAPH_INFERENCE(JDK8), FUNCTIONAL_INTERFACE_MOST_SPECIFIC(JDK8), POST_APPLICABILITY_VARARGS_ACCESS_CHECK(JDK8), MAP_CAPTURES_TO_BOUNDS(MIN, JDK7), PRIVATE_SAFE_VARARGS(JDK9), DIAMOND_WITH_ANONYMOUS_CLASS_CREATION(JDK9, Fragments.FeatureDiamondAndAnonClass, DiagKind.NORMAL), UNDERSCORE_IDENTIFIER(MIN, JDK8), PRIVATE_INTERFACE_METHODS(JDK9, Fragments.FeaturePrivateIntfMethods, DiagKind.PLURAL), LOCAL_VARIABLE_TYPE_INFERENCE(JDK10), VAR_SYNTAX_IMPLICIT_LAMBDAS(JDK11, Fragments.FeatureVarSyntaxInImplicitLambda, DiagKind.PLURAL), IMPORT_ON_DEMAND_OBSERVABLE_PACKAGES(JDK1_2, JDK8), SWITCH_MULTIPLE_CASE_LABELS(JDK14, Fragments.FeatureMultipleCaseLabels, DiagKind.PLURAL), SWITCH_RULE(JDK14, Fragments.FeatureSwitchRules, DiagKind.PLURAL), SWITCH_EXPRESSION(JDK14, Fragments.FeatureSwitchExpressions, DiagKind.PLURAL), TEXT_BLOCKS(JDK14, Fragments.FeatureTextBlocks, DiagKind.PLURAL), PATTERN_MATCHING_IN_INSTANCEOF(JDK14, Fragments.FeaturePatternMatchingInstanceof, DiagKind.NORMAL), REIFIABLE_TYPES_INSTANCEOF(JDK14, Fragments.FeatureReifiableTypesInstanceof, DiagKind.PLURAL), RECORDS(JDK14, Fragments.FeatureRecords, DiagKind.PLURAL), ;
55. PRIVATE_SAFE_VARARGS(JDK9), DIAMOND_WITH_ANONYMOUS_CLASS_CREATION(JDK9, Fragments.FeatureDiamondAndAnonClass, DiagKind.NORMAL), UNDERSCORE_IDENTIFIER(MIN, JDK8), PRIVATE_INTERFACE_METHODS(JDK9, Fragments.FeaturePrivateIntfMethods, DiagKind.PLURAL), LOCAL_VARIABLE_TYPE_INFERENCE(JDK10), VAR_SYNTAX_IMPLICIT_LAMBDAS(JDK11, Fragments.FeatureVarSyntaxInImplicitLambda, DiagKind.PLURAL), IMPORT_ON_DEMAND_OBSERVABLE_PACKAGES(JDK1_2, JDK8), SWITCH_MULTIPLE_CASE_LABELS(JDK14, Fragments.FeatureMultipleCaseLabels, DiagKind.PLURAL), SWITCH_RULE(JDK14, Fragments.FeatureSwitchRules, DiagKind.PLURAL), SWITCH_EXPRESSION(JDK14, Fragments.FeatureSwitchExpressions, DiagKind.PLURAL), TEXT_BLOCKS(JDK14, Fragments.FeatureTextBlocks, DiagKind.PLURAL), PATTERN_MATCHING_IN_INSTANCEOF(JDK14, Fragments.FeaturePatternMatchingInstanceof, DiagKind.NORMAL), REIFIABLE_TYPES_INSTANCEOF(JDK14, Fragments.FeatureReifiableTypesInstanceof, DiagKind.PLURAL), RECORDS(JDK14, Fragments.FeatureRecords, DiagKind.PLURAL),
56. PRIVATE_SAFE_VARARGS(JDK9), DIAMOND_WITH_ANONYMOUS_CLASS_CREATION(JDK9, Fragments.FeatureDiamondAndAnonClass, DiagKind.NORMAL), UNDERSCORE_IDENTIFIER(MIN, JDK8), PRIVATE_INTERFACE_METHODS(JDK9, Fragments.FeaturePrivateIntfMethods, DiagKind.PLURAL), LOCAL_VARIABLE_TYPE_INFERENCE(JDK10), VAR_SYNTAX_IMPLICIT_LAMBDAS(JDK11, Fragments.FeatureVarSyntaxInImplicitLambda, DiagKind.PLURAL), IMPORT_ON_DEMAND_OBSERVABLE_PACKAGES(JDK1_2, JDK8), SWITCH_MULTIPLE_CASE_LABELS(JDK14, Fragments.FeatureMultipleCaseLabels, DiagKind.PLURAL), SWITCH_RULE(JDK14, Fragments.FeatureSwitchRules, DiagKind.PLURAL), SWITCH_EXPRESSION(JDK14, Fragments.FeatureSwitchExpressions, DiagKind.PLURAL), TEXT_BLOCKS(JDK14, Fragments.FeatureTextBlocks, DiagKind.PLURAL), PATTERN_MATCHING_IN_INSTANCEOF(JDK14, Fragments.FeaturePatternMatchingInstanceof, DiagKind.NORMAL), REIFIABLE_TYPES_INSTANCEOF(JDK14, Fragments.FeatureReifiableTypesInstanceof, DiagKind.PLURAL), RECORDS(JDK14, Fragments.FeatureRecords, DiagKind.PLURAL),
57. PRIVATE_SAFE_VARARGS(JDK9), DIAMOND_WITH_ANONYMOUS_CLASS_CREATION(JDK9, Fragments.FeatureDiamondAndAnonClass, DiagKind.NORMAL), UNDERSCORE_IDENTIFIER(MIN, JDK8), PRIVATE_INTERFACE_METHODS(JDK9, Fragments.FeaturePrivateIntfMethods, DiagKind.PLURAL), LOCAL_VARIABLE_TYPE_INFERENCE(JDK10), VAR_SYNTAX_IMPLICIT_LAMBDAS(JDK11, Fragments.FeatureVarSyntaxInImplicitLambda, DiagKind.PLURAL), IMPORT_ON_DEMAND_OBSERVABLE_PACKAGES(JDK1_2, JDK8), SWITCH_MULTIPLE_CASE_LABELS(JDK14, Fragments.FeatureMultipleCaseLabels, DiagKind.PLURAL), SWITCH_RULE(JDK14, Fragments.FeatureSwitchRules, DiagKind.PLURAL), SWITCH_EXPRESSION(JDK14, Fragments.FeatureSwitchExpressions, DiagKind.PLURAL), TEXT_BLOCKS(JDK14, Fragments.FeatureTextBlocks, DiagKind.PLURAL), PATTERN_MATCHING_IN_INSTANCEOF(JDK14, Fragments.FeaturePatternMatchingInstanceof, DiagKind.NORMAL), REIFIABLE_TYPES_INSTANCEOF(JDK14, Fragments.FeatureReifiableTypesInstanceof, DiagKind.PLURAL), RECORDS(JDK14, Fragments.FeatureRecords, DiagKind.PLURAL), What if we…
58. PRIVATE_SAFE_VARARGS(JDK8), DIAMOND_WITH_ANONYMOUS_CLASS_CREATION(JDK8, Fragments.FeatureDiamondAndAnonClass, DiagKind.NORMAL), UNDERSCORE_IDENTIFIER(MIN, JDK8), PRIVATE_INTERFACE_METHODS(JDK9, Fragments.FeaturePrivateIntfMethods, DiagKind.PLURAL), LOCAL_VARIABLE_TYPE_INFERENCE(JDK8), VAR_SYNTAX_IMPLICIT_LAMBDAS(JDK8, Fragments.FeatureVarSyntaxInImplicitLambda, DiagKind.PLURAL), IMPORT_ON_DEMAND_OBSERVABLE_PACKAGES(JDK1_2, JDK8), SWITCH_MULTIPLE_CASE_LABELS(JDK8, Fragments.FeatureMultipleCaseLabels, DiagKind.PLURAL), SWITCH_RULE(JDK8, Fragments.FeatureSwitchRules, DiagKind.PLURAL), SWITCH_EXPRESSION(JDK8, Fragments.FeatureSwitchExpressions, DiagKind.PLURAL), TEXT_BLOCKS(JDK8, Fragments.FeatureTextBlocks, DiagKind.PLURAL), PATTERN_MATCHING_IN_INSTANCEOF(JDK8, Fragments.FeaturePatternMatchingInstanceof, DiagKind.NORMAL), REIFIABLE_TYPES_INSTANCEOF(JDK8, Fragments.FeatureReifiableTypesInstanceof, DiagKind.PLURAL), RECORDS(JDK8, Fragments.FeatureRecords, DiagKind.PLURAL),
59. How?
60. let’s instrument it!
61. void checkSourceLevel(Feature feature) { checkSourceLevel(token.pos, feature); } @bsideup
62. void checkSourceLevel(Feature feature) { if (feature.allowedInSource(Source.JDK8)) { feature = Source.Feature.LAMBDA; } checkSourceLevel(token.pos, feature); } @bsideup
63. Field field = Source.Feature.class.getDeclaredField("minLevel"); field.setAccessible(true); for (Source.Feature feature : ENABLED_FEATURES) { field.set(feature, Source.JDK8); if (!feature.allowedInSource(Source.JDK8)) { throw new IllegalStateException(feature.name() + " minLevel instrumentation failed!"); } } @bsideup
64. Field field = Source.Feature.class.getDeclaredField("minLevel"); field.setAccessible(true); for (Source.Feature feature : ENABLED_FEATURES) { field.set(feature, Source.JDK8); if (!feature.allowedInSource(Source.JDK8)) { throw new IllegalStateException(feature.name() + " minLevel instrumentation failed!"); } } @bsideup
65. Set<Source.Feature> ENABLED_FEATURES = Stream .of( "PRIVATE_SAFE_VARARGS", "SWITCH_EXPRESSION", "SWITCH_RULE", "SWITCH_MULTIPLE_CASE_LABELS", "LOCAL_VARIABLE_TYPE_INFERENCE", "VAR_SYNTAX_IMPLICIT_LAMBDAS", "DIAMOND_WITH_ANONYMOUS_CLASS_CREATION", "EFFECTIVELY_FINAL_VARIABLES_IN_TRY_WITH_RESOURCES", "TEXT_BLOCKS", "PATTERN_MATCHING_IN_INSTANCEOF", "REIFIABLE_TYPES_INSTANCEOF", "RECORDS" ) .map(name -> { try { return Source.Feature.valueOf(name); } catch (IllegalArgumentException e) { return null; } }) .filter(Objects::nonNull) .collect(Collectors.toSet()); @bsideup
66. Demo
67. Takeaways • http://github.com/bsideup/jabel • Easy to integrate • Reuses compiler’s code (except for records) • Pretty safe to use
 @bsideup
68. @bsideup bsideup