Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Escaping The Jar Hell With Jigsaw Layers

Escaping The Jar Hell With Jigsaw Layers

Jigsaw modules are often criticized for the lack of versioning that exists in alternative module systems for Java, such as OSGi. One of the main purposes of versioning is to solve the so called Jar Hell problem that arises when your application depends on two different versions of the same library. While Jigsaw is able to detect this conflicting situation, it won’t allow you to load both versions that are placed on the module path. However, it is not always possible to eliminate all version conflicts from your application, because other versions of the same libraries can come to your application indirectly via dependencies that you do not control. Fortunately, there is a native solution for the problem in Jigsaw called Layers. In this session, I will show what problems Jigsaw could have introduced if it had explicit versions for modules, and how Jigsaw Layers in conjunction with Jigsaw Services help to solve the Jar Hell problem safely.

Nikita Lipsky

November 24, 2018
Tweet

More Decks by Nikita Lipsky

Other Decks in Programming

Transcript

  1. Excelsior JET? •  AOT-centric Java SE implementation –  certified as

    Java Compatible since 2005 •  AOT compiler + Java Runtime –  mixed compilation: AOT + JIT –  AOT support for custom classloaders (Eclipse RCP, Tomcat, Spring Boot) •  Toolkit –  Startup Optimizer –  Deployment 3
  2. Nikita Lipsky •  20+ years in software development •  Excelsior

    JET project initiator –  18+ years of contributions –  compiler engineer –  product department lead –  etc. •  Twitter: @pjBooms •  Team blog: https://www.excelsiorjet.com/blog 4
  3. Agenda •  Very short introduction to Jigsaw •  Jar Hell

    problem •  Versioning as a solution to Jar Hell problem •  Problems with versioning •  Jigsaw Layers •  Jar Hell resolution using Jigsaw Layers •  Demo 5
  4. Module example // src/java.sql/module-info.java module java.sql { requires transitive java.logging;

    requires transitive java.xml; exports java.sql; exports javax.sql; exports javax.transaction.xa; uses java.sql.Driver; } 7
  5. Java Platform Module System Jigsaw module •  imports: –  modules:

    requires –  services: uses •  exports: –  packages: exports –  servicies: provides SomeService with ServiceImpl 8
  6. Java Platform Module System Module A Class A Module B

    Class B Module C Class C exports packageC; requires: A exports packageA; I requires A; requires C; 9
  7. Versioning •  Jigsaw module might have a version in the

    first drafts of Jigsaw to solve Jar Hell problem – import/export was qualified by version – If two modules require a library module of two different versions, both versions will be loaded 17
  8. Versioning Question: How to implement versioning? Task: For given A

    module importing Lib (v1) library, and B module importing Lib (v2), it is required that both versions of Lib working without conflicts. Solution: load the versions of the library by different classloaders. 20
  9. Versioning Question: How to implement versioning? Task: For a given

    module A importing library Lib (v1), and module B importing Lib (v2), it is required that both versions of Lib work without conflicts. Solution: load the versions of the library by different classloaders. 21
  10. Versioning Question: How to implement versioning? Task: For a given

    module A importing library Lib (v1), and module B importing Lib (v2), it is required that both versions of Lib work without conflicts. Solution: load versions of the library by different classloaders. 22
  11. Versioning •  ClassLoader forms a unique class namespace •  If

    each module is loaded by its own classloader then there will be no conflicts with classes of other modules 23
  12. Versioning com.foo App 1.0 com.foo parse-api 2.0 com.foo persist-api 3.0

    org.apache commons 2.1 org.apache commons 3.1 CL1 CL2 CL3 CL4 CL5 Different versions of apache commons may simultaneously present in the JVM 24
  13. Versioning Versioning immediately means: 1 module ßà 1 classloader It

    was exactly so in the first Jigsaw drafts! 27
  14. Jigsaw and classloaders Jigsaw is not just modules, but also

    the Java SE platform split into modules. 28
  15. Jigsaw and classloaders Backward compatibility problems: According to the specification

    getClassloader() == null for core platform classes.то противоречит That precludes splitting of the platform into modules, with each module loaded by its loader 29
  16. Jigsaw and classloaders Backward compatibility problems: According to the specification

    getClassloader() == null for core platform classes.то противоречит It precludes splitting of the platform into modules, with each module loaded by its loader 30
  17. Jigsaw and classloaders Backward compatibility problems: Many classloaders for an

    application may also cause problems: – classloaders delegation Ф 31
  18. Jigsaw and classloaders Backward compatibility problems: Even the fact that

    Application (system) Classloader does not extend URLClassloader anymore causes problems Ф 32
  19. 33

  20. Versioning? App 1.0 Foo 2.0 Bar 3.0 Baz 2.1 Baz

    3.1 A No, I’m “A”! A I’m “A”! 42
  21. Loading constraints •  Loading constraints prohibit two classes with the

    same fully qualified name to appear in the namespace of another class 45
  22. Loading constraints B.java: class B { T1 f1 = A.f;

    int f2 = A.m(t2); } A.java: class A { static T1 f; static int m(T2 t) If B is loaded by L1 classloader and A is loaded by L2 then the JVM will check that (T1, L1) = (T1, L2) and (T2, L1) = (T2, L2) == 46
  23. Versioning Another detail: import in early Jigsaw versions was qualified

    by not a single version but by а version range: –  A module may declare that it can work with a dependency of “from” version to “to” version 49
  24. Versioning Problem 3: Resolving dependencies (wiring modules) from version ranges

    is an NP- complete problem! –  Reduced to 3-SAT 51
  25. Versioning … after that versioning in JPMS breathed its last.

    No versioning – no classloaders for modules. 53
  26. Jar Hell But how to solve the Jar Hell now?

    App 1.0 Foo 2.0 Bar 3.0 Baz 2.1 Baz 3.1 54
  27. Jar Hell Stop! Jigsaw does not have versions! App 1.0

    Foo 2.0 Bar 3.0 Baz 2.1 Baz 3.1 56
  28. Jar Hell Stop! Jigsaw does not have versions! The most

    probably two different versions of the same library have same packages. 57
  29. Jar Hell Stop! Jigsaw does not have versions! The most

    probably two different versions of the same library have same packages. Jigsaw prohibits for two modules to have same packages (split packages) 58
  30. Jar Hell But how to solve the Jar Hell problem

    after all? App 1.0 Foo 2.0 Bar 3.0 Baz 2.1 Baz 3.1 59
  31. 62

  32. Jar Hell Another solution: relocate classes of two versions of

    a library to different Java packages (Maven Shade plugin) 64
  33. Jar Hell Another solution: relocate classes of two versions of

    a library to different Java packages (Maven Shade plugin) It may not work because of reflection. 65
  34. Jar Hell Third solution: employ classloaders Be ready to fight

    with classloaders constraints violation problem 67
  35. Jigsaw Layers * •  Local module system •  Split packages

    are prohibited inside one layer •  Two modules containing the same package have to belong to different layers 70
  36. Jigsaw Layers * * The picture is from Alex Buckley’s

    presentation: Project Jigsaw Under the hood 71
  37. Jigsaw Layers * •  Jigsaw layers are implemented …через загрузчики

    классов. •  У каждого слоя по крайней мере один свой загрузчик (может быть несколько) •  Но при этом невозможно по построению нарушение classloader constraints! 72
  38. Jigsaw Layers * •  Jigsaw layers are implemented via classloaders!ерез

    загрузчики классов. •  У каждого слоя по крайней мере один свой загрузчик (может быть несколько) •  Но при этом невозможно по построению нарушение classloader constraints! 73
  39. Jigsaw Layers * •  Jigsaw layers are implemented via classloaders!

    •  Each layer has at least one unique classloader (may have several classloaders) 74
  40. Jigsaw Layers * •  Jigsaw layers are implemented via classloaders!

    •  Each layer has at least one unique classloader (may have several classloaders) •  However, classloader constraints are impossible to violate by construction! 75
  41. Jigsaw Layers Jar Hell Resolution App 1.0 Library Foo 2.0

    Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 76
  42. Jigsaw Layers Jar Hell Resolution App 1.0 Library Foo 2.0

    Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Boot Layer 77
  43. Jigsaw Layers Jar Hell Resolution App 1.0 Library Foo 2.0

    Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer 78
  44. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer 79
  45. Jigsaw Layers Jar Hell Resolution App 1.0 Library Foo 2.0

    Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 80
  46. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer 81
  47. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer 82
  48. Jigsaw Layers •  Every layer has a parent layer • 

    Modules of a child layer can import modules of parent layers •  However parent layer modules cannot import child layers modules! •  That’s why it’s impossible to violate classloaders constraints! 83
  49. Jigsaw Layers •  Every layer has a parent layer • 

    Modules of a child layer can import modules of parent layers •  However parent layer modules cannot import child layers modules! •  That’s why it’s impossible to violate classloaders constraints! 84
  50. Jigsaw Layers •  Every layer has a parent layer • 

    Modules of a child layer can import modules of parent layers •  However, parent layer modules cannot import child layers modules! •  That’s why it’s impossible to violate classloaders constraints! 85
  51. Jigsaw Layers •  Every layer has a parent layer • 

    Modules of a child layer can import modules of parent layers •  However, parent layer modules cannot import child layers modules! •  That’s why it’s impossible to violate classloaders constraints! 86
  52. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer A A 87
  53. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 A A Boot Layer 88
  54. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 A A Boot Layer 89
  55. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 A A Boot Layer There is a space between us A place that I can’t go 90
  56. Jigsaw Layers Question: if application modules cannot import child layers

    modules then how they can employ functionality implemented in these child layers? 91
  57. 92

  58. Jigsaw Layers Question: if application modules cannot import child layers

    modules then how they can employ functionality implemented in these child layers? 93
  59. Jigsaw Layers Question: if application modules cannot import child layers

    modules then how they can employ functionality implemented in these child layers? Answer: employ IoC design pattern via Jigsaw Services: child layers can provide services to modules of parent layers 94
  60. Jigsaw Layers Question: if application modules cannot import child layers

    modules then how they can employ functionality implemented in these child layers? Answer: employ IoC design pattern via Jigsaw Services: child layers can provide services to modules of parent layers 95
  61. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Library

    Foo 2.0 Library Bar 3.0 Library Baz 2.1 Library Baz 3.1 Layer 2 Boot Layer 96
  62. Layer 3 Jigsaw Layers Jar Hell Resolution App 1.0 Foo

    2.0 Bar 3.0 Baz 2.1 Baz 3.1 Layer 2 App Service Provider 1 App Service Provider 2 Boot Layer 97
  63. 98

  64. How to use var finder = ModuleFinder.of(Paths.get(“MyLayer”)); var parent =

    ModuleLayer.boot(); Configuration cf = parent.configuration().resolve( finder, ModuleFinder.of(), Set.of(“myRootMod")); ModuleLayer myL = parent.defineModulesWithOneLoader( cf, ClassLoader.getSystemClassLoader()); 99
  65. How to use var finder = ModuleFinder.of(Paths.get(“MyLayer”)); var parent =

    ModuleLayer.boot(); Configuration cf = parent.configuration().resolve( finder, ModuleFinder.of(), Set.of(“myRootMod")); ModuleLayer myL = parent.defineModulesWithOneLoader( cf, ClassLoader.getSystemClassLoader()); 100
  66. How to use var finder = ModuleFinder.of(Paths.get(“MyLayer”)); var parent =

    ModuleLayer.boot(); Configuration cf = parent.configuration().resolve( finder, ModuleFinder.of(), Set.of(“myRootMod")); ModuleLayer myL = parent.defineModulesWithOneLoader( cf, ClassLoader.getSystemClassLoader()); 101
  67. How to use var finder = ModuleFinder.of(Paths.get(“MyLayer”)); var parent =

    ModuleLayer.boot(); Configuration cf = parent.configuration().resolve( finder, ModuleFinder.of(), Set.of(“myRootMod")); ModuleLayer myL = parent.defineModulesWithOneLoader( cf, ClassLoader.getSystemClassLoader()); 102
  68. How to use var finder = ModuleFinder.of(Paths.get(“MyLayer”)); var parent =

    ModuleLayer.boot(); Configuration cf = parent.configuration().resolve( finder, ModuleFinder.of(), Set.of(“myRootMod")); ModuleLayer myL = parent.defineModulesWithOneLoader( cf, ClassLoader.getSystemClassLoader()); var services = ServiceLoader.load(myL, MySrv.class); 103
  69. Where else we can use layers Problem: the servlet container

    standard knows nothing about modules so far –  Dependencies in war files are old plain classpath in fact! Solution: it’s possible to fork your favorite app server, adding a Jigsaw layer into its web application loader –  example: 105
  70. Where else we can use layers Problem: the servlet container

    standard knows nothing about modules so far –  Dependencies in war files are old plain classpath in fact! Solution: it’s possible to fork your favorite app server, adding a Jigsaw layer into its web application loader –  example: 107
  71. Where else we can use layers Problem: the servlet container

    standard knows nothing about modules so far –  Dependencies in war files are old plain classpath in fact! Solution: it’s possible to fork your favorite app server, adding a Jigsaw layer into its web application loader –  example: https://github.com/pjBooms/tomcat 108
  72. Tomcat •  Tomcat classpath contains bootstrap.jar •  Tomcat is loaded

    by its own classloader (org.apache.catalina.startup.ClassLoaderFactory) •  Each web application is loaded by its own classloader (org.apache.catalina.loader.WebappClassLoaderBase) 109
  73. Tomcat •  Any classloader has its own Unnamed module • 

    Without special efforts from a classloader implementer, classes loaded by a classloader would belong to its Unnamed module – even if сlasses formally belong to a named module (to a jar with module-info.class) 110
  74. Modular War Example Example (https://github.com/pjBooms/modular-war-example): –  2 modules: helloworld-provider, helloworld-web-app

    –  helloworld-web-app outputs module names of loaded classes on a web page: "<H3>HelloServlet module: " + HelloServlet.class.getModule().getName() + "</H3>\n" + "<H3>HelloWorldProvider module: " + HelloWorldProvider.class.getModule().getName() + "</H3>\n" + 111
  75. Modular War Example Example (https://github.com/pjBooms/modular-war-example): –  2 modules: helloworld-provider, helloworld-web-app

    –  helloworld-web-app outputs module names of loaded classes on a web page: "<H3>HelloServlet module: " + HelloServlet.class.getModule().getName() + "</H3>\n" + "<H3>HelloWorldProvider module: " + HelloWorldProvider.class.getModule().getName() + "</H3>\n" + 112
  76. Tomcat fork ClassLoaderFactory.createClassLoader modification: instead of return new URLClassLoader(array); modular

    layer creation: ModuleFinder finder = ModuleFinder.of(array); … ModuleLayer moduleLayer = parentLayer.defineModulesWithOneLoader(cf, parent); return moduleLayer.findLoader(moduleNames.get(0)); 115
  77. Tomcat fork ClassLoaderFactory.createClassLoader modification: instead of return new URLClassLoader(array); modular

    layer creation: ModuleFinder finder = ModuleFinder.of(array); … ModuleLayer moduleLayer = parentLayer.defineModulesWithOneLoader(cf, parent); return moduleLayer.findLoader(moduleNames.get(0)); 116
  78. Tomcat fork ClassLoaderFactory.createClassLoader modification: instead of return new URLClassLoader(array); modular

    layer creation: ModuleFinder finder = ModuleFinder.of(array); … ModuleLayer moduleLayer = parentLayer.defineModulesWithOneLoader(cf, parent); return moduleLayer.findLoader(moduleNames.get(0)); 117
  79. Tomcat fork WebAppClassloaderBase.start() modification modular layer creation: WebResource modules =

    resources.getResource("/WEB-INF/modules"); if (modules.exists()) { ModuleFinder finder = ModuleFinder.of(Paths.get(modules)); ModuleLayer parent = Catalina.class.getModule().getLayer(); … ModuleLayer moduleLayer = parent.defineModulesWithOneLoader(cf, getParent()); 118
  80. Tomcat fork WebAppClassloaderBase.start() modification modular layer creation: WebResource modules =

    resources.getResource("/WEB-INF/modules"); if (modules.exists()) { ModuleFinder finder = ModuleFinder.of(Paths.get(modules)); ModuleLayer parent = Catalina.class.getModule().getLayer(); … ModuleLayer moduleLayer = parent.defineModulesWithOneLoader(cf, getParent()); 119
  81. Tomcat fork WebAppClassloaderBase.start() modification modular layer creation: WebResource modules =

    resources.getResource("/WEB-INF/modules"); if (modules.exists()) { ModuleFinder finder = ModuleFinder.of(Paths.get(modules)); ModuleLayer parent = Catalina.class.getModule().getLayer(); … ModuleLayer moduleLayer = parent.defineModulesWithOneLoader(cf, getParent()); 120
  82. Tomcat fork WebAppClassloaderBase.loadClass() modification The layer is used to load

    web application classes: if (moduleLayerLoader != null) { try { return moduleLayerLoader.loadClass(name); } catch (ClassNotFoundException e) { // Ignore } } 121
  83. What about Spring Boot? •  Classes of Spring Boot executable

    Jar are loaded by a Spring Boot classloader •  The classloader knows nothing about modules •  Hence there are no modules at runtime for Spring Boot applications 124
  84. What about Spring Boot? •  Classes of Spring Boot executable

    Jar are loaded by a Spring Boot classloader •  The classloader knows nothing about modules •  Hence there are no modules at runtime for Spring Boot applications 125
  85. What about Spring Boot? •  Classes of Spring Boot executable

    Jar are loaded by a Spring Boot classloader •  The classloader knows nothing about modules •  Hence there are no modules at runtime for Spring Boot applications 126
  86. What about Spring Boot? •  Classes of Spring Boot executable

    Jar are loaded by a Spring Boot classloader •  The classloader knows nothing about modules •  Hence there are no modules at runtime for Spring Boot applications 127
  87. What about Spring Boot? Good news: Spring Boot applications can

    run without Spring Boot classloader – with all their dependencies put on classpath It is possible to modularize Spring Boot applications using standard migration path! 128
  88. What about Spring Boot? Good news: Spring Boot applications can

    run without Spring Boot classloader – with all their dependencies put on classpath It is possible to modularize Spring Boot applications using standard migration path! 129
  89. Conclusion Versions for modules look like a good idea but

    there are many problems with them: – backward compatibility – unsafety (classloaders constraints) – NP-completeness on dependencies resolution 131
  90. Conclusion Jigsaw Layers allow to solve the Jar Hell problem

    safely: – no classloaders constraints violations by construction – Jigsaw services help employing functionality implemented in a child layer from a parent layer 132
  91. Conclusion Knowledge about the layers can help in modularizing of

    Java applications including server side applications 133