GlassFish Story

Subtitle: How to Make a Huge Unmaintainable Obsoleted Project Maintainable and Modern

Eclipse GlassFish has a very long history. Many years ago hardware did not have so huge performance and developing, fixing and maintaining large projects was very problematic. As of 2023 we have many tools targeting maintenance of the software. Evolution is very fast and GlassFish was not left behind as it is still - and will be - used on production environments, as it is still probably the most complete platform of the whole Java EE ecosystem and we decided to contribute it to keep it modern.

But how? Is it even possible?

Transcript

1. GlassFish Story How to Make a Huge Unmaintainable Obsoleted Project Maintainable and Modern David Matějček @ OmniFish OŰ
2. How to Create Unmaintainable Project • Don’t care about maintenance ◦ Because it is too much expensive ◦ Because you have to test what you did ◦ Because it is too late ◦ Because nobody loves the project ◦ Because it will die soon ◦ Because nobody pays for it ◦ Because customer would not pay for a support of something that just works ◦ Because you are lazy ◦ Because you are not competent ◦ Because your hardware is lazy ◦ Because disabling tests is easier than fixing them • You succeeded! Now it is an unmaintainable project! “When you discover you are riding a dead horse, the best strategy is to dismount.” — Well, and then what?
3. Trap: Developer’s Hardware • Is your hardware strong enough? • Don’t hesitate to ask for a professional laptop for all tasks on the project - or buy your own! • Authorization: administrator account
4. Trap: Metaphors and Buzzwords • Technical Debt • Zombie • Unit Test • Integration Test • Performance Test • Load Test • Test Coverage • Cloud Think about meanings!
5. Trap: Benders for Narrowers • Quick Fixes • Prototypes • Temporary Implementations • Workarounds • Workarounds for workarounds • Workarounds for workarounds for workarounds • Workarounds for workarounds for workarounds for workarounds • Workarounds for workarounds for workarounds for workarounds for workaro Basic unit of the temporarity - is the eternity
6. Trap: Premature Optimization • Expensive ◦ Caches ◦ Locks ◦ Thread pools ◦ Asynchronous operations ◦ Cloud clusters • Cheap ◦ Refactoring ◦ Performance tests ◦ Diagnostics ◦ Documentation The donkey was fine, so he went to slide on the ice and broke his legs
7. Trap: Whitespaces • “Oh no, diff is ugly!” • Indentation: 4 spaces is the most used standard today. • Forbid trailing spaces • Forbid tabs • Time to introduce checkstyle
8. Help: Checkstyle • The maven-checkstyle-plugin • Enabling rules one by one and never go back <plugin > < artifactId >maven-checkstyle-plugin</ artifactId > < version >3.3.0</ version > < configuration > < configLocation >org/glassfish/qa/config/checkstyle/checkstyle.xml</ configLocation > <suppressionsLocation >org/glassfish/qa/config/checkstyle/checkstyle-suppressions.xml</ suppressionsLocation > < includeTestSourceDirectory >true</ includeTestSourceDirectory > < skip>${checkstyle.skip}</ skip> < logViolationsToConsole >true</ logViolationsToConsole > < excludes >**/generated-sources/**/*, **/module-info.java</ excludes > <!-- build-helper-plugin adds the root as a resource path, but checkstyle overrides the filter and adds all property files in the tree --> < resourceExcludes > **/appserver/**/src/main/resources/**/*, **/deployment/**/src/main/resources/**/*, **/target/**/*, </ resourceExcludes > </ configuration > < dependencies > < dependency > < groupId >org.glassfish.main</ groupId > < artifactId >glassfish-qa-config</ artifactId > < version >${project.version}</ version > < scope >runtime</ scope > </ dependency > < dependency > < groupId >com.puppycrawl.tools</ groupId > < artifactId >checkstyle</ artifactId > < version >10.12.4</ version > </ dependency > </ dependencies > < executions > < execution > < id>${project.groupId}-${project.artifactId}-checkstyle</ id> < goals > < goal>check</ goal> </ goals > <!-- why: tests first, quality requirements later --> < phase >verify</ phase > </ execution > </ executions > </plugin >
9. Help: PMD • Not used in GlassFish yet • Detects common antipatterns in sources
10. Help: CPD / Copy&Paste Detector • TODO, not used in GlassFish yet • Detects copy pasted code in sources and forces you to refactor • Too much copy pasted code -> refuse PR, do it better!
11. Help: SpotBugs • Successor of FindBugs • Detects common antipatterns in bytecode • FindBugs was used by GF years ago… • … We plan to use SpotBugs.
12. Help: JaCoCo - Java Code Coverage • Standard component of Eclipse IDE now • Add Java Agent to the JVM which will report the code visited in runtime. • Ability to merge results • Important is not the code covered, but the code uncovered ◦ Is it a dead code to be deleted? ◦ Should we write more tests? ◦ How far are we from 100%? • Misunderstanding ◦ Covering 100% doesn’t mean 100% correctness. ◦ Covering 5% means you are in a trouble. Something burns. Soon. Too soon.
13. Help: SonarQube
14. Help: Snyk • Security issues • Links to CWE • Recommendations
15. GlassFish Archeology • December 1994: The first release of Netsite by Netscape • 1996: Netscape Enterprise Web Server • 2003: Sun ONE Application Server 7 – that was a great thing. Java 1.3, containers, instances configured from a single UI. In 2007 it was the first “GlassFish” I used on one huge project with extreme technical debt and I started learning how to fight such debt. • 2009: Sun GlassFish Enterprise Server 2.1.1.2 – the last version I liked. Since then the concept is mostly the same, but … • 2010: Sun was acquired by Oracle • 2013: Oracle declared that the commercial support for GlassFish will stop. I patched GlassFish on my own as it’s SVN repository was public at the time. • 2014: Payara was born, the fork of the Oracle GlassFish. The C2B2 team and contributors started fixing bugs in a really impressive speed. And I loved it. • 2017: Oracle released several versions of GlassFish until that year, but all contained so many bugs that GlassFish couldn’t be used on any of our production environments.
16. Immortal Eclipse GlassFish • 2019: Oracle transferred GlassFish to Eclipse Foundation together with other Java EE projects. The Jakarta EE was born. However, GlassFish 5.1 was still buggy. • June 2019 – April 2021: I worked for Payara, we also pushed our fixes back to GlassFish. • May 2021: I left Payara and started contributing directly to Eclipse GlassFish. Since then, GlassFish has seen a huge spike in the number of commits, with almost 800 commits during the next 12 months from all contributors - more than all commits for the previous 2 and half years since GlassFish was donated to the Eclipse Foundation. • April 2022: We created the new company, OmniFish. Now it is serious! • December 2022: GlassFish 7.0.0 ◦ The first Jakarta EE 10 compliant implementation ◦ Commercial support by OmniFish • October 2023: GlassFish 7.0.10 • 2024: GlassFish 8.0.0
17. GlassFish Family • Fujitsu Software Enterprise Application Platform ◦ Fork, with own patches and own identity ◦ syncs with upstream and contributes changes to upstream. • Payara Server ◦ Fork from GlassFish 4.1. ◦ Uses original CDDL/GPL+CP license. ◦ Contributed to GlassFish Jakarta EE ◦ Contributes to components such as Soteria. • Primeton AppServer ◦ Redistribution of GlassFish in China, with Chinese download sites and support channels. • ManageFish Server ◦ Redistribution of GlassFish by The ManageCat, with support for upstream GlassFish. • Piranha Cloud ◦ Independently developed server, but uses all components from GlassFish. • InforSuite AS ◦ Redistribution of GlassFish in China, with Chinese download sites and support channels. • More ???
18. New: GlassFish Docker Image • PR for official GlassFish repository is waiting to be accepted by DockerHub • Meanwhile you can use temporary omnifish/glassfish repository • The omnifish/glassfish is in progress of transferring to the Eclipse Foundation • Compliant with the DockerHub rules and best practices • TODO: Optimize (Automatically?)
19. New: TestContainers • First usage in GlassFish: mail server for the TCK • GlassFish Docker Image can be used by your tests • We welcome your feedback!
20. New: GlassFishTestEnvironment • “TestContainers without containers” • Uses system hooks to ensure GlassFish shutdown • Provides some asadmin API and usual tasks and paths • Advantages ◦ Run tests from your IDE, repeatability ◦ No magic, you can use it as you need • Disadvantages ◦ Less possibilities than TC ▪ testing external hazards, networking issues, exhausted OS resources, etc <dependency > <groupId>org.glassfish.main</ groupId> <artifactId >glassfish-itest-tools</ artifactId > <version>7.0.10</ version> <scope>test</scope> </dependency >
21. public class GlassFishTestEnvironment { private static final Logger LOG = Logger.getLogger(GlassFishTestEnvironment. class.getName()); private static final File BASEDIR = detectBasedir(); private static final File GF_ROOT = resolveGlassFishRoot(); private static final String ADMIN_USER = "admin"; private static final String ADMIN_PASSWORD = "admintest" ; private static final File ASADMIN = findAsadmin(); private static final File KEYTOOL = findKeyTool(); private static final File PASSWORD_FILE_FOR_UPDATE = findPasswordFile("password_update.txt" ); private static final File PASSWORD_FILE = findPasswordFile("password.txt" ); private static final int ASADMIN_START_DOMAIN_TIMEOUT = 30_000; static { LOG.log(Level. INFO, "Using basedir: {0}" , BASEDIR); LOG.log(Level. INFO, "Expected GlassFish directory: {0}" , GF_ROOT); changePassword(); Thread hook = new Thread(() -> { getAsadmin().exec(10_000, "stop-domain" , "--kill", "--force"); }); Runtime. getRuntime().addShutdownHook( hook); Asadmin asadmin = getAsadmin().withEnv( ADMIN_USER, ADMIN_PASSWORD); if (System.getenv("AS_START_TIMEOUT" ) == null) { // AS_START_TIMEOUT for the detection that "the server is running!" // START_DOMAIN_TIMEOUT for us waiting for the end of the asadmin start-domain process. asadmin.withEnv("AS_START_TIMEOUT" , Integer.toString(ASADMIN_START_DOMAIN_TIMEOUT - 5000)); } // This is the absolutely first start - if it fails, all other starts will fail too. assertThat(asadmin.exec(ASADMIN_START_DOMAIN_TIMEOUT, "start-domain" , "--debug"), asadminOK()); } /** * @return {@link Asadmin} command api for tests. */ public static Asadmin getAsadmin() { return new Asadmin(ASADMIN, ADMIN_USER, PASSWORD_FILE); } public static KeyTool getKeyTool() { return new KeyTool(KEYTOOL); } New: GlassFishTestEnvironment
22. New: GlassFishTestEnvironment class ManagedExecutorDefinitionWebTest { private static final Logger LOG = System.getLogger(ManagedExecutorDefinitionWebTest.class.getName()); private static final Asadmin ASADMIN = GlassFishTestEnvironment.getAsadmin(); private static final String APP_NAME = ManagedExecutorDefinitionServlet.class.getSimpleName(); @BeforeAll static void deploy() { File war = createDeployment(); try { AsadminResult result = ASADMIN.exec("deploy", "--contextroot", "/", "--name", APP_NAME, war.getAbsolutePath()); assertThat(result, AsadminResultMatcher.asadminOK()); } finally { war.delete(); } } @AfterAll static void undeploy() { AsadminResult result = ASADMIN.exec("undeploy", APP_NAME); assertThat(result, AsadminResultMatcher.asadminOK()); } @Test void testCopyCompletableFutureTwice() throws Exception { HttpURLConnection connection = GlassFishTestEnvironment.openConnection(8080, "/"); connection.setRequestMethod("GET"); assertEquals(200, connection.getResponseCode()); } private static File createDeployment() { WebArchive war = ShrinkWrap.create(WebArchive.class, "ManagedExecutorDefinitionWebTest.war") .addClasses(ManagedExecutorDefinitionServlet.class, IntContextProvider.class, StringContextProvider.class) .addAsServiceProvider(ThreadContextProvider.class, IntContextProvider.class, StringContextProvider.class); LOG.log(INFO, war.toString(true)); try { File tempFile = File.createTempFile(APP_NAME, ".war"); war.as(ZipExporter.class).exportTo(tempFile, true); return tempFile; } catch (IOException e) { throw new IllegalStateException("Deployment failed - cannot load the input archive!", e); } } }
23. New: GlassFishTestEnvironment
24. GlassFish’s Builds • Build Time: 70 seconds with mvn clean install -Pfastest -T4C • Build Time with all mavenized tests: 11 minutes. • Build Time with all tests in the repo: 1-2 hours • TCK tests: 1 day (parallel on Jenkins). ◦ Good news! We can run just blocks as reproducers. • Quite standard maven project - possible to use -amd, -am, -pl, -rf
25. GlassFish’s Ant Zombie Army • Cute little zombies, we release them from a spell of some great wizard. • We fixed several thousands tests and we maintain and use them • They are slowly revisited and migrated to maven ◦ Executable from Eclipse IDE individually or in groups • Around 5000 integration tests
26. GlassFish’s TckRunner • More zombies • The TCK team now refactors around 60000 integration tests ◦ Ant + Java -> mostly to Maven + Arquillian + Junit5/TestNG ◦ Will be finished for Jakarta EE 11 ◦ The team is happy for any help! • Jakarta EE 10 TCK Tests ◦ Already refactored tests ▪ mvn clean install -Ptck -pl :jakarta-connectors-full-tck,:glassfish-external-tck-connectors-full ◦ Zombies ▪ mvn clean install -Ptck -pl :tck-runner -Dit.test=JdbcITest ▪ mvn clean install -Ptck -pl :tck-runner -Dit.test=CustomITest -Dtck.module=ejb30/bb/session/stateful
27. GlassFish Pains • Most of features covered by integration tests ◦ around 70000 tests running approximately 90 hours (and in 2001…?) • High technical debt • Modularity without unit tests • For 20 years we are at 70% of path to unit test support with hk2+ejb+gfcore • JMX support looks like a prototype for 20 years - but works! • Deployment library - based on slow XML parser, overcomplicated • Missing or disabled tests for clustering • Missing contributors and sponsors
28. GlassFish Treatment • 2021: Tests executable just on Jenkins • 2022: Tests executable anywhere on Linux, reproducible! • 2023: Tests executable anywhere on Linux, Windows and Mac (Maven) • Several refactoring iterations done • JUnit3 and JUnit4 -> JUnit5 • Deployment module is now faster and with less bugs • Logging is faster and more reliable • JNDI is more maintainable, based on value objects, not strings • Authorization and Authentication separated and cleaned up • Visited some dependencies and resolved their issues • Refactored classloaders
29. GlassFish Treatment • Refactored glassfishbuild-maven-plugin • Removed all osgi.bundle files, the Felix plugin does a good job • Fixed Embedded GlassFish • Fixed several serious memory leaks • Fixed incompatibilities with new Java 17 rules (reflecion, method handles, proxies) • Currently uses Java 17, compiles for 11, tests also for 11 ◦ Experimental compilations also with 18, 19, 20, 21
30. GlassFish Health Report • Stabilized, production ready again. • Treatment continues along development of new features. • Makes new friends, have plans about the future together. • Zombies are not biting any more, they are waiting to be cured too. • New Maven tests, new use cases, new ways of testing in a modern way.
31. GlassFish JUL Extension • We rewrote GlassFish logging system • Logs look nearly the same • New: SimpleLogHandler + OneLineFormatter • New: Microseconds instead of milliseconds • New: Command line options -> can be used in unit tests • New: Logging of logging (oh, don’t use it … did you try it?) • New: Thread-safety + optimizations + streaming = faster • New: Logging from the real start until the real end. • New: Logging is independent on GlassFish, but GlassFish can reconfigure it. • New: Test Coverage 87% (Line)
32. Microprofile Support GlassFish supports two Micrprofile specs now • Microprofile Config (Helidon) • Microprofile JWT (OmniFaces) ◦ Authentication mechanism for Jakarta Security We plan to support more in future versions - in negotiation with partners like Fujitsu and Payara.
33. Extracted Components to Standalone Projects • GlassFish Jakarta Authorization -> Eclipse Exousia • GlassFish Jakarta Authentication -> Eclipse Epicyro • GlassFish Jakarta Transaction ◦ in progress, now it is OmniFish Transact, later will be donated to Eclipse • These components can be used also by other projects ◦ Example: OmniFish Piranha
34. TODO - GlassFish in 2024 • GlassFish 8 and Jakarta EE 11 and Java 21 • JMH, performance and load tests • Production ready domain.xml? • Stability tests (TestContainers - failing networks, poor HW simulations, etc) • New Admin UI? Java FX?
35. Visions - GlassFish in 2040 • GlassFish in IoT? • GlassFish in cars? • GlassFish in cell phones? • Support of new protocols? • How decentralized can be the communication in the future? • Autotuning? • Cloud and semiautomatic autoscaling? ◦ Levels of autoscaling? ◦ How much can subsystems communicate to find the optimal configuration? ◦ Need to define rules, ranges, combinations … new discipline for testing
36. Thank You! https://glassfish.org/ https://projects.eclipse.org/projects/ee4j.glassfish https://github.com/eclipse-ee4j/glassfish https://omnifish.ee/solutions/