Slide 1

Slide 1 text

GlassFish Story How to Make a Huge Unmaintainable Obsoleted Project Maintainable and Modern David Matějček @ OmniFish OŰ

Slide 2

Slide 2 text

How to Create Unmaintainable Project ● Don’t care about maintenance ○ Because it is too much expensive ○ Because you have to test what you did ○ Because it is too late ○ Because nobody loves the project ○ Because it will die soon ○ Because nobody pays for it ○ Because customer would not pay for a support of something that just works ○ Because you are lazy ○ Because you are not competent ○ Because your hardware is lazy ○ Because disabling tests is easier than fixing them ● You succeeded! Now it is an unmaintainable project! “When you discover you are riding a dead horse, the best strategy is to dismount.” — Well, and then what?

Slide 3

Slide 3 text

Trap: Developer’s Hardware ● Is your hardware strong enough? ● Don’t hesitate to ask for a professional laptop for all tasks on the project - or buy your own! ● Authorization: administrator account

Slide 4

Slide 4 text

Trap: Metaphors and Buzzwords ● Technical Debt ● Zombie ● Unit Test ● Integration Test ● Performance Test ● Load Test ● Test Coverage ● Cloud Think about meanings!

Slide 5

Slide 5 text

Trap: Benders for Narrowers ● Quick Fixes ● Prototypes ● Temporary Implementations ● Workarounds ● Workarounds for workarounds ● Workarounds for workarounds for workarounds ● Workarounds for workarounds for workarounds for workarounds ● Workarounds for workarounds for workarounds for workarounds for workaro Basic unit of the temporarity - is the eternity

Slide 6

Slide 6 text

Trap: Premature Optimization ● Expensive ○ Caches ○ Locks ○ Thread pools ○ Asynchronous operations ○ Cloud clusters ● Cheap ○ Refactoring ○ Performance tests ○ Diagnostics ○ Documentation The donkey was fine, so he went to slide on the ice and broke his legs

Slide 7

Slide 7 text

Trap: Whitespaces ● “Oh no, diff is ugly!” ● Indentation: 4 spaces is the most used standard today. ● Forbid trailing spaces ● Forbid tabs ● Time to introduce checkstyle

Slide 8

Slide 8 text

Help: Checkstyle ● The maven-checkstyle-plugin ● Enabling rules one by one and never go back < artifactId >maven-checkstyle-plugin < version >3.3.0 < configuration > < configLocation >org/glassfish/qa/config/checkstyle/checkstyle.xml org/glassfish/qa/config/checkstyle/checkstyle-suppressions.xml < includeTestSourceDirectory >true < skip>${checkstyle.skip} < logViolationsToConsole >true < excludes >**/generated-sources/**/*, **/module-info.java < resourceExcludes > **/appserver/**/src/main/resources/**/*, **/deployment/**/src/main/resources/**/*, **/target/**/*, < dependencies > < dependency > < groupId >org.glassfish.main < artifactId >glassfish-qa-config < version >${project.version} < scope >runtime < dependency > < groupId >com.puppycrawl.tools < artifactId >checkstyle < version >10.12.4 < executions > < execution > < id>${project.groupId}-${project.artifactId}-checkstyle < goals > < goal>check < phase >verify

Slide 9

Slide 9 text

Help: PMD ● Not used in GlassFish yet ● Detects common antipatterns in sources

Slide 10

Slide 10 text

Help: CPD / Copy&Paste Detector ● TODO, not used in GlassFish yet ● Detects copy pasted code in sources and forces you to refactor ● Too much copy pasted code -> refuse PR, do it better!

Slide 11

Slide 11 text

Help: SpotBugs ● Successor of FindBugs ● Detects common antipatterns in bytecode ● FindBugs was used by GF years ago… ● … We plan to use SpotBugs.

Slide 12

Slide 12 text

Help: JaCoCo - Java Code Coverage ● Standard component of Eclipse IDE now ● Add Java Agent to the JVM which will report the code visited in runtime. ● Ability to merge results ● Important is not the code covered, but the code uncovered ○ Is it a dead code to be deleted? ○ Should we write more tests? ○ How far are we from 100%? ● Misunderstanding ○ Covering 100% doesn’t mean 100% correctness. ○ Covering 5% means you are in a trouble. Something burns. Soon. Too soon.

Slide 13

Slide 13 text

Help: SonarQube

Slide 14

Slide 14 text

Help: Snyk ● Security issues ● Links to CWE ● Recommendations

Slide 15

Slide 15 text

GlassFish Archeology ● December 1994: The first release of Netsite by Netscape ● 1996: Netscape Enterprise Web Server ● 2003: Sun ONE Application Server 7 – that was a great thing. Java 1.3, containers, instances configured from a single UI. In 2007 it was the first “GlassFish” I used on one huge project with extreme technical debt and I started learning how to fight such debt. ● 2009: Sun GlassFish Enterprise Server 2.1.1.2 – the last version I liked. Since then the concept is mostly the same, but … ● 2010: Sun was acquired by Oracle ● 2013: Oracle declared that the commercial support for GlassFish will stop. I patched GlassFish on my own as it’s SVN repository was public at the time. ● 2014: Payara was born, the fork of the Oracle GlassFish. The C2B2 team and contributors started fixing bugs in a really impressive speed. And I loved it. ● 2017: Oracle released several versions of GlassFish until that year, but all contained so many bugs that GlassFish couldn’t be used on any of our production environments.

Slide 16

Slide 16 text

Immortal Eclipse GlassFish ● 2019: Oracle transferred GlassFish to Eclipse Foundation together with other Java EE projects. The Jakarta EE was born. However, GlassFish 5.1 was still buggy. ● June 2019 – April 2021: I worked for Payara, we also pushed our fixes back to GlassFish. ● May 2021: I left Payara and started contributing directly to Eclipse GlassFish. Since then, GlassFish has seen a huge spike in the number of commits, with almost 800 commits during the next 12 months from all contributors - more than all commits for the previous 2 and half years since GlassFish was donated to the Eclipse Foundation. ● April 2022: We created the new company, OmniFish. Now it is serious! ● December 2022: GlassFish 7.0.0 ○ The first Jakarta EE 10 compliant implementation ○ Commercial support by OmniFish ● October 2023: GlassFish 7.0.10 ● 2024: GlassFish 8.0.0

Slide 17

Slide 17 text

GlassFish Family ● Fujitsu Software Enterprise Application Platform ○ Fork, with own patches and own identity ○ syncs with upstream and contributes changes to upstream. ● Payara Server ○ Fork from GlassFish 4.1. ○ Uses original CDDL/GPL+CP license. ○ Contributed to GlassFish Jakarta EE ○ Contributes to components such as Soteria. ● Primeton AppServer ○ Redistribution of GlassFish in China, with Chinese download sites and support channels. ● ManageFish Server ○ Redistribution of GlassFish by The ManageCat, with support for upstream GlassFish. ● Piranha Cloud ○ Independently developed server, but uses all components from GlassFish. ● InforSuite AS ○ Redistribution of GlassFish in China, with Chinese download sites and support channels. ● More ???

Slide 18

Slide 18 text

New: GlassFish Docker Image ● PR for official GlassFish repository is waiting to be accepted by DockerHub ● Meanwhile you can use temporary omnifish/glassfish repository ● The omnifish/glassfish is in progress of transferring to the Eclipse Foundation ● Compliant with the DockerHub rules and best practices ● TODO: Optimize (Automatically?)

Slide 19

Slide 19 text

New: TestContainers ● First usage in GlassFish: mail server for the TCK ● GlassFish Docker Image can be used by your tests ● We welcome your feedback!

Slide 20

Slide 20 text

New: GlassFishTestEnvironment ● “TestContainers without containers” ● Uses system hooks to ensure GlassFish shutdown ● Provides some asadmin API and usual tasks and paths ● Advantages ○ Run tests from your IDE, repeatability ○ No magic, you can use it as you need ● Disadvantages ○ Less possibilities than TC ■ testing external hazards, networking issues, exhausted OS resources, etc org.glassfish.main glassfish-itest-tools 7.0.10 test

Slide 21

Slide 21 text

public class GlassFishTestEnvironment { private static final Logger LOG = Logger.getLogger(GlassFishTestEnvironment. class.getName()); private static final File BASEDIR = detectBasedir(); private static final File GF_ROOT = resolveGlassFishRoot(); private static final String ADMIN_USER = "admin"; private static final String ADMIN_PASSWORD = "admintest" ; private static final File ASADMIN = findAsadmin(); private static final File KEYTOOL = findKeyTool(); private static final File PASSWORD_FILE_FOR_UPDATE = findPasswordFile("password_update.txt" ); private static final File PASSWORD_FILE = findPasswordFile("password.txt" ); private static final int ASADMIN_START_DOMAIN_TIMEOUT = 30_000; static { LOG.log(Level. INFO, "Using basedir: {0}" , BASEDIR); LOG.log(Level. INFO, "Expected GlassFish directory: {0}" , GF_ROOT); changePassword(); Thread hook = new Thread(() -> { getAsadmin().exec(10_000, "stop-domain" , "--kill", "--force"); }); Runtime. getRuntime().addShutdownHook( hook); Asadmin asadmin = getAsadmin().withEnv( ADMIN_USER, ADMIN_PASSWORD); if (System.getenv("AS_START_TIMEOUT" ) == null) { // AS_START_TIMEOUT for the detection that "the server is running!" // START_DOMAIN_TIMEOUT for us waiting for the end of the asadmin start-domain process. asadmin.withEnv("AS_START_TIMEOUT" , Integer.toString(ASADMIN_START_DOMAIN_TIMEOUT - 5000)); } // This is the absolutely first start - if it fails, all other starts will fail too. assertThat(asadmin.exec(ASADMIN_START_DOMAIN_TIMEOUT, "start-domain" , "--debug"), asadminOK()); } /** * @return {@link Asadmin} command api for tests. */ public static Asadmin getAsadmin() { return new Asadmin(ASADMIN, ADMIN_USER, PASSWORD_FILE); } public static KeyTool getKeyTool() { return new KeyTool(KEYTOOL); } New: GlassFishTestEnvironment

Slide 22

Slide 22 text

New: GlassFishTestEnvironment class ManagedExecutorDefinitionWebTest { private static final Logger LOG = System.getLogger(ManagedExecutorDefinitionWebTest.class.getName()); private static final Asadmin ASADMIN = GlassFishTestEnvironment.getAsadmin(); private static final String APP_NAME = ManagedExecutorDefinitionServlet.class.getSimpleName(); @BeforeAll static void deploy() { File war = createDeployment(); try { AsadminResult result = ASADMIN.exec("deploy", "--contextroot", "/", "--name", APP_NAME, war.getAbsolutePath()); assertThat(result, AsadminResultMatcher.asadminOK()); } finally { war.delete(); } } @AfterAll static void undeploy() { AsadminResult result = ASADMIN.exec("undeploy", APP_NAME); assertThat(result, AsadminResultMatcher.asadminOK()); } @Test void testCopyCompletableFutureTwice() throws Exception { HttpURLConnection connection = GlassFishTestEnvironment.openConnection(8080, "/"); connection.setRequestMethod("GET"); assertEquals(200, connection.getResponseCode()); } private static File createDeployment() { WebArchive war = ShrinkWrap.create(WebArchive.class, "ManagedExecutorDefinitionWebTest.war") .addClasses(ManagedExecutorDefinitionServlet.class, IntContextProvider.class, StringContextProvider.class) .addAsServiceProvider(ThreadContextProvider.class, IntContextProvider.class, StringContextProvider.class); LOG.log(INFO, war.toString(true)); try { File tempFile = File.createTempFile(APP_NAME, ".war"); war.as(ZipExporter.class).exportTo(tempFile, true); return tempFile; } catch (IOException e) { throw new IllegalStateException("Deployment failed - cannot load the input archive!", e); } } }

Slide 23

Slide 23 text

New: GlassFishTestEnvironment

Slide 24

Slide 24 text

GlassFish’s Builds ● Build Time: 70 seconds with mvn clean install -Pfastest -T4C ● Build Time with all mavenized tests: 11 minutes. ● Build Time with all tests in the repo: 1-2 hours ● TCK tests: 1 day (parallel on Jenkins). ○ Good news! We can run just blocks as reproducers. ● Quite standard maven project - possible to use -amd, -am, -pl, -rf

Slide 25

Slide 25 text

GlassFish’s Ant Zombie Army ● Cute little zombies, we release them from a spell of some great wizard. ● We fixed several thousands tests and we maintain and use them ● They are slowly revisited and migrated to maven ○ Executable from Eclipse IDE individually or in groups ● Around 5000 integration tests

Slide 26

Slide 26 text

GlassFish’s TckRunner ● More zombies ● The TCK team now refactors around 60000 integration tests ○ Ant + Java -> mostly to Maven + Arquillian + Junit5/TestNG ○ Will be finished for Jakarta EE 11 ○ The team is happy for any help! ● Jakarta EE 10 TCK Tests ○ Already refactored tests ■ mvn clean install -Ptck -pl :jakarta-connectors-full-tck,:glassfish-external-tck-connectors-full ○ Zombies ■ mvn clean install -Ptck -pl :tck-runner -Dit.test=JdbcITest ■ mvn clean install -Ptck -pl :tck-runner -Dit.test=CustomITest -Dtck.module=ejb30/bb/session/stateful

Slide 27

Slide 27 text

GlassFish Pains ● Most of features covered by integration tests ○ around 70000 tests running approximately 90 hours (and in 2001…?) ● High technical debt ● Modularity without unit tests ● For 20 years we are at 70% of path to unit test support with hk2+ejb+gfcore ● JMX support looks like a prototype for 20 years - but works! ● Deployment library - based on slow XML parser, overcomplicated ● Missing or disabled tests for clustering ● Missing contributors and sponsors

Slide 28

Slide 28 text

GlassFish Treatment ● 2021: Tests executable just on Jenkins ● 2022: Tests executable anywhere on Linux, reproducible! ● 2023: Tests executable anywhere on Linux, Windows and Mac (Maven) ● Several refactoring iterations done ● JUnit3 and JUnit4 -> JUnit5 ● Deployment module is now faster and with less bugs ● Logging is faster and more reliable ● JNDI is more maintainable, based on value objects, not strings ● Authorization and Authentication separated and cleaned up ● Visited some dependencies and resolved their issues ● Refactored classloaders

Slide 29

Slide 29 text

GlassFish Treatment ● Refactored glassfishbuild-maven-plugin ● Removed all osgi.bundle files, the Felix plugin does a good job ● Fixed Embedded GlassFish ● Fixed several serious memory leaks ● Fixed incompatibilities with new Java 17 rules (reflecion, method handles, proxies) ● Currently uses Java 17, compiles for 11, tests also for 11 ○ Experimental compilations also with 18, 19, 20, 21

Slide 30

Slide 30 text

GlassFish Health Report ● Stabilized, production ready again. ● Treatment continues along development of new features. ● Makes new friends, have plans about the future together. ● Zombies are not biting any more, they are waiting to be cured too. ● New Maven tests, new use cases, new ways of testing in a modern way.

Slide 31

Slide 31 text

GlassFish JUL Extension ● We rewrote GlassFish logging system ● Logs look nearly the same ● New: SimpleLogHandler + OneLineFormatter ● New: Microseconds instead of milliseconds ● New: Command line options -> can be used in unit tests ● New: Logging of logging (oh, don’t use it … did you try it?) ● New: Thread-safety + optimizations + streaming = faster ● New: Logging from the real start until the real end. ● New: Logging is independent on GlassFish, but GlassFish can reconfigure it. ● New: Test Coverage 87% (Line)

Slide 32

Slide 32 text

Microprofile Support GlassFish supports two Micrprofile specs now ● Microprofile Config (Helidon) ● Microprofile JWT (OmniFaces) ○ Authentication mechanism for Jakarta Security We plan to support more in future versions - in negotiation with partners like Fujitsu and Payara.

Slide 33

Slide 33 text

Extracted Components to Standalone Projects ● GlassFish Jakarta Authorization -> Eclipse Exousia ● GlassFish Jakarta Authentication -> Eclipse Epicyro ● GlassFish Jakarta Transaction ○ in progress, now it is OmniFish Transact, later will be donated to Eclipse ● These components can be used also by other projects ○ Example: OmniFish Piranha

Slide 34

Slide 34 text

TODO - GlassFish in 2024 ● GlassFish 8 and Jakarta EE 11 and Java 21 ● JMH, performance and load tests ● Production ready domain.xml? ● Stability tests (TestContainers - failing networks, poor HW simulations, etc) ● New Admin UI? Java FX?

Slide 35

Slide 35 text

Visions - GlassFish in 2040 ● GlassFish in IoT? ● GlassFish in cars? ● GlassFish in cell phones? ● Support of new protocols? ● How decentralized can be the communication in the future? ● Autotuning? ● Cloud and semiautomatic autoscaling? ○ Levels of autoscaling? ○ How much can subsystems communicate to find the optimal configuration? ○ Need to define rules, ranges, combinations … new discipline for testing

Slide 36

Slide 36 text

Thank You! https://glassfish.org/ https://projects.eclipse.org/projects/ee4j.glassfish https://github.com/eclipse-ee4j/glassfish https://omnifish.ee/solutions/