Modernizing the Groovy build Cédric Champeau, Gradle Inc. @CedricChampeau

Cédric Champeau Gradle, Inc. Principal Software Engineer JVM Team at Gradle Worked on dependency management, performance, improving Java ecosystem support Former Groovy committer (wrote the static compiler)

Me and Apache Groovy Started as a user, submitted some bugfixes Became committer in 2011 Contributed a number of things mostly around DSL and compile-time metaprogramming: - AST transformations - Static type checker and static compilation (@CompileStatic) - Traits - Type checking extensions - Template engine with static compilation of templates

Me and Apache Groovy Joined the Gradle team in 2015 Left the Apache Groovy PMC in march 2019 after disagreement on use of Kotlin Contributed build changes… and left around build changes Now back to the build!

History of the Groovy build Groovy is an old project (17 years old!) Started with Apache Ant Attempt to migrate to Apache Maven but never finished/too complicated Gradle introduced in 2008 as a parallel build Really usable in 2012 after the Gradle 1.0 release This is where I started to contribute to the build

State of the build as of 1st october 2020 A working Gradle build, including build cache, but: ⬢ Written by many different people ⬢ With “conventions” inherited from older build tools (Ant, Maven) ⬢ With very old releases of Gradle ○ Many things were harder to do back then ○ Many outdated patterns ○ Accidental dependencies ○ Lack of guidance ○ ...

Complicating factors Groovy is built with Gradle Gradle uses a Groovy DSL (or Kotlin ;)) Groovy provides different flavors of the same thing: - “Classic” Groovy - “Invokedynamic” Groovy - “Grooid” Groovy

Apache Groovy 4 is coming!

Groovy 4 and Gradle 7 Groovy 4 Changes to dependency (GAV) coordinates: - From org.codehaus.groovy to org.apache.groovy Removal of the classic flavor Gradle 7 Removes the old “maven” plugin in favor of “maven-publish” Ideal to leverage some of the dependency management features introduced in Gradle 6!

Let’s take a quick look at the “old” build

Idiomatic Gradle

Idiomatic Gradle goals Provide clear guidance on best practices Reduce the maintenance burden Improve the learning curve Improve performance Deliver better quality software

Declarative build scripts Most build scripts should only consist of: - Applying one or more plugins - Declarative configuration What matters is the model A Java library != a Java application != a Micronaut application != a Groovy library != ...

Types of modules in the Groovy build Groovy core The “main” project: it’s a library, but also provides the Groovy compiler itself Groovy modules JSON support, Date utils, Template engines, … Aggregating projects Groovy “all” (which isn’t Groovy all but a subset of Groovy + Groovy modules) The BOM The distribution Utilities (stress tests, …)

Example: groovy-dateutil

What’s wrong with the previous pattern? ⬢ No plugin applied - hard to tell what this script is about. Hard to figure out where to make modifications. “Magic” going on. ⬢ Explicit task creation - Duplicated code between modules. Hardcoded task name ⬢ Explicit task dependency - and wrong! ⬢ How do you know it’s part of groovy-all?

Example 2: stress tests ⬢ Explicit project type: the intent is clear. It’s not a Groovy module, it’s not a publication, it’s a special kind of module ⬢ Distinct test suite: avoids dirty configuration to disable tests when we mostly don’t want to run them

Example 3: the BOM ⬢ Explicit project type: it’s a platform ⬢ Dependency declaration still a bit convoluted (better pattern coming)

Major changes 01 allprojects { … } → explicit application of a convention plugin in each project 02 Inline tasks → explicit task implementations in buildSrc 03 apply from: “...” → plugins { id … }

Example: get rid of “inline tasks”

Does it mean I need to publish plugins?!

buildSrc to the rescue! The buildSrc directory should be used to put all custom logic: - Custom tasks - Models (extensions) of different module types - Expose logic via plugins And we have precompiled script plugins!

Plugins all the way! Gradle uses composition over inheritance Different aspects of a project can be applied by different plugins Plugins can react to the presence of another plugin

Precompiled script plugins Those are lightweight plugins It’s a plugin without all the ceremony Most of custom build logic should move to plugins Can be easily composed

Example: the Groovy platform plugin buildSrc/main/groovy/org.apache.groovy-platform.gradle

Example: the Groovy platform plugin buildSrc/main/groovy/org.apache.groovy-common.gradle

Shared configuration (only applied to the root)

Project specific configuration

Extensions are created by plugins ⬢ Extensions should make the build script readable ⬢ They talk about the what and they abstract the how ⬢ Creates tasks if needed A build script just applies the plugin and configures the extension The extension does the “hard work”

Avoid reaching out to other projects directly

Strong encapsulation Do not depend on another project task - E.g “dependsOn rootProject.task(“jar”) This is fragile, unsafe and it breaks encapsulation. Was challenging with the Groovy build which abused reaching out to other projects everywhere. https://docs.gradle.org/current/userguide/cross_project_publications.html

Task configuration avoidance

Configuration avoidance Before After Description tasks.create(“...”) tasks.register To define a new task tasks.findByName(...) tasks.named(...) { … } Configure an existing task by name tasks.withType(Foo).all { … } tasks.withType(Foo) .configureEach { … } Configures a task by type

Configuration avoidance: Groovy DSL gotchas

The lazy configuration API Use of Provider and Property: - Fixes most of ordering issues during configuration (get rid of afterEvaluate) - Can carry task dependency information (ideal to get rid of accidental dependencies)

The lazy configuration API: derived values

Gradle has evolved

Test fixtures Tests from different projects need common testing utilities Previously, Groovy did this: - Create utility classes in `src/test` (so they belonged to the test sources) - Add the test runtime classpath of the main project to the test compile classpath of modules

The java-test-fixtures plugin Creates a separate source set for test fixtures (src/testFixtures/java) Makes it easy to use from other projects

Optional dependencies There are no optional dependencies, only dependencies you need when you use a particular feature

Optional dependencies Each feature can have its own dependencies

Optional dependencies publication Published as optional dependencies for Maven (POM.xml) and feature variants for Gradle No ugly configuration like before!

Let’s talk about publishing Publishing was in a terrible shape - Based on the deprecated publication mechanisms (maven plugin) - Used a lot of internal APIs - Made lots of customizations to the generated POM files - Because of relocated dependencies - Because of the multiple flavors

Publishing now Uses the standard maven-publish plugin - Got rid of all internal APIs - Publishes Gradle Module Metadata - Running `./gradlew publish` generates a local repository in `build/repo` which allows seeing what is going to be published Gradle Module Metadata benefits: - Users will get an error if they use both the old Groovy (Codehaus) and new Groovy (Apache) - They can depend on optional features and get the dependencies they need automatically - They get automatic version alignment for free!

There’s more we can do

Future work Use of Gradle Java Toolchain API - Allows compiling with a JDK, running/testing with another in a declarative manner - Automatic download of JDK! - Removes the need for special tweaks in the build depending on what is used to run Gradle itself But… - Requires Gradle 6.8 for Groovy compilation support - Requires changes to the build pipeline (but it makes it simpler!)

Future work Make the root project a “real” root project - groovy-core should become a subproject itself - Allows moving some remaining shared configuration to where it belongs Remove old Ant tasks - Some tasks like `jarjar` still depend on very old Ant tasks - May require some extra work from Gradle

Thank you! Special thanks to Paul King for the review of all changes, including bugs in introduced in the process!