Modernizing the Groovy build

Transcript

1. Modernizing the Groovy build Cédric Champeau, Gradle Inc. @CedricChampeau
2. Cédric Champeau Gradle, Inc. Principal Software Engineer JVM Team at Gradle Worked on dependency management, performance, improving Java ecosystem support Former Groovy committer (wrote the static compiler)
3. Me and Apache Groovy Started as a user, submitted some bugfixes Became committer in 2011 Contributed a number of things mostly around DSL and compile-time metaprogramming: - AST transformations - Static type checker and static compilation (@CompileStatic) - Traits - Type checking extensions - Template engine with static compilation of templates
4. Me and Apache Groovy Joined the Gradle team in 2015 Left the Apache Groovy PMC in march 2019 after disagreement on use of Kotlin Contributed build changes… and left around build changes Now back to the build!
5. History of the Groovy build Groovy is an old project (17 years old!) Started with Apache Ant Attempt to migrate to Apache Maven but never finished/too complicated Gradle introduced in 2008 as a parallel build Really usable in 2012 after the Gradle 1.0 release This is where I started to contribute to the build
6. State of the build as of 1st october 2020 A working Gradle build, including build cache, but: ⬢ Written by many different people ⬢ With “conventions” inherited from older build tools (Ant, Maven) ⬢ With very old releases of Gradle ◦ Many things were harder to do back then ◦ Many outdated patterns ◦ Accidental dependencies ◦ Lack of guidance ◦ ...
7. Complicating factors Groovy is built with Gradle Gradle uses a Groovy DSL (or Kotlin ;)) Groovy provides different flavors of the same thing: - “Classic” Groovy - “Invokedynamic” Groovy - “Grooid” Groovy
8. Apache Groovy 4 is coming!
9. Groovy 4 and Gradle 7 Groovy 4 Changes to dependency (GAV) coordinates: - From org.codehaus.groovy to org.apache.groovy Removal of the classic flavor Gradle 7 Removes the old “maven” plugin in favor of “maven-publish” Ideal to leverage some of the dependency management features introduced in Gradle 6!
10. Let’s take a quick look at the “old” build
11. Idiomatic Gradle
12. Idiomatic Gradle goals Provide clear guidance on best practices Reduce the maintenance burden Improve the learning curve Improve performance Deliver better quality software
13. Declarative build scripts Most build scripts should only consist of: - Applying one or more plugins - Declarative configuration What matters is the model A Java library != a Java application != a Micronaut application != a Groovy library != ...
14. Types of modules in the Groovy build Groovy core The “main” project: it’s a library, but also provides the Groovy compiler itself Groovy modules JSON support, Date utils, Template engines, … Aggregating projects Groovy “all” (which isn’t Groovy all but a subset of Groovy + Groovy modules) The BOM The distribution Utilities (stress tests, …)
15. Example: groovy-dateutil
16. What’s wrong with the previous pattern? ⬢ No plugin applied - hard to tell what this script is about. Hard to figure out where to make modifications. “Magic” going on. ⬢ Explicit task creation - Duplicated code between modules. Hardcoded task name ⬢ Explicit task dependency - and wrong! ⬢ How do you know it’s part of groovy-all?
17. Example 2: stress tests ⬢ Explicit project type: the intent is clear. It’s not a Groovy module, it’s not a publication, it’s a special kind of module ⬢ Distinct test suite: avoids dirty configuration to disable tests when we mostly don’t want to run them
18. Example 3: the BOM ⬢ Explicit project type: it’s a platform ⬢ Dependency declaration still a bit convoluted (better pattern coming)
19. Major changes 01 allprojects { … } → explicit application of a convention plugin in each project 02 Inline tasks → explicit task implementations in buildSrc 03 apply from: “...” → plugins { id … }
20. Example: get rid of “inline tasks”
21. Does it mean I need to publish plugins?!
22. buildSrc to the rescue! The buildSrc directory should be used to put all custom logic: - Custom tasks - Models (extensions) of different module types - Expose logic via plugins And we have precompiled script plugins!
23. Plugins all the way! Gradle uses composition over inheritance Different aspects of a project can be applied by different plugins Plugins can react to the presence of another plugin
24. Precompiled script plugins Those are lightweight plugins It’s a plugin without all the ceremony Most of custom build logic should move to plugins Can be easily composed
25. Example: the Groovy platform plugin buildSrc/main/groovy/org.apache.groovy-platform.gradle
26. Example: the Groovy platform plugin buildSrc/main/groovy/org.apache.groovy-common.gradle
27. Shared configuration (only applied to the root)
28. Project specific configuration
29. Extensions are created by plugins ⬢ Extensions should make the build script readable ⬢ They talk about the what and they abstract the how ⬢ Creates tasks if needed A build script just applies the plugin and configures the extension The extension does the “hard work”
30. Avoid reaching out to other projects directly
31. Strong encapsulation Do not depend on another project task - E.g “dependsOn rootProject.task(“jar”) This is fragile, unsafe and it breaks encapsulation. Was challenging with the Groovy build which abused reaching out to other projects everywhere. https://docs.gradle.org/current/userguide/cross_project_publications.html
32. Task configuration avoidance
33. Configuration avoidance Before After Description tasks.create(“...”) tasks.register To define a new task tasks.findByName(...) tasks.named(...) { … } Configure an existing task by name tasks.withType(Foo).all { … } tasks.withType(Foo) .configureEach { … } Configures a task by type
34. Configuration avoidance: Groovy DSL gotchas
35. The lazy configuration API Use of Provider and Property: - Fixes most of ordering issues during configuration (get rid of afterEvaluate) - Can carry task dependency information (ideal to get rid of accidental dependencies)
36. The lazy configuration API: derived values
37. Gradle has evolved
38. Test fixtures Tests from different projects need common testing utilities Previously, Groovy did this: - Create utility classes in `src/test` (so they belonged to the test sources) - Add the test runtime classpath of the main project to the test compile classpath of modules
39. The java-test-fixtures plugin Creates a separate source set for test fixtures (src/testFixtures/java) Makes it easy to use from other projects
40. Optional dependencies There are no optional dependencies, only dependencies you need when you use a particular feature
41. Optional dependencies Each feature can have its own dependencies
42. Optional dependencies publication Published as optional dependencies for Maven (POM.xml) and feature variants for Gradle No ugly configuration like before!
43. Let’s talk about publishing Publishing was in a terrible shape - Based on the deprecated publication mechanisms (maven plugin) - Used a lot of internal APIs - Made lots of customizations to the generated POM files - Because of relocated dependencies - Because of the multiple flavors
44. Publishing now Uses the standard maven-publish plugin - Got rid of all internal APIs - Publishes Gradle Module Metadata - Running `./gradlew publish` generates a local repository in `build/repo` which allows seeing what is going to be published Gradle Module Metadata benefits: - Users will get an error if they use both the old Groovy (Codehaus) and new Groovy (Apache) - They can depend on optional features and get the dependencies they need automatically - They get automatic version alignment for free!
45. There’s more we can do
46. Future work Use of Gradle Java Toolchain API - Allows compiling with a JDK, running/testing with another in a declarative manner - Automatic download of JDK! - Removes the need for special tweaks in the build depending on what is used to run Gradle itself But… - Requires Gradle 6.8 for Groovy compilation support - Requires changes to the build pipeline (but it makes it simpler!)
47. Future work Make the root project a “real” root project - groovy-core should become a subproject itself - Allows moving some remaining shared configuration to where it belongs Remove old Ant tasks - Some tasks like `jarjar` still depend on very old Ant tasks - May require some extra work from Gradle
48. Thank you! Special thanks to Paul King for the review of all changes, including bugs in introduced in the process!