Modernizing the Groovy build

Transcript

1. Modernizing the
   Groovy build
   Cédric Champeau, Gradle Inc.
   @CedricChampeau
   

   View Slide

2. Cédric Champeau
   Gradle, Inc.
   Principal Software Engineer
   JVM Team at Gradle
   Worked on dependency management, performance,
   improving Java ecosystem support
   Former Groovy committer (wrote the static compiler)
   

   View Slide

3. Me and Apache Groovy
   Started as a user, submitted some bugfixes
   Became committer in 2011
   Contributed a number of things mostly around DSL and compile-time metaprogramming:
   - AST transformations
   - Static type checker and static compilation (@CompileStatic)
   - Traits
   - Type checking extensions
   - Template engine with static compilation of templates
   

   View Slide

4. Me and Apache Groovy
   Joined the Gradle team in 2015
   Left the Apache Groovy PMC in march 2019 after disagreement on use of Kotlin
   Contributed build changes… and left around build changes
   Now back to the build!
   

   View Slide

5. History of the Groovy build
   Groovy is an old project (17 years old!)
   Started with Apache Ant
   Attempt to migrate to Apache Maven but never finished/too complicated
   Gradle introduced in 2008 as a parallel build
   Really usable in 2012 after the Gradle 1.0 release
   This is where I started to contribute to the build
   

   View Slide

6. State of the build as of 1st october 2020
   A working Gradle build, including build cache, but:
   ⬢ Written by many different people
   ⬢ With “conventions” inherited from older build tools (Ant, Maven)
   ⬢ With very old releases of Gradle
   ○ Many things were harder to do back then
   ○ Many outdated patterns
   ○ Accidental dependencies
   ○ Lack of guidance
   ○ ...
   

   View Slide

7. Complicating factors
   Groovy is built with Gradle
   Gradle uses a Groovy DSL (or Kotlin ;))
   Groovy provides different flavors of the same
   thing:
   - “Classic” Groovy
   - “Invokedynamic” Groovy
   - “Grooid” Groovy
   

   View Slide

8. Apache Groovy 4 is coming!
   

   View Slide

9. Groovy 4 and Gradle 7
   Groovy 4
   Changes to dependency (GAV) coordinates:
   - From org.codehaus.groovy to org.apache.groovy
   Removal of the classic flavor
   Gradle 7
   Removes the old “maven” plugin in favor of “maven-publish”
   Ideal to leverage some of the dependency management features introduced in Gradle 6!
   

   View Slide

10. Let’s take a quick look at the “old” build
    

    View Slide

11. Idiomatic Gradle
    

    View Slide

12. Idiomatic Gradle goals
    Provide clear guidance on best practices
    Reduce the maintenance burden
    Improve the learning curve
    Improve performance
    Deliver better quality software
    

    View Slide

13. Declarative build scripts
    Most build scripts should only consist of:
    - Applying one or more plugins
    - Declarative configuration
    What matters is the model
    A Java library != a Java application != a Micronaut
    application != a Groovy library != ...
    

    View Slide

14. Types of modules in the Groovy build
    Groovy core
    The “main” project: it’s a library, but also provides the Groovy compiler itself
    Groovy modules
    JSON support, Date utils, Template engines, …
    Aggregating projects
    Groovy “all” (which isn’t Groovy all but a subset of Groovy + Groovy modules)
    The BOM
    The distribution
    Utilities (stress tests, …)
    

    View Slide

15. Example: groovy-dateutil
    

    View Slide

16. What’s wrong with the previous
    pattern?
    ⬢ No plugin applied - hard to tell what
    this script is about. Hard to figure
    out where to make modifications.
    “Magic” going on.
    ⬢ Explicit task creation - Duplicated
    code between modules. Hardcoded
    task name
    ⬢ Explicit task dependency - and
    wrong!
    ⬢ How do you know it’s part of
    groovy-all?
    

    View Slide

17. Example 2: stress tests
    ⬢ Explicit project type: the intent is
    clear. It’s not a Groovy module, it’s
    not a publication, it’s a special kind
    of module
    ⬢ Distinct test suite: avoids dirty
    configuration to disable tests when
    we mostly don’t want to run them
    

    View Slide

18. Example 3: the BOM
    ⬢ Explicit project type: it’s a platform
    ⬢ Dependency declaration still a bit
    convoluted (better pattern coming)
    

    View Slide

19. Major changes
    01 allprojects { … } → explicit application of a convention plugin
    in each project
    02 Inline tasks → explicit task implementations in buildSrc
    03 apply from: “...” → plugins { id … }
    

    View Slide

20. Example: get rid of “inline tasks”
    

    View Slide

21. Does it mean I need to publish plugins?!
    

    View Slide

22. buildSrc to the rescue!
    The buildSrc directory should be used to put all custom
    logic:
    - Custom tasks
    - Models (extensions) of different module types
    - Expose logic via plugins
    And we have precompiled script plugins!
    

    View Slide

23. Plugins all the way!
    Gradle uses composition over inheritance
    Different aspects of a project can be applied by
    different plugins
    Plugins can react to the presence of another plugin
    

    View Slide

24. Precompiled script plugins
    Those are lightweight plugins
    It’s a plugin without all the ceremony
    Most of custom build logic should move to plugins
    Can be easily composed
    

    View Slide

25. Example: the Groovy platform plugin
    buildSrc/main/groovy/org.apache.groovy-platform.gradle
    

    View Slide

26. Example: the Groovy platform plugin
    buildSrc/main/groovy/org.apache.groovy-common.gradle
    

    View Slide

27. Shared configuration (only applied to the root)
    

    View Slide

28. Project specific configuration
    

    View Slide

29. Extensions are created by plugins
    ⬢ Extensions should make the build
    script readable
    ⬢ They talk about the what and they
    abstract the how
    ⬢ Creates tasks if needed
    A build script just applies the
    plugin and configures the
    extension
    The extension does the
    “hard work”
    

    View Slide

30. Avoid reaching out to other projects directly
    

    View Slide

31. Strong encapsulation
    Do not depend on another project task
    - E.g “dependsOn rootProject.task(“jar”)
    This is fragile, unsafe and it breaks encapsulation.
    Was challenging with the Groovy build which abused reaching
    out to other projects everywhere.
    https://docs.gradle.org/current/userguide/cross_project_publications.html
    

    View Slide

32. Task configuration avoidance
    

    View Slide

33. Configuration avoidance
    Before After Description
    tasks.create(“...”) tasks.register To define a new task
    tasks.findByName(...) tasks.named(...) { … }
    Configure an existing task by
    name
    tasks.withType(Foo).all { … }
    tasks.withType(Foo)
    .configureEach { … }
    Configures a task by type
    

    View Slide

34. Configuration avoidance: Groovy DSL gotchas
    

    View Slide

35. The lazy configuration API
    Use of Provider and Property:
    - Fixes most of ordering issues during configuration (get rid of afterEvaluate)
    - Can carry task dependency information (ideal to get rid of accidental dependencies)
    

    View Slide

36. The lazy configuration API: derived values
    

    View Slide

37. Gradle has evolved
    

    View Slide

38. Test fixtures
    Tests from different projects need common testing utilities
    Previously, Groovy did this:
    - Create utility classes in `src/test` (so they belonged to
    the test sources)
    - Add the test runtime classpath of the main project to
    the test compile classpath of modules
    

    View Slide

39. The java-test-fixtures plugin
    Creates a separate source set for test fixtures (src/testFixtures/java)
    Makes it easy to use from other projects
    

    View Slide

40. Optional dependencies
    There are no optional dependencies, only
    dependencies you need when you use a particular
    feature
    

    View Slide

41. Optional dependencies
    Each feature can have its own dependencies
    

    View Slide

42. Optional dependencies publication
    Published as optional dependencies for Maven (POM.xml) and feature variants for Gradle
    No ugly configuration like before!
    

    View Slide

43. Let’s talk about publishing
    Publishing was in a terrible shape
    - Based on the deprecated publication mechanisms (maven plugin)
    - Used a lot of internal APIs
    - Made lots of customizations to the generated POM files
    - Because of relocated dependencies
    - Because of the multiple flavors
    

    View Slide

44. Publishing now
    Uses the standard maven-publish plugin
    - Got rid of all internal APIs
    - Publishes Gradle Module Metadata
    - Running `./gradlew publish` generates a local repository in `build/repo` which allows
    seeing what is going to be published
    Gradle Module Metadata benefits:
    - Users will get an error if they use both the old Groovy (Codehaus) and new Groovy
    (Apache)
    - They can depend on optional features and get the dependencies they need automatically
    - They get automatic version alignment for free!
    

    View Slide

45. There’s more we can do
    

    View Slide

46. Future work
    Use of Gradle Java Toolchain API
    - Allows compiling with a JDK, running/testing with another in a declarative manner
    - Automatic download of JDK!
    - Removes the need for special tweaks in the build depending on what is used to run Gradle
    itself
    But…
    - Requires Gradle 6.8 for Groovy compilation support
    - Requires changes to the build pipeline (but it makes it simpler!)
    

    View Slide

47. Future work
    Make the root project a “real” root project
    - groovy-core should become a subproject itself
    - Allows moving some remaining shared configuration to where it belongs
    Remove old Ant tasks
    - Some tasks like `jarjar` still depend on very old Ant tasks
    - May require some extra work from Gradle
    

    View Slide

48. Thank you!
    Special thanks to Paul King for the
    review of all changes, including bugs in
    introduced in the process!
    

    View Slide