State of the art Gradle multi-module builds

State of the art Gradle multi-module builds

Large, complex projects often consist of multiple modules based on particular functionality and logical boundaries. The benefits are obvious: better maintainability, less coupling and high cohesion of code. Gradle provides powerful support for configuring and executing modularized projects. In the course of this session, we will discuss the state of the art techniques for tackling highly-customized multi-project builds. Furthermore, this talk will give an outlook on the features to come.

With the help of demos, we'll focus on the following topics:

* Different approaches for structuring multi-project builds
* Adapting to legacy project structures
* Creating projects dynamically as part of the settings logic
* Externalizing Settings logic into a plugin
* Declaring project dependencies
* Switching between project and artifact dependencies
* Dependency substitution rules
* IDE support
* Optimizing multi-project builds for performance

8f2248c6bfcc6df39a2cd8edf4267cb5?s=128

Benjamin Muschko

June 13, 2015
Tweet

Transcript

  1. 8.

    Multi-module or multi-project? A B C  Component  or  module  is

     term   originating  from  software  engineering
  2. 9.

    Multi-module or multi-project? A B C  Component  or  module  is

     term   originating  from  software  engineering A B C Project  is  a  term  referring  to   the  domain  object  in  Gradle
  3. 10.

    Projects can depend on each other A B C No

     project  depends  on  the     output  of  another  project
  4. 11.

    Projects can depend on each other A B C A

    B C No  project  depends  on  the     output  of  another  project        Projects  define  one  or  many   dependencies  on  other  project(s)
  5. 13.

    Project structure - hierarchical or flat A Flat:  structure  only

     one  level  deep    Hierarchical:  deeply  nested  structure B C A B C D E
  6. 14.

    Project vs. binary dependencies A B C Different  VCS  repositories,

      dependencies  are  published   artifacts  in  a  binary  repository VCS  repo VCS  repo VCS  repo
  7. 15.

    Project vs. binary dependencies A B C A B C

    Different  VCS  repositories,   dependencies  are  published   artifacts  in  a  binary  repository      Projects  usually  live  in            same  VCS  repository VCS  repo VCS  repo VCS  repo VCS  repo
  8. 16.

    Declaring project dependencies Method  Project#project(String)   provides  access  to  Project

     reference 
 dependencies {
 compile project(':c')
 } build.gradle A C Creates  JAR  of  project  C  and  adds  it  to  the  compile  classpath  of  project  A.
  9. 17.

    Declaring module dependencies A  binary  dependency  is  resolved  by  its

      coordinates  referring  to  a  published  artifact 
 dependencies {
 compile 'org.gradle:c:1.0'
 } build.gradle A C Binary  repository
  10. 18.

    Main role of the settings file Declares  projects  that  should

     take  part  in  the   build 
 include 'a'
 include 'b:e'
 include 'c' settings.gradle 
 includeFlat 'a'
 includeFlat 'b'
 includeFlat 'c' settings.gradle                      Flat     project  structure        Hierarchical     project  structure
  11. 19.

    How many? Where to put it? Single  settings  file  per

     multi-­‐project  build Root  project  name  is  derived  of  dir.  name  containing  settings  file. A B C settings.gradle A B C master settings.gradle
  12. 20.

    Resolving the settings file 2-­‐step  approach   1. Search  for

     file  in  directory  named  master with  same  nesting  level  as  current  directory.   2. If  no  file  is  found  search  in  parent   directories  starting  from  current  directory.
  13. 21.

    Evaluation of logic in settings file Initialization      

         Phase Configuration              Phase Execution        Phase Gradle’s  Build  Lifecycle Executed  in  the  first  phase  of  Gradle’s  build  lifecycle. settings.gradle
  14. 23.

    Configuration injection Define  common  configuration  for  1..n   projects 


    subprojects {
 ...
 } All  subprojects  of  the  project     that  defines  this  logic
  15. 24.

    Configuration injection Define  common  configuration  for  1..n   projects 


    subprojects {
 ...
 } 
 allprojects {
 ...
 } All  subprojects  of  the  project     that  defines  this  logic All  subprojects  of  the  project     that  defines  this  logic  and  the   project  itself
  16. 26.

    Using configuration injection A B C D E 
 subprojects

    {
 apply plugin: 'java'
 } C D E Java Java
  17. 27.

    Using configuration injection A B C D E 
 subprojects

    {
 apply plugin: 'java'
 } 
 allprojects {
 apply plugin: 'java'
 } C D E C D E Java Java Java Java Java
  18. 28.

    Filtering based on subproject configuration A B C D E

    
 def webProjects() {
 subprojects.findAll { subproject -> subproject.plugins.hasPlugin('war') }
 }
 
 gradle.projectsEvaluated {
 configure(webProjects()) {
 ...
 }
 }
  19. 29.

    Filtering based on subproject configuration A B C D E

    
 def webProjects() {
 subprojects.findAll { subproject -> subproject.plugins.hasPlugin('war') }
 }
 
 gradle.projectsEvaluated {
 configure(webProjects()) {
 ...
 }
 } Alternative:   evaluationDependsOnChildren()
  20. 30.

    Task execution based on name matching Find  and  execute  task(s)

     with  requested  name A B C D E 
 ~/dev/myApp/c $ gradle war :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava :e:processResources UP-TO-DATE :e:classes :e:war BUILD SUCCESSFUL
  21. 31.

    Task execution based on name matching Find  and  execute  task(s)

     with  requested  name A B C D E 
 ~/dev/myApp/c $ gradle war :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava :e:processResources UP-TO-DATE :e:classes :e:war BUILD SUCCESSFUL Project  E  depends  on  D
  22. 32.

    Project and task execution path notation Full  path  notation  allows

     for  being  specific A B C D E 
 ~/dev/myApp/c $ gradle e:compileJava :d:compileJava :d:processResources UP-TO-DATE :d:classes :d:jar :e:compileJava BUILD SUCCESSFUL Project Task
  23. 35.

    Selecting project(s) A B C D E 
 project.rootProject A

    
 project.project(':c').children D E
  24. 36.

    Selecting project(s) A B C D E 
 project.rootProject A

    
 project.project(':c').children D E 
 project.project(':')    Top-­‐level     root  project
  25. 37.

    Case-study application • High-­‐level  information  about  available   Gradle  plugins

      • Query  Bintray  API  to  retrieve  data   • Render  information  on  page
  26. 39.

    Single build file Only  recommended  for  builds  with  little  logic

    
 project(':a') {
 ...
 }
 
 project(':b') {
 ...
 }
 
 project(':c') {
 ...
 } 
 include 'a'
 include 'b'
 include 'c' build.gradle settings.gradle Requires  explicit  use  of  project  method.
  27. 41.

    Multiple build files Best  default  option  for  organizing  build  logic

    A B C build.gradle A  project  is  not  required  to  have  a  build.gradle  file. 
 include 'a'
 include 'b'
 include 'c' settings.gradle build.gradle build.gradle build.gradle
  28. 43.

    Hybrid approach Avoid  using  this  option  as  it  might  cause

      potential  maintenance  issues  down  the  road A B C build.gradle 
 project(':b') {
 ...
 } Root  project  build.gradle 
 include 'a'
 include 'b'
 include 'c' settings.gradle build.gradle build.gradle
  29. 45.

    Partial builds of a subproject “build  this  project  but  none

     of  its  project  dependencies”:   --no-rebuild, -a “build  &  test  this  project  and  its  project  dependencies”:   buildNeeded “build  &  test  this  project  and  projects  that  depend  on   current  project”:   buildDependents
  30. 47.

    Highly customized Settings configuration Typical  scenarios:   • Changing  default

     project  names   • Changing  default  project  directories   • Changing  default  build  file  names   • Including  projects  through  external  file   definition
  31. 48.

    Changing project names I  inherited  a  legacy  project  structure.  Directory

     names  are   not  descriptive  enough.  How  do  I  change  the  project  names?
  32. 49.

    Changing project names I  inherited  a  legacy  project  structure.  Directory

     names  are   not  descriptive  enough.  How  do  I  change  the  project  names? 
 rootProject.name = 'myApp'
 include 'shared', 'web', 'client' 
 rootProject.children.each {
 it.name = 'myApp-' + it.name
 }
 settings.gradle
  33. 51.

    Changing build file names It  gets  pretty  confusing  in  my

     IDE  if  I  work  on  multiple   build.gradle  files  in  parallel.  Can  I  give  them  custom  names?
  34. 52.

    Changing build file names It  gets  pretty  confusing  in  my

     IDE  if  I  work  on  multiple   build.gradle  files  in  parallel.  Can  I  give  them  custom  names? 
 rootProject.name = 'myApp'
 include 'shared', 'web', 'client' 
 rootProject.children.each {
 it.buildFileName = it.name + '.gradle' - 'myApp' }
 settings.gradle
  35. 54.

    Including projects by external file definition My  list  of  projects

     and  their  build  file  names  has  become  very     long.  Can  I  externalize  this  definition  into  another  file?
  36. 55.

    Including projects by external file definition My  list  of  projects

     and  their  build  file  names  has  become  very     long.  Can  I  externalize  this  definition  into  another  file? 
 {
 "projects": {
 "shared": {
 "name": "myApp-shared",
 "buildFileName": "shared.gradle"
 },
 "web": {
 "name": "myApp-web",
 "buildFileName": "web.gradle"
 }
 }
 }
  37. 57.

    Externalizing Settings logic into a plugin As  soon  as  the

     logic  becomes  more  complex,   write  a  plugin. 
 package org.gradle.training
 
 import org.gradle.api.Plugin
 import org.gradle.api.initialization.Settings
 
 class DynamicProjectsSettingsPlugin implements Plugin<Settings> {
 @Override
 void apply(Settings settings) {
 // work on Settings instance
 }
 }
  38. 59.

    Working on componentized projects A B C VCS  repo D

    VCS  repo E VCS  repo F VCS  repo Deliverable  artifact  comprises  of  all  components.
  39. 60.

    Declared module dependencies A B C VCS  repo D VCS

     repo E VCS  repo F VCS  repo 
 dependencies {
 compile 'org.gradle:f:1.0'
 } 
 dependencies {
 compile 'org.gradle:d:2.3'
 }
  40. 61.

    Working on multiple components during development A B C VCS

     repo D VCS  repo E VCS  repo F VCS  repo Requires  publishing  of  F  and  D.  What  about  E?
  41. 62.

    Replacing module with project dependencies A B C VCS  repo

    D VCS  repo E VCS  repo F VCS  repo 
 dependencies {
 compile project(':f')
 } 
 dependencies {
 compile project(':d')
 }
  42. 63.

    The problem Dynamically  switching  between  module  and   project  dependencies

      • Different  scenarios  (e.g.  development   environment  vs.  CI).   • How  do  I  tell  Gradle  which  one  to  use?   • How  do  I  make  it  work  in  the  IDE?
  43. 64.

    Open Source approaches 
 dynamicDependencies {
 compile group: 'com.example', name:

    'example', version: '1.0'
 } 
 dependencies {
 compile elastic('com.example:example:1.0', 'example')
 } Prezi  Pride  (https:/ /github.com/prezi/pride) Elastic  Deps     (https:/ /github.com/pniederw/elastic-­‐deps)
  44. 66.

    Dependency substitution rules Gradle’s  API  cousin  to  dependency  resolve  

    rules   • Replace  external  with  project  dependency   • Replace  project  with  external  dependency Gradle  2.5
  45. 67.

    Replace external with project dependency Scenario:  You  are  working  on

     a  module  and   want  to  constantly  integrate  the  code  -­‐  “fast   turnaround”. 
 configurations.all {
 resolutionStrategy {
 dependencySubstitution {
 withModule('org.utils:api') {
 useTarget project(':api') 
 }
 }
 }
 }
  46. 68.

    Replace project with external dependency Scenario:  You  know  the  code

     of  the  module   hasn’t  changed  -­‐  “build  avoidance”. 
 configurations.all {
 resolutionStrategy {
 dependencySubstitution {
 withProject(':api') {
 useTarget 'org.utils:api:1.3' }
 }
 }
 }
  47. 69.

    More down the road… Current  functionality  is  a  low-­‐level  API

      intended  to  build  on  top. Low-­‐level            API DSL              IDE   capabilities
  48. 71.

    IDE support No  support  for  IntelliJ  at  the  moment Eclipse

     STS  available  through  custom           model  used  with  tooling  API Planned  support  for  Buildship
  49. 74.

    Upgrade to >= Gradle 2.4 Significant  performance  improvements  in  

    Gradle  internals Speeds  up  build  up  to  40%!
  50. 75.

    Performance-related command line options Building  projects  in  parallel:    

    --parallel, --parallel-threads Only  configure  relevant  projects:   --configure-on-demand Avoid  rebuilding  project  dependencies:   --no-rebuild, -a
  51. 76.

    Using the new configuration model Caching  and  reusing  the  project

     model A B C D E    Cache Work  in  progress
  52. 77.

    Modeling projects as SourceSets Minimize  time  spend  in  configuration  phase

      by  not  using  projects/project  dependencies A B C D E SourceSet  A SourceSet  B SourceSet  C SourceSet  D SourceSet  E compileClasspath Requires  a  lot  of  custom  code!