State of the art Gradle multi-module builds

Large, complex projects often consist of multiple modules based on particular functionality and logical boundaries. The benefits are obvious: better maintainability, less coupling and high cohesion of code. Gradle provides powerful support for configuring and executing modularized projects. In the course of this session, we will discuss the state of the art techniques for tackling highly-customized multi-project builds. Furthermore, this talk will give an outlook on the features to come.

With the help of demos, we'll focus on the following topics:

* Different approaches for structuring multi-project builds
* Adapting to legacy project structures
* Creating projects dynamically as part of the settings logic
* Externalizing Settings logic into a plugin
* Declaring project dependencies
* Switching between project and artifact dependencies
* Dependency substitution rules
* IDE support
* Optimizing multi-project builds for performance

Transcript

1. State of the Art Gradle
   Multi-Module Builds
   Benjamin Muschko, Gradle Inc.
   

   View Slide

2. Complex software consists of modules
   

   View Slide

3. Benefits of modularization
   Maintainability
   Reusability
   Separation  of  concerns
   

   View Slide

4. Enablers for implementing separation of concerns
   Minimize  coupling
   Maximize  cohesion
   

   View Slide

5. It’s all about your view of the world
   

   View Slide

6. Your view might involve legacy structures
   

   View Slide

7. Configurability is key
   

   View Slide

8. Multi-module or multi-project?
   A
   B
   C
    Component  or  module  is  term  
   originating  from  software  engineering
   

   View Slide

9. Multi-module or multi-project?
   A
   B
   C
    Component  or  module  is  term  
   originating  from  software  engineering
   A
   B
   C
   Project  is  a  term  referring  to  
   the  domain  object  in  Gradle
   

   View Slide

10. Projects can depend on each other
    A
    B
    C
    No  project  depends  on  the    
    output  of  another  project
    

    View Slide

11. Projects can depend on each other
    A
    B
    C
    A
    B
    C
    No  project  depends  on  the    
    output  of  another  project
           Projects  define  one  or  many  
    dependencies  on  other  project(s)
    

    View Slide

12. Project structure - hierarchical or flat
       Hierarchical:  deeply  nested  structure
    A
    B
    C
    D
    E
    

    View Slide

13. Project structure - hierarchical or flat
    A
    Flat:  structure  only  one  level  deep
       Hierarchical:  deeply  nested  structure
    B
    C
    A
    B
    C
    D
    E
    

    View Slide

14. Project vs. binary dependencies
    A
    B
    C
    Different  VCS  repositories,  
    dependencies  are  published  
    artifacts  in  a  binary  repository
    VCS  repo VCS  repo
    VCS  repo
    

    View Slide

15. Project vs. binary dependencies
    A
    B
    C
    A
    B
    C
    Different  VCS  repositories,  
    dependencies  are  published  
    artifacts  in  a  binary  repository
         Projects  usually  live  in    
           same  VCS  repository
    VCS  repo VCS  repo
    VCS  repo
    VCS  repo
    

    View Slide

16. Declaring project dependencies
    Method  Project#project(String)  
    provides  access  to  Project  reference
    

    dependencies {

    compile project(':c')

    }
    build.gradle
    A C
    Creates  JAR  of  project  C  and  adds  it  to  the  compile  classpath  of  project  A.
    

    View Slide

17. Declaring module dependencies
    A  binary  dependency  is  resolved  by  its  
    coordinates  referring  to  a  published  artifact
    

    dependencies {

    compile 'org.gradle:c:1.0'

    }
    build.gradle
    A C
    Binary  repository
    

    View Slide

18. Main role of the settings file
    Declares  projects  that  should  take  part  in  the  
    build
    

    include 'a'

    include 'b:e'

    include 'c'
    settings.gradle
    

    includeFlat 'a'

    includeFlat 'b'

    includeFlat 'c'
    settings.gradle
                         Flat    
    project  structure
           Hierarchical    
    project  structure
    

    View Slide

19. How many? Where to put it?
    Single  settings  file  per  multi-­‐project  build
    Root  project  name  is  derived  of  dir.  name  containing  settings  file.
    A
    B
    C
    settings.gradle
    A
    B
    C
    master settings.gradle
    

    View Slide

20. Resolving the settings file
    2-­‐step  approach  
    1. Search  for  file  in  directory  named  master
    with  same  nesting  level  as  current  directory.  
    2. If  no  file  is  found  search  in  parent  
    directories  starting  from  current  directory.
    

    View Slide

21. Evaluation of logic in settings file
    Initialization  
             Phase
    Configuration  
               Phase
    Execution  
         Phase
    Gradle’s  Build  Lifecycle
    Executed  in  the  first  phase  of  Gradle’s  build  lifecycle.
    settings.gradle
    

    View Slide

22. Configuration injection
    Define  common  configuration  for  1..n  
    projects
    

    View Slide

23. Configuration injection
    Define  common  configuration  for  1..n  
    projects
    

    subprojects {

    ...

    }
    All  subprojects  of  the  project    
    that  defines  this  logic
    

    View Slide

24. Configuration injection
    Define  common  configuration  for  1..n  
    projects
    

    subprojects {

    ...

    }
    

    allprojects {

    ...

    }
    All  subprojects  of  the  project    
    that  defines  this  logic
    All  subprojects  of  the  project    
    that  defines  this  logic  and  the  
    project  itself
    

    View Slide

25. Using configuration injection
    A
    B
    C
    D
    E
    

    View Slide

26. Using configuration injection
    A
    B
    C
    D
    E
    

    subprojects {

    apply plugin: 'java'

    }
    C
    D
    E
    Java
    Java
    

    View Slide

27. Using configuration injection
    A
    B
    C
    D
    E
    

    subprojects {

    apply plugin: 'java'

    }
    

    allprojects {

    apply plugin: 'java'

    }
    C
    D
    E
    C
    D
    E
    Java
    Java
    Java
    Java
    Java
    

    View Slide

28. Filtering based on subproject configuration
    A
    B
    C
    D
    E
    

    def webProjects() {

    subprojects.findAll { subproject ->
    subproject.plugins.hasPlugin('war')
    }

    }

    

    gradle.projectsEvaluated {

    configure(webProjects()) {

    ...

    }

    }
    

    View Slide

29. Filtering based on subproject configuration
    A
    B
    C
    D
    E
    

    def webProjects() {

    subprojects.findAll { subproject ->
    subproject.plugins.hasPlugin('war')
    }

    }

    

    gradle.projectsEvaluated {

    configure(webProjects()) {

    ...

    }

    } Alternative:  
    evaluationDependsOnChildren()
    

    View Slide

30. Task execution based on name matching
    Find  and  execute  task(s)  with  requested  name
    A
    B
    C
    D
    E
    

    ~/dev/myApp/c $ gradle war
    :d:compileJava
    :d:processResources UP-TO-DATE
    :d:classes
    :d:jar
    :e:compileJava
    :e:processResources UP-TO-DATE
    :e:classes
    :e:war
    BUILD SUCCESSFUL
    

    View Slide

31. Task execution based on name matching
    Find  and  execute  task(s)  with  requested  name
    A
    B
    C
    D
    E
    

    ~/dev/myApp/c $ gradle war
    :d:compileJava
    :d:processResources UP-TO-DATE
    :d:classes
    :d:jar
    :e:compileJava
    :e:processResources UP-TO-DATE
    :e:classes
    :e:war
    BUILD SUCCESSFUL
    Project  E  depends  on  D
    

    View Slide

32. Project and task execution path notation
    Full  path  notation  allows  for  being  specific
    A
    B
    C
    D
    E
    

    ~/dev/myApp/c $ gradle e:compileJava
    :d:compileJava
    :d:processResources UP-TO-DATE
    :d:classes
    :d:jar
    :e:compileJava
    BUILD SUCCESSFUL
    Project
    Task
    

    View Slide

33. Selecting project(s)
    A
    B
    C
    D
    E
    

    View Slide

34. Selecting project(s)
    A
    B
    C
    D
    E
    

    project.rootProject A
    

    View Slide

35. Selecting project(s)
    A
    B
    C
    D
    E
    

    project.rootProject A
    

    project.project(':c').children
    D
    E
    

    View Slide

36. Selecting project(s)
    A
    B
    C
    D
    E
    

    project.rootProject A
    

    project.project(':c').children
    D
    E
    

    project.project(':')
       Top-­‐level    
    root  project
    

    View Slide

37. Case-study application
    • High-­‐level  information  about  available  
    Gradle  plugins  
    • Query  Bintray  API  to  retrieve  data  
    • Render  information  on  page
    

    View Slide

38. DEMO
    Showcasing  the  application  functionality
    

    View Slide

39. Single build file
    Only  recommended  for  builds  with  little  logic
    

    project(':a') {

    ...

    }

    

    project(':b') {

    ...

    }

    

    project(':c') {

    ...

    }
    

    include 'a'

    include 'b'

    include 'c'
    build.gradle settings.gradle
    Requires  explicit  use  of  project  method.
    

    View Slide

40. DEMO
    Using  a  single  build  file  to  structure  logic
    

    View Slide

41. Multiple build files
    Best  default  option  for  organizing  build  logic
    A
    B
    C
    build.gradle
    A  project  is  not  required  to  have  a  build.gradle  file.
    

    include 'a'

    include 'b'

    include 'c'
    settings.gradle
    build.gradle
    build.gradle
    build.gradle
    

    View Slide

42. DEMO
    Using  a  multiple  build  files  to  structure  logic
    

    View Slide

43. Hybrid approach
    Avoid  using  this  option  as  it  might  cause  
    potential  maintenance  issues  down  the  road
    A
    B
    C
    build.gradle
    

    project(':b') {

    ...

    }
    Root  project  build.gradle
    

    include 'a'

    include 'b'

    include 'c'
    settings.gradle
    build.gradle
    build.gradle
    

    View Slide

44. DEMO
    Using  a  hybrid  approach  to  structure  logic
    

    View Slide

45. Partial builds of a subproject
    “build  this  project  but  none  of  its  project  dependencies”:  
    --no-rebuild, -a
    “build  &  test  this  project  and  its  project  dependencies”:  
    buildNeeded
    “build  &  test  this  project  and  projects  that  depend  on  
    current  project”:  
    buildDependents
    

    View Slide

46. DEMO
    Using  partial  build  options
    

    View Slide

47. Highly customized Settings configuration
    Typical  scenarios:  
    • Changing  default  project  names  
    • Changing  default  project  directories  
    • Changing  default  build  file  names  
    • Including  projects  through  external  file  
    definition
    

    View Slide

48. Changing project names
    I  inherited  a  legacy  project  structure.  Directory  names  are  
    not  descriptive  enough.  How  do  I  change  the  project  names?
    

    View Slide

49. Changing project names
    I  inherited  a  legacy  project  structure.  Directory  names  are  
    not  descriptive  enough.  How  do  I  change  the  project  names?
    

    rootProject.name = 'myApp'

    include 'shared', 'web', 'client'
    

    rootProject.children.each {

    it.name = 'myApp-' + it.name

    }

    settings.gradle
    

    View Slide

50. DEMO
    Assigning  custom  project  names
    

    View Slide

51. Changing build file names
    It  gets  pretty  confusing  in  my  IDE  if  I  work  on  multiple  
    build.gradle  files  in  parallel.  Can  I  give  them  custom  names?
    

    View Slide

52. Changing build file names
    It  gets  pretty  confusing  in  my  IDE  if  I  work  on  multiple  
    build.gradle  files  in  parallel.  Can  I  give  them  custom  names?
    

    rootProject.name = 'myApp'

    include 'shared', 'web', 'client'
    

    rootProject.children.each {

    it.buildFileName = it.name + '.gradle' - 'myApp'
    }

    settings.gradle
    

    View Slide

53. DEMO
    Assigning  custom  project  file  names
    

    View Slide

54. Including projects by external file definition
    My  list  of  projects  and  their  build  file  names  has  become  very    
    long.  Can  I  externalize  this  definition  into  another  file?
    

    View Slide

55. Including projects by external file definition
    My  list  of  projects  and  their  build  file  names  has  become  very    
    long.  Can  I  externalize  this  definition  into  another  file?
    

    {

    "projects": {

    "shared": {

    "name": "myApp-shared",

    "buildFileName": "shared.gradle"

    },

    "web": {

    "name": "myApp-web",

    "buildFileName": "web.gradle"

    }

    }

    }
    

    View Slide

56. DEMO
    Deriving  project  inclusions  from  JSON  file
    

    View Slide

57. Externalizing Settings logic into a plugin
    As  soon  as  the  logic  becomes  more  complex,  
    write  a  plugin.
    

    package org.gradle.training

    

    import org.gradle.api.Plugin

    import org.gradle.api.initialization.Settings

    

    class DynamicProjectsSettingsPlugin implements Plugin {

    @Override

    void apply(Settings settings) {

    // work on Settings instance

    }

    }
    

    View Slide

58. DEMO
    Writing  a  Settings  plugin
    

    View Slide

59. Working on componentized projects
    A
    B
    C
    VCS  repo
    D
    VCS  repo
    E
    VCS  repo
    F
    VCS  repo
    Deliverable  artifact  comprises  of  all  components.
    

    View Slide

60. Declared module dependencies
    A
    B
    C
    VCS  repo
    D
    VCS  repo
    E
    VCS  repo
    F
    VCS  repo
    

    dependencies {

    compile 'org.gradle:f:1.0'

    }
    

    dependencies {

    compile 'org.gradle:d:2.3'

    }
    

    View Slide

61. Working on multiple components during
    development
    A
    B
    C
    VCS  repo
    D
    VCS  repo
    E
    VCS  repo
    F
    VCS  repo
    Requires  publishing  of  F  and  D.  What  about  E?
    

    View Slide

62. Replacing module with project dependencies
    A
    B
    C
    VCS  repo
    D
    VCS  repo
    E
    VCS  repo
    F
    VCS  repo
    

    dependencies {

    compile project(':f')

    }
    

    dependencies {

    compile project(':d')

    }
    

    View Slide

63. The problem
    Dynamically  switching  between  module  and  
    project  dependencies  
    • Different  scenarios  (e.g.  development  
    environment  vs.  CI).  
    • How  do  I  tell  Gradle  which  one  to  use?  
    • How  do  I  make  it  work  in  the  IDE?
    

    View Slide

64. Open Source approaches
    

    dynamicDependencies {

    compile group: 'com.example', name: 'example', version: '1.0'

    }
    

    dependencies {

    compile elastic('com.example:example:1.0', 'example')

    }
    Prezi  Pride  (https:/
    /github.com/prezi/pride)
    Elastic  Deps    
    (https:/
    /github.com/pniederw/elastic-­‐deps)
    

    View Slide

65. DEMO
    Exploring  a  simple  custom  solution
    

    View Slide

66. Dependency substitution rules
    Gradle’s  API  cousin  to  dependency  resolve  
    rules  
    • Replace  external  with  project  dependency  
    • Replace  project  with  external  dependency
    Gradle  2.5
    

    View Slide

67. Replace external with project dependency
    Scenario:  You  are  working  on  a  module  and  
    want  to  constantly  integrate  the  code  -­‐  “fast  
    turnaround”.
    

    configurations.all {

    resolutionStrategy {

    dependencySubstitution {

    withModule('org.utils:api') {

    useTarget project(':api') 

    }

    }

    }

    }
    

    View Slide

68. Replace project with external dependency
    Scenario:  You  know  the  code  of  the  module  
    hasn’t  changed  -­‐  “build  avoidance”.
    

    configurations.all {

    resolutionStrategy {

    dependencySubstitution {

    withProject(':api') {

    useTarget 'org.utils:api:1.3'
    }

    }

    }

    }
    

    View Slide

69. More down the road…
    Current  functionality  is  a  low-­‐level  API  
    intended  to  build  on  top.
    Low-­‐level  
             API
    DSL
                 IDE  
    capabilities
    

    View Slide

70. DEMO
    Using  dependency  substitution  rules  in  
    practice
    

    View Slide

71. IDE support
    No  support  for  IntelliJ  at  the  moment
    Eclipse  STS  available  through  custom          
    model  used  with  tooling  API
    Planned  support  for  Buildship
    

    View Slide

72. DEMO
    Showcasing  Eclipse  STS  support
    

    View Slide

73. Managing large multi-project builds
    Configuration  time  increases  with  #  projects
    

    View Slide

74. Upgrade to >= Gradle 2.4
    Significant  performance  improvements  in  
    Gradle  internals
    Speeds  up  build  up  to  40%!
    

    View Slide

75. Performance-related command line options
    Building  projects  in  parallel:    
    --parallel, --parallel-threads
    Only  configure  relevant  projects:  
    --configure-on-demand
    Avoid  rebuilding  project  dependencies:  
    --no-rebuild, -a
    

    View Slide

76. Using the new configuration model
    Caching  and  reusing  the  project  model
    A
    B
    C
    D
    E
       Cache
    Work  in  progress
    

    View Slide

77. Modeling projects as SourceSets
    Minimize  time  spend  in  configuration  phase  
    by  not  using  projects/project  dependencies
    A
    B
    C
    D
    E
    SourceSet  A
    SourceSet  B
    SourceSet  C
    SourceSet  D
    SourceSet  E
    compileClasspath
    Requires  a  lot  of  custom  code!
    

    View Slide

78. Thank  You!
    Please  ask  questions…
    https:/
    /www.github.com/bmuschko
    @bmuschko
    

    View Slide