Advanced Dependency Management with Gradle

Slide 1

Slide 1 text

Advanced Dependency Management with Gradle Benjamin Muschko, Gradle Inc.

Slide 2

Slide 2 text

Custom requirements in complex builds Dependency management requires conscious decisions and trade-‐offs… Transitive dependencies Caching strategies Broken module versions Accessing res. artifacts Enterprise requirements Modifying metadata

Slide 3

Slide 3 text

Deep model and API as enabler Dependency management runtime behavior can be fine-‐tuned… • Dependency resolve rules • Component metadata rules • Component selection rules • Artifact Query API

Slide 4

Slide 4 text

Gradle’s dependency cache Enabler for bandwidth efficiency, artifact integrity and dependency resolution performance

Slide 5

Slide 5 text

Opaque cache I updated the Gradle version and now all dependencies are downloaded again. What happened to the cache?

Slide 6

Slide 6 text

Opaque cache I updated the Gradle version and now all dependencies are downloaded again. What happened to the cache? The cache structure is versioned and might change across Gradle versions. Allows for cache optimizations and reorganizations

Slide 7

Slide 7 text

Cache structure Resolved artifacts Metadata

Slide 8

Slide 8 text

Performance features Lazy download of binary artifacts Checksum-‐based download Minimize number of HTTP calls TTL for dynamic/changing modules

Slide 9

Slide 9 text

Other features Store metadata on artifact’s origin Concurrency-‐safe Optimize disk usage

Slide 10

Slide 10 text

Time To Live (TTL) for cached dependencies My build consumes a SNAPSHOT dependency. I know that it changed 10 mins ago but Gradle doesn’t resolve it properly.

Slide 11

Slide 11 text

Time To Live (TTL) for cached dependencies My build consumes a SNAPSHOT dependency. I know that it changed 10 mins ago but Gradle doesn’t resolve it properly. Gradle caching kicks in. Changing and dynamic versions are not checked on the remote repository for 24 hours.

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Fine-tuning dependency caching TTL for changing and dynamic modules can be customized   configurations.all {  resolutionStrategy.cacheDynamicVersionsFor 10, 'minutes'  resolutionStrategy.cacheChangingModulesFor 4, 'hours'  }

Slide 14

Slide 14 text

DEMO Changing the TTL for changing dependencies

Slide 15

Slide 15 text

Resolving a particular Maven SNAPSHOT version All new SNAPSHOT versions of library Y from today are broken. Can I depend on a SNAPSHOT with a timestamp?

Slide 16

Slide 16 text

Resolving a particular Maven SNAPSHOT version All new SNAPSHOT versions of library Y from today are broken. Can I depend on a SNAPSHOT with a timestamp?   dependencies {  compile 'org.gradle.training:mylib:1.0-20150423.152626-8'  }  Gradle 2.4

Slide 17

Slide 17 text

DEMO Changing the TTL for dependencies with dynamic version identifier

Slide 18

Slide 18 text

Cache command line options Working offline: --offline Fresh resolve of cache dependencies: --refresh-dependencies

Slide 19

Slide 19 text

Debugging resolution failures A failed resolution doesn’t give you the cause…   $ gradle dependencies --configuration compile compile - Compile classpath for source set 'main'. +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.12 FAILED

Slide 20

Slide 20 text

Debugging resolution failures Logging levels are your friend…   $ gradle dependencies --configuration compile --info compile - Compile classpath for source set 'main'. Resource missing. [HTTP GET: https://repo1.maven.org/maven2/ javax/mail/mail/1.3.12/mail-1.3.12.pom] Resource missing. [HTTP HEAD: https://repo1.maven.org/maven2/ javax/mail/mail/1.3.12/mail-1.3.12.jar] +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.12 FAILED

Slide 21

Slide 21 text

Resolution works, compilation fails Dependency report looks good…   $ gradle dependencies --configuration compile compile - Compile classpath for source set 'main'. +--- commons-lang:commons-lang:2.5 \--- javax.mail:mail:1.3.1 \--- javax.activation:activation:1.0.2

Slide 22

Slide 22 text

Resolution works, compilation fails Compilation requires compile configuration as input.   $ gradle compileJava FAILURE: Build failed with an exception. * What went wrong: Could not resolve all dependencies for configuration ':compile'. > Could not find mail.jar (javax.mail:mail:1.3.1). Searched in the following locations: https://repo1.maven.org/maven2/javax/mail/mail/1.3.1/ mail-1.3.1.jar

Slide 23

Slide 23 text

Resolution works, compilation fails No JAR file available

Slide 24

Slide 24 text

Using dependency resolve rules We never want our projects to use the version X of library Y. How do we automatically pick version Z for consumers?

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Using dependency resolve rules We never want our projects to use the version X of library Y. How do we automatically pick version Z for consumers? We want to standardize on “good” versions for library Y. How can we recommend this version for consumers? Library Y got new coordinates. Whenever it is requested with old and new coordinates, how do we force the new one?

Slide 27

Slide 27 text

Enforcing an artifact version   // Apply rule to specific configuration  configurations.all {   // Iterate over resolved dependencies  resolutionStrategy.eachDependency { DependencyResolveDetails details ->   // Filter for a specific dependency  if (details.requested.group == 'org.springframework') {   // Force a different version  details.useVersion '3.2.13.RELEASE'  }  }  }

Slide 28

Slide 28 text

DEMO Enforcing an artifact version of Spring framework

Slide 29

Slide 29 text

Recommending artifact versions • Registry for commonly-‐used artifact versions • Flexible version storage data structure • Allow for user to look up default version • Similar concept as Maven POM’s dependencyManagement section

Slide 30

Slide 30 text

Repurposing the artifact version identifier Instead of a concrete version use the self-‐ defined keyword default   dependencies {  compile 'org.springframework:spring-core:default'  compile 'org.springframework:spring-web:default'  }

Slide 31

Slide 31 text

DEMO Recommending an artifact version stored in a simple data structure

Slide 32

Slide 32 text

Using other data structures What happens if the number of default versions exceeds 10 entries?   {  "defaultVersions": [  {  "group": "org.springframework",  "name": "spring-core",  "version": "3.2.13.RELEASE"  },  ... ]  }

Slide 33

Slide 33 text

DEMO Recommending an artifact version stored in a JSON file

Slide 34

Slide 34 text

Recommending artifact versions Want a more fluent DSL that ties into Gradle’s extensibility capabilities?   dependencies {  compile defaultVersion('org.springframework', 'spring-core')  compile defaultVersion('org.springframework', 'spring-web')  }

Slide 35

Slide 35 text

DEMO Extending Gradle’s DSL for providing a default artifact version

Slide 36

Slide 36 text

Open Source plugins Nebula Recommender Plugin (https:/ / github.com/nebula-‐plugins/nebula-‐ dependency-‐recommender) Spring Dependency Management Plugin (https:/ /github.com/spring-‐gradle-‐plugins/ dependency-‐management-‐plugin)

Slide 37

Slide 37 text

Using component metadata rules Customize dependency metadata after module descriptor has been downloaded • Status scheme • Changing status

Slide 38

Slide 38 text

Defining a custom status for published module         ...

Slide 39

Slide 39 text

Using custom status to resolve latest version   repositories {  ivy {  url "file://$projectDir/../repo"  }  }    configurations {  myConf  }    dependencies {  myConf 'org.gradle.training:mylib:latest.snapshot'  }

Slide 40

Slide 40 text

Using custom status to resolve latest version ComponentMetadataDetails describes a resolved component's metadata   dependencies {  components {  all { ComponentMetadataDetails details ->   // modify descriptor metadata  }  }  }

Slide 41

Slide 41 text

DEMO Assigning changing status based on custom status scheme

Slide 42

Slide 42 text

Using component selection rules Reject a module selection based on custom rules e.g. group, name, version   configurations.all.resolutionStrategy {  componentSelection.all { ComponentSelection selection -> // Filter selection candidate  if(selection.candidate.group == 'com.google.collections') { // Reject selection with message  selection.reject("The dependency is deprecated.")  }  }  }

Slide 43

Slide 43 text

DEMO Rejecting legacy module Google Collections with Guava

Slide 44

Slide 44 text

Replacing legacy modules Module coordinates have changed but consumers still have access to new and legacy artifacts   dependencies {  modules {  module('com.google.collections:google-collections') {  replacedBy('com.google.guava:guava')  }  }  }

Slide 45

Slide 45 text

DEMO Replacing legacy module Google Collections with Guava

Slide 46

Slide 46 text

Using the artifact query API Query for raw artifacts resolved by a Configuration Sources artifact Javadoc artifact Metadata artifact (ivy.xml, pom.xml)

Slide 47

Slide 47 text

Resolving selected component IDs   // Use specific configuration  Configuration configuration = project.configurations.compile    // Determine resolution result  ResolutionResult result = configuration.incoming.resolutionResult    // Get all dependencies (resolved and unresolved)  Set extends DependencyResult> allDeps = result.allDependencies    // Filter resolved dependencies and collect component IDs  def componentIds = allDeps.collect { depResult ->  if(dependencyResult instanceof ResolvedDependencyResult) {  return it.selected.id  }  }

Slide 48

Slide 48 text

Creating and executing query   // Get dependency handler  DependencyHandler handler = project.dependencies    // Create artifact resolution query  ArtifactResolutionQuery query = handler.createArtifactResolutionQuery()  .forComponents(componentIds)  .withArtifacts(JvmLibrary, SourcesArtifact, JavadocArtifact)    // Execute artifact resolution query  ArtifactResolutionResult result = query.execute()

Slide 49

Slide 49 text

Using resolved artifacts   // Iterate over resolved components  result.resolvedComponents.each { component ->  // Get sources artifacts  Set sourceArtifacts = component.getArtifacts(SourcesArtifact)    sourceArtifacts.each {   println "Source artifact for ${component.id}: ${it.file}"   }    // Get Javadoc artifacts  Set javadocArtifacts = component.getArtifacts(JavadocArtifact)    ...  }

Slide 50

Slide 50 text

Metadata artifact query types Component type: IvyModule Artifact type: IvyDescriptorArtifact Component type: MavenModule Artifact type: MavenPomArtifact ArtifactResolutionQuery withArtifacts(Class extends Component> componentType, Class extends Artifact>... artifactTypes) ArtifactResolutionQuery API

Slide 51

Slide 51 text

DEMO Inspecting metadata of a resolved POM artifact

Slide 52

Slide 52 text

Building an offline repository Use case: Bundle resolved dependencies and ship them with a distribution Benefits: No need to download artifacts from repository Being able to work offline from the get-‐go

Slide 53

Slide 53 text

DEMO Building an offline Maven repository

Slide 54

Slide 54 text

Solving other real-world Enterprise use cases Common scenarios do not have a one-‐stop solution yet…

Slide 55

Slide 55 text

Slide 56

Slide 56 text

Locking dependency versions My project depends on published modules of other projects. I always want to use the their latest version during develop-‐ ment. What about reproducibility after releasing my project? Scenario: Enforce strong integration pressure Use version identifier latest.release during development. After releasing the project, pin to resolved versions.

Slide 57

Slide 57 text

Nebula dependency lock plugin https:/ /github.com/nebula-‐plugins/gradle-‐dependency-‐lock-‐plugin Binary Repository Gradle build.gradle Nebula dependencies.lock

Slide 58

Slide 58 text

Nebula dependency lock plugin build.gradle   dependencies {  compile 'com.google.guava:guava:14.+'  compile 'commons-lang:commons-lang:latest.release'  }

Slide 59

Slide 59 text

Nebula dependency lock plugin build.gradle dependencies.lock   dependencies {  compile 'com.google.guava:guava:14.+'  compile 'commons-lang:commons-lang:latest.release'  }   {  "com.google.guava:guava": { "locked": "14.0.1", "requested": "14.+" },  "commons-lang:commons-lang": { "locked": "2.5", "requested": "latest.release" }  }

Slide 60

Slide 60 text

Slide 61

Slide 61 text

Slide 62

Slide 62 text

Custom conflict resolution strategy My team is transitioning from build tool X to Gradle. Developers are used to a different version conflict resolution strategy. How can we emulate the “known” strategy? Gradle doesn’t support configuring a custom strategy. It’s always latest version but you can you force module versions. Scenario: Smoother migration between tools

Slide 63

Slide 63 text

Example: Simplified breadth-first Forcing top-‐level dependency versions ( the ones declared in the build script)   configurations.all { config ->  resolutionStrategy {  config.allDependencies.each { dep ->  force "$dep.group:$dep.name:$dep.version"  }  } } Possible, but better to train the team “the Gradle way”!

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Switching between binary & project dependencies We work on multiple detached projects at the same time. Every change to one of the projects needs to be published. To simplify my work I want to use project dependencies. Three options here: Home-‐grown Gradle code, Prezi Pride or dependency substitution rules in Grade 2.5. Scenario: Flexible dependency definitions

Slide 66

Slide 66 text

Open Source approaches   dynamicDependencies {  compile group: 'com.example', name: 'example', version: '1.0'  }   dependencies {  compile elastic('com.example:example:1.0', 'example')  } Prezi Pride (https:/ /github.com/prezi/pride) Elastic Deps (https:/ /github.com/pniederw/elastic-‐deps)

Slide 67

Slide 67 text

Dependency substitution rules Replace a project dependency with an external dependency and vice versa   configurations.all {  resolutionStrategy {  dependencySubstitution {  withModule('com.example:my-module') {  useTarget project(':project1')   }  }  }  } Gradle 2.5

Slide 68

Slide 68 text

Thank You! Please ask questions… https:/ /www.github.com/bmuschko @bmuschko