Building  a  Con,nuous  Delivery  pipeline    with  Gradle  and  Jenkins   Benjamin  Muschko  

bmuschko   bmuschko   [email protected]   Gradle  in  Ac,on   Java/Groovy  developer   Gradle  contributor   Open  Source  enthusiast   Benjamin  Muschko   Presenta,on  will  be  available  on   hFps://speakerdeck.com/bmuschko  

Releases  don’t  have  to  be  painful  

Con,nuous  Delivery   Deliver  soKware  fast  and  frequently  

#1  Every  commit  can  result  in  a  release   #2  Automate  everything!         #3  Automated  tests  are  essen,al   #4  Done  means  released  

Build  pipeline   Automated  manifesta,on  of  delivery  process  

Build  quality  in!   Establish  automated  quality  gates  

Test   Compile/Unit  Tests   Integra,on  Tests   Code  Analysis   Package/Deploy   UAT   Prod   Acceptance  Tests  

!    But     how?  

The  “revolu,onary”  sample  applica,on   Internet To Do application Browser Data store Writes Reads

Mul,-­‐project  dependencies   Model Web Repository depend depend depend

Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat

Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat

Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Define  project-­‐specific                            behavior  

Project  hierarchy   include 'model' include 'repository' include 'web' settings.gradle Defines  which  projects  are        taking  part  in  the  build   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat

Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Always  use  Wrapper   to  execute  the  build!  

Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat    Externalize  concerns  into  script  plugins   and  organize  them  in  dedicated  directory   Examples:     ü  Versioning  strategy   ü  Integra,on  and  func,onal  test  setup   ü  Deployment  func,onality   ü  ...        

Project  ar,facts   root project model project repository project web project JAR   JAR   WAR   Deployable  ar,fact  

Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works                at  a  technical  level  

Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works  on  a   func,onal/non-­‐func,onal  level  

Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Trigger  manually   Trigger  manually  

Commit  stage:  Compile/unit  tests   Rapid  feedback  (<  5  mins)   Run  on  every  VCS  check-­‐in   Priority:  fix  broken  build   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution

Commit  stage:  Integra,on  tests   Long  running  tests   Require  environment  setup   Hard  to  maintain   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution

Separate  tests  in  project  layout   todo model repository web src integTest java main java test java Integra,on  test  Java  sources   Produc,on  Java  sources   Unit  test  Java  sources  

Separate  tests  with  SourceSets   
   sourceSets {
 integrationTest {
 java.srcDir file('src/integTest/java')
 resources.srcDir file('src/integTest/resources')
 compileClasspath = sourceSets.main.output + configurations.testRuntime
 runtimeClasspath = output + compileClasspath
 }
 } task integrationTest(type: Test) {
 description = 'Runs the integration tests.'
 group = 'verification'
 testClassesDir = sourceSets.integrationTest.output.classesDir
 classpath = sourceSets.integrationTest.runtimeClasspath
 testResultsDir = file("$testResultsDir/integration")
 }
          Custom  test     results  directory   Set  source  and  resources  directory    Set  compile  and     run,me  classpath   gradlew integrationTest

Database  integra,on  tests   stop Database test start Database build Schema integration Test check build ...

Database  integra,on  tests   
   apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn startAndPrepareDatabase
 stopDatabase.dependsOn integrationTest
 task databaseIntegrationTest(dependsOn: stopDatabase) check.dependsOn databaseIntegrationTests   stop Database test start Database build Schema integration Test check build ... Separate  complex  setup  logic  into  script  plugin   Integrate  tasks  into  build  lifecycle  

Picking  the  “right”  code  coverage  tool   Cobertura   Offline  bytecode      instrumenta,on          Source  code      instrumenta,on   Offline  bytecode      instrumenta,on   On-­‐the-­‐fly  bytecode            instrumenta,on  

On-­‐the-­‐fly  bytecode  instrumenta,on   No  modifica,on  to  source  or  bytecode  

Code  coverage  with  JaCoCo   
   buildscript {
 repositories {
 mavenCentral()
 }
 
 dependencies {
 classpath 'org.ajoberstar:gradle-jacoco:0.3.0'
 }
 }
 
 apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin
 
 jacoco {
 integrationTestTaskName = 'integrationTest'
 }   Configures  instrumenta,on     for  integra,on  tests  as  well        Add  plugin  to  the   buildscript’s  classpath   Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  

Genera,ng  coverage  reports   test integration Test ... ... .exec   jacoco TestReport jacocoIntegration TestReport .exec   .html   .html  

Commit  stage:  Code  analysis   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution Perform  code  health  check   Fail  build  for  low  quality   Record  progress  over  ,me  

Sta,c  code  analysis  tools   Checkstyle   FindBugs   apply plugin: 'pmd'
 
 pmd {
 ignoreFailures = true
 }
 
 tasks.withType(Pmd) {
 reports {
 xml.enabled = false
 html.enabled = true
 }
 }   apply plugin: 'jdepend’
 
 jdepend {
 toolVersion = '2.9.1'
 ignoreFailures = true
 }   gradlew check

Measure  quality  over  ,me  with  Sonar   Sonar database Gradle Sonar Runner root model repo. web analyzes publishes reads / writes Gradle project

Applying  the  Sonar  Runner  plugin   apply plugin: 'sonar-runner'
 
 sonarRunner {
 sonarProperties {
 property 'sonar.projectName', 'todo'
 property 'sonar.projectDescription', 'A task management application'
 }
 }
 
 subprojects {
 sonarRunner {
 sonarProperties {
 property 'sonar.sourceEncoding', 'UTF-8'
 }
 }
 }   gradlew sonarRunner

No content

Commit  stage:  Assemble  distribu,on   Exclude  env.  configura,on   Include  build  informa,on   Choose  versioning  strategy   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution

Versioning  strategy   1.0-­‐SNAPSHOT   1.0   during  development   when  released   …the  Maven  way   Change  version  with  Maven  Release  plugin  

Versioning  strategy   1.0.134   1.0.134   during  development   when  released   …the  Con,nuous  Delivery  way   1.0.134   Jenkins  build  number   Project  version  number  

Versioning  strategy   …implemented  with  Gradle   allprojects {
 apply from: "$rootDir/gradle/versioning.gradle"
 }   todo model repository build.gradle settings.gradle web gradle versioning.gradle Contains  version  implementa,on  

Versioning  strategy   …implemented  with  Gradle   ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss')
 
 version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)
 
 class ProjectVersion {
 Integer major
 Integer minor
 String build
 
 ProjectVersion(Integer major, Integer minor, String build) {
 this.major = major
 this.minor = minor
 this.build = build
 }
 
 @Override
 String toString() { String fullVersion = "$major.$minor"
 if(build) {
 fullVersion += ".$build" }
 fullVersion
 } }   Jenkins  Build  Number                Builds  version     String  representa,on  

Packaging  the  deployable  ar,fact   project(':web') { apply plugin: 'war' 
 task createBuildInfoFile << {
 def buildInfoFile = new File("$buildDir/build-info.properties")
 Properties props = new Properties()
 props.setProperty('version', project.version.toString())
 props.setProperty('timestamp', project.buildTimestamp)
 props.store(buildInfoFile.newWriter(), null)
 }
 
 war {
 dependsOn createBuildInfoFile
 baseName = 'todo'
 
 from(buildDir) {
 include 'build-info.properties'
 into('WEB-INF/classes')
 }
 } }   Creates  file  containing   build  informa,on   Include  build  info  file   Into  WAR  distribu,on   gradlew assemble

Commit  stage:  Publish  binaries   Version  ar,fact(s)   Use  binary  repository   Publish  once,  then  reuse   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution

Publishing  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33   1.0.34  

Defining  build  configura,on   binaryRepository {
 url = 'http://mycompany.bin.repo:8081/artifactory'
 username = 'admin'
 password = 'password'
 name = 'libs-release-local'
 }
 
 environments {
 test {
 server {
 hostname = 'mycompany.test'
 port = 8099
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 }
 
 uat {
 server {
 hostname = 'mycompany.uat'
 port = 8199
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 } 
 ...
 }   Environment-­‐specific        configura,on        Common    configura,on            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es  

Reading  build  configura,on   task loadConfiguration {
 def env = project.hasProperty('env') ? project.getProperty('env') : 'test'
 logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy")
 def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())
 
 allprojects {
 ext.config = parsedConfig
 }
 }   Initialization phase Conguration phase Execution phase gradlew –Penv=uat ... Assign  configura,on  to                  extra  property  

Using  the  Maven  Publishing  plugin   apply plugin: 'maven-publish'
 ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name”
 
 publishing {
 publications {
 webApp(MavenPublication) {
 from components.web }
 }
 
 repositories {
 maven {
 url fullRepoUrl
 
 credentials {
 username = config.binaryRepository.username
 password = config.binaryRepository.password
 }
 }
 }
 }   Build  repository  URL   from  configura,on   gradlew publish Assign  publica,on  name              and  component  

No content

Acceptance  stage:  Retrieve  binaries   Request  versioned  ar,fact   Store  in  temp.  directory   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries

Downloading  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33   1.0.34   1.0.34   Test   UAT   Prod  

Task  for  downloading  ar,fact   ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 
 task downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {
 ext {
 repoPath = project.group.replaceAll('\\.', '/')
 repoBaseArtifactName = war.baseName
 repoVersion = project.version.toString()
 repoArtifact = war.archiveName
 binaryUrl = "${config.binaryRepository.url}/simple/ ${config.binaryRepository.name}/${repoPath}/ ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"
 }
 
 sourceUrl = binaryUrl
 targetBinary = downloadedArtifact
 }   Target  loca,on  for   downloaded  ar,fact   gradlew downloadBinaryArchive Build  exposed     binary  URL  from   configura,on  

Acceptance  stage:  Deploy  binaries   Deployment  on  request   Make  it  a  reliable  process   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Use  process  for  all  envs.  

Deploying  to  mul,ple  environments   Test   UAT   Prod   –Penv=prod –Penv=uat –Penv=test

Deployment  with  the  Cargo  plugin   cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote
 cargoUndeployRemote {
 onlyIf appContextStatus
 } 
 cargo {
 containerId = 'tomcat7x'
 port = config.server.port
 
 deployable {
 file = downloadedArtifact
 context = config.server.context
 }
 
 remote {
 hostname = config.server.hostname
 username = config.server.username
 password = config.server.password
 }
 }              Download  ar,fact  from  binary   repository  and  undeploy  exis,ng  one    Only  undeploy  if   URL  context  exists   Use  environment-­‐ specific  configura,on   gradlew –Penv=uat cargoDeploy

No content

Acceptance  stage:  Func,onal  tests   Test  all  UI  permuta,ons   Test  important  use  cases   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Run  against  different  envs.  

In-­‐container  func,onal  tests   functional JettyStop functional TestClasses functional JettyRun functional Test check ... ... 
   task functionalTest(type: Test) {
 ...
 }
 
 task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {
 httpPort = functionalJettyHttpPort
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 contextPath = functionalJettyContextPath
 daemon = true
 }
 
 task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 }
 
 functionalJettyRun.dependsOn functionalTestClasses
 functionalTest.dependsOn functionalJettyRun
 functionalJettyStop.dependsOn functionalTest
 task inContainerFunctionalTest(dependsOn: functionalJettyStop)   Func,onal  test  task   Custom  JeFy  Run  task   Custom  JeFy  Stop  task  

Execu,ng  remote  func,onal  tests   ext {
 functionalTestReportDir = file("$testReportDir/functional")
 functionalTestResultsDir = file("$testResultsDir/functional")
 functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")]
 } task remoteFunctionalTest(type: Test) {
 testClassesDir = sourceSets.functionalTest.output.classesDir
 classpath = sourceSets.functionalTest.runtimeClasspath
 testReportDir = functionalTestReportDir
 testResultsDir = functionalTestResultsDir
 systemProperties functionalCommonSystemProperties
 systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/"
 }
   gradlew –Penv=test remoteFunctionalTest Build  URL  from     env.  configura,on   Reuse  setup   proper,es  

Going  further:  Capacity  tes,ng   buildscript {
 repositories {
 mavenCentral()
 } 
 dependencies {
 classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"
 }
 }
 
 apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin   ext.loadTestResources = "$projectDir/src/loadTest/resources"
 
 jmeterRun.configure {
 jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]
 jmeterPropertyFile = file("$loadTestResources/jmeter.properties")
 jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"]
 logging.captureStandardError LogLevel.INFO }   gradlew –Penv=uat jmeterRun

Let’s  bring  Jenkins  into  play!   Test   UAT   Prod        Deployment   Acceptance  Tests   Publish   Download   Trigger  Build   Pull  Source  Code  

Model  pipeline  as  series  of  jobs   Triggered  job  when  change              to  SCM  is  detected  

Ini,al  Jenkins  Build  Job  

Build  Name  SeFer  Plugin  

JaCoCo  Plugin  

Parameterized  Trigger  Plugin  

Always  use  the  Wrapper!   Gradle  Plugin   Run  clean  task  to  remove              exis,ng  ar,facts  

Use  Jenkins  environment  variable   Clearly  iden,fy  a  build   through  an  expressive                    build  name   Build  Name  SeFer  Plugin  

Next  job  to  trigger  if            build  is  stable   Defines  build  number   parameter  provided  to   subsequent  jobs   Parameterized  Trigger  Plugin  

Archive  all  files   Only  archive  if  build          was  successful   Clone  Workspace  SCM  Plugin  

Point  to  separated  test  results   Fail  build  of  quality  gate          criteria  are  not  met   JaCoCo  Plugin   Point  to  JaCoCo  files  as  well        as  source  and  class  files  

Reuse  ini,al  workspace   Reuse  ini,al  build  number   Clone  Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  

Define  the  target  environment   Downstream  job  that  requires  manual  execu,on   Build  Pipeline  Plugin  

Build  Pipeline  Plugin   Visualiza,on  of  chained  pipeline  jobs  

No content

> gradle qa! :askQuestions ! ! BUILD SUCCESSFUL! ! Total time: 300 secs