Building a Continuous Delivery pipeline with Gradle and Jenkins

Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.

With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.

Transcript

1. Building  a  Con,nuous  Delivery  pipeline    with  Gradle  and  Jenkins   Benjamin  Muschko  
2. bmuschko   bmuschko   benjamin.muschko@gmail.com   Gradle  in  Ac,on   Java/Groovy  developer   Gradle  contributor   Open  Source  enthusiast   Benjamin  Muschko   Presenta,on  will  be  available  on   hFps://speakerdeck.com/bmuschko  
3. Releases  don’t  have  to  be  painful  
4. Con,nuous  Delivery   Deliver  soKware  fast  and  frequently  
5. #1  Every  commit  can  result  in  a  release   #2  Automate  everything!         #3  Automated  tests  are  essen,al   #4  Done  means  released  
6. Build  pipeline   Automated  manifesta,on  of  delivery  process  
7. Build  quality  in!   Establish  automated  quality  gates  
8. Test   Compile/Unit  Tests   Integra,on  Tests   Code  Analysis   Package/Deploy   UAT   Prod   Acceptance  Tests  
9. !    But     how?  
10. The  “revolu,onary”  sample  applica,on   Internet To Do application Browser Data store Writes Reads
11. Mul,-­‐project  dependencies   Model Web Repository depend depend depend
12. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
13. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
14. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Define  project-­‐specific                            behavior  
15. Project  hierarchy   include 'model' include 'repository' include 'web' settings.gradle Defines  which  projects  are        taking  part  in  the  build   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
16. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Always  use  Wrapper   to  execute  the  build!  
17. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat    Externalize  concerns  into  script  plugins   and  organize  them  in  dedicated  directory   Examples:     ü  Versioning  strategy   ü  Integra,on  and  func,onal  test  setup   ü  Deployment  func,onality   ü  ...        
18. Project  ar,facts   root project model project repository project web project JAR   JAR   WAR   Deployable  ar,fact  
19. Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works                at  a  technical  level  
20. Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works  on  a   func,onal/non-­‐func,onal  level  
21. Stages  in  build  pipeline   Acceptance stage Functional tests Publish Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Trigger  manually   Trigger  manually  
22. Commit  stage:  Compile/unit  tests   Rapid  feedback  (<  5  mins)   Run  on  every  VCS  check-­‐in   Priority:  fix  broken  build   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
23. Commit  stage:  Integra,on  tests   Long  running  tests   Require  environment  setup   Hard  to  maintain   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
24. Separate  tests  in  project  layout   todo model repository web src integTest java main java test java Integra,on  test  Java  sources   Produc,on  Java  sources   Unit  test  Java  sources  
25. Separate  tests  with  SourceSets   
   sourceSets {
 integrationTest {
 java.srcDir file('src/integTest/java')
 resources.srcDir file('src/integTest/resources')
 compileClasspath = sourceSets.main.output + configurations.testRuntime
 runtimeClasspath = output + compileClasspath
 }
 } task integrationTest(type: Test) {
 description = 'Runs the integration tests.'
 group = 'verification'
 testClassesDir = sourceSets.integrationTest.output.classesDir
 classpath = sourceSets.integrationTest.runtimeClasspath
 testResultsDir = file("$testResultsDir/integration")
 }
          Custom  test     results  directory   Set  source  and  resources  directory    Set  compile  and     run,me  classpath   gradlew integrationTest
26. Database  integra,on  tests   stop Database test start Database build Schema integration Test check build ...
27. Database  integra,on  tests   
   apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn startAndPrepareDatabase
 stopDatabase.dependsOn integrationTest
 task databaseIntegrationTest(dependsOn: stopDatabase) check.dependsOn databaseIntegrationTests   stop Database test start Database build Schema integration Test check build ... Separate  complex  setup  logic  into  script  plugin   Integrate  tasks  into  build  lifecycle  
28. Picking  the  “right”  code  coverage  tool   Cobertura   Offline  bytecode      instrumenta,on          Source  code      instrumenta,on   Offline  bytecode      instrumenta,on   On-­‐the-­‐fly  bytecode            instrumenta,on  
29. On-­‐the-­‐fly  bytecode  instrumenta,on   No  modifica,on  to  source  or  bytecode  
30. Code  coverage  with  JaCoCo   
   buildscript {
 repositories {
 mavenCentral()
 }
 
 dependencies {
 classpath 'org.ajoberstar:gradle-jacoco:0.3.0'
 }
 }
 
 apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin
 
 jacoco {
 integrationTestTaskName = 'integrationTest'
 }   Configures  instrumenta,on     for  integra,on  tests  as  well        Add  plugin  to  the   buildscript’s  classpath   Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  
31. Genera,ng  coverage  reports   test integration Test ... ... .exec   jacoco TestReport jacocoIntegration TestReport .exec   .html   .html  
32. Commit  stage:  Code  analysis   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution Perform  code  health  check   Fail  build  for  low  quality   Record  progress  over  ,me  
33. Sta,c  code  analysis  tools   Checkstyle   FindBugs   apply plugin: 'pmd'
 
 pmd {
 ignoreFailures = true
 }
 
 tasks.withType(Pmd) {
 reports {
 xml.enabled = false
 html.enabled = true
 }
 }   apply plugin: 'jdepend’
 
 jdepend {
 toolVersion = '2.9.1'
 ignoreFailures = true
 }   gradlew check
34. Measure  quality  over  ,me  with  Sonar   Sonar database Gradle Sonar Runner root model repo. web analyzes publishes reads / writes Gradle project
35. Applying  the  Sonar  Runner  plugin   apply plugin: 'sonar-runner'
 
 sonarRunner {
 sonarProperties {
 property 'sonar.projectName', 'todo'
 property 'sonar.projectDescription', 'A task management application'
 }
 }
 
 subprojects {
 sonarRunner {
 sonarProperties {
 property 'sonar.sourceEncoding', 'UTF-8'
 }
 }
 }   gradlew sonarRunner
37. Commit  stage:  Assemble  distribu,on   Exclude  env.  configura,on   Include  build  informa,on   Choose  versioning  strategy   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
38. Versioning  strategy   1.0-­‐SNAPSHOT   1.0   during  development   when  released   …the  Maven  way   Change  version  with  Maven  Release  plugin  
39. Versioning  strategy   1.0.134   1.0.134   during  development   when  released   …the  Con,nuous  Delivery  way   1.0.134   Jenkins  build  number   Project  version  number  
40. Versioning  strategy   …implemented  with  Gradle   allprojects {
 apply from: "$rootDir/gradle/versioning.gradle"
 }   todo model repository build.gradle settings.gradle web gradle versioning.gradle Contains  version  implementa,on  
41. Versioning  strategy   …implemented  with  Gradle   ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss')
 
 version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)
 
 class ProjectVersion {
 Integer major
 Integer minor
 String build
 
 ProjectVersion(Integer major, Integer minor, String build) {
 this.major = major
 this.minor = minor
 this.build = build
 }
 
 @Override
 String toString() { String fullVersion = "$major.$minor"
 if(build) {
 fullVersion += ".$build" }
 fullVersion
 } }   Jenkins  Build  Number                Builds  version     String  representa,on  
42. Packaging  the  deployable  ar,fact   project(':web') { apply plugin: 'war' 
 task createBuildInfoFile << {
 def buildInfoFile = new File("$buildDir/build-info.properties")
 Properties props = new Properties()
 props.setProperty('version', project.version.toString())
 props.setProperty('timestamp', project.buildTimestamp)
 props.store(buildInfoFile.newWriter(), null)
 }
 
 war {
 dependsOn createBuildInfoFile
 baseName = 'todo'
 
 from(buildDir) {
 include 'build-info.properties'
 into('WEB-INF/classes')
 }
 } }   Creates  file  containing   build  informa,on   Include  build  info  file   Into  WAR  distribu,on   gradlew assemble
43. Commit  stage:  Publish  binaries   Version  ar,fact(s)   Use  binary  repository   Publish  once,  then  reuse   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
44. Publishing  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33   1.0.34  
45. Defining  build  configura,on   binaryRepository {
 url = 'http://mycompany.bin.repo:8081/artifactory'
 username = 'admin'
 password = 'password'
 name = 'libs-release-local'
 }
 
 environments {
 test {
 server {
 hostname = 'mycompany.test'
 port = 8099
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 }
 
 uat {
 server {
 hostname = 'mycompany.uat'
 port = 8199
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 } 
 ...
 }   Environment-­‐specific        configura,on        Common    configura,on            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es  
46. Reading  build  configura,on   task loadConfiguration {
 def env = project.hasProperty('env') ? project.getProperty('env') : 'test'
 logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy")
 def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())
 
 allprojects {
 ext.config = parsedConfig
 }
 }   Initialization phase Conguration phase Execution phase gradlew –Penv=uat ... Assign  configura,on  to                  extra  property  
47. Using  the  Maven  Publishing  plugin   apply plugin: 'maven-publish'
 ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name”
 
 publishing {
 publications {
 webApp(MavenPublication) {
 from components.web }
 }
 
 repositories {
 maven {
 url fullRepoUrl
 
 credentials {
 username = config.binaryRepository.username
 password = config.binaryRepository.password
 }
 }
 }
 }   Build  repository  URL   from  configura,on   gradlew publish Assign  publica,on  name              and  component  
49. Acceptance  stage:  Retrieve  binaries   Request  versioned  ar,fact   Store  in  temp.  directory   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries
50. Downloading  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33   1.0.34   1.0.34   Test   UAT   Prod  
51. Task  for  downloading  ar,fact   ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 
 task downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {
 ext {
 repoPath = project.group.replaceAll('\\.', '/')
 repoBaseArtifactName = war.baseName
 repoVersion = project.version.toString()
 repoArtifact = war.archiveName
 binaryUrl = "${config.binaryRepository.url}/simple/ ${config.binaryRepository.name}/${repoPath}/ ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"
 }
 
 sourceUrl = binaryUrl
 targetBinary = downloadedArtifact
 }   Target  loca,on  for   downloaded  ar,fact   gradlew downloadBinaryArchive Build  exposed     binary  URL  from   configura,on  
52. Acceptance  stage:  Deploy  binaries   Deployment  on  request   Make  it  a  reliable  process   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Use  process  for  all  envs.  
53. Deploying  to  mul,ple  environments   Test   UAT   Prod   –Penv=prod –Penv=uat –Penv=test
54. Deployment  with  the  Cargo  plugin   cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote
 cargoUndeployRemote {
 onlyIf appContextStatus
 } 
 cargo {
 containerId = 'tomcat7x'
 port = config.server.port
 
 deployable {
 file = downloadedArtifact
 context = config.server.context
 }
 
 remote {
 hostname = config.server.hostname
 username = config.server.username
 password = config.server.password
 }
 }              Download  ar,fact  from  binary   repository  and  undeploy  exis,ng  one    Only  undeploy  if   URL  context  exists   Use  environment-­‐ specific  configura,on   gradlew –Penv=uat cargoDeploy
56. Acceptance  stage:  Func,onal  tests   Test  all  UI  permuta,ons   Test  important  use  cases   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Run  against  different  envs.  
57. In-­‐container  func,onal  tests   functional JettyStop functional TestClasses functional JettyRun functional Test check ... ... 
   task functionalTest(type: Test) {
 ...
 }
 
 task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {
 httpPort = functionalJettyHttpPort
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 contextPath = functionalJettyContextPath
 daemon = true
 }
 
 task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 }
 
 functionalJettyRun.dependsOn functionalTestClasses
 functionalTest.dependsOn functionalJettyRun
 functionalJettyStop.dependsOn functionalTest
 task inContainerFunctionalTest(dependsOn: functionalJettyStop)   Func,onal  test  task   Custom  JeFy  Run  task   Custom  JeFy  Stop  task  
58. Execu,ng  remote  func,onal  tests   ext {
 functionalTestReportDir = file("$testReportDir/functional")
 functionalTestResultsDir = file("$testResultsDir/functional")
 functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")]
 } task remoteFunctionalTest(type: Test) {
 testClassesDir = sourceSets.functionalTest.output.classesDir
 classpath = sourceSets.functionalTest.runtimeClasspath
 testReportDir = functionalTestReportDir
 testResultsDir = functionalTestResultsDir
 systemProperties functionalCommonSystemProperties
 systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/"
 }
   gradlew –Penv=test remoteFunctionalTest Build  URL  from     env.  configura,on   Reuse  setup   proper,es  
59. Going  further:  Capacity  tes,ng   buildscript {
 repositories {
 mavenCentral()
 } 
 dependencies {
 classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"
 }
 }
 
 apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin   ext.loadTestResources = "$projectDir/src/loadTest/resources"
 
 jmeterRun.configure {
 jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]
 jmeterPropertyFile = file("$loadTestResources/jmeter.properties")
 jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"]
 logging.captureStandardError LogLevel.INFO }   gradlew –Penv=uat jmeterRun
60. Let’s  bring  Jenkins  into  play!   Test   UAT   Prod        Deployment   Acceptance  Tests   Publish   Download   Trigger  Build   Pull  Source  Code  
61. Model  pipeline  as  series  of  jobs   Triggered  job  when  change              to  SCM  is  detected  
62. Ini,al  Jenkins  Build  Job  
63. Build  Name  SeFer  Plugin  
64. JaCoCo  Plugin  
65. Parameterized  Trigger  Plugin  
66. Always  use  the  Wrapper!   Gradle  Plugin   Run  clean  task  to  remove              exis,ng  ar,facts  
67. Use  Jenkins  environment  variable   Clearly  iden,fy  a  build   through  an  expressive                    build  name   Build  Name  SeFer  Plugin  
68. Next  job  to  trigger  if            build  is  stable   Defines  build  number   parameter  provided  to   subsequent  jobs   Parameterized  Trigger  Plugin  
69. Archive  all  files   Only  archive  if  build          was  successful   Clone  Workspace  SCM  Plugin  
70. Point  to  separated  test  results   Fail  build  of  quality  gate          criteria  are  not  met   JaCoCo  Plugin   Point  to  JaCoCo  files  as  well        as  source  and  class  files  
71. Reuse  ini,al  workspace   Reuse  ini,al  build  number   Clone  Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  
72. Define  the  target  environment   Downstream  job  that  requires  manual  execu,on   Build  Pipeline  Plugin  
73. Build  Pipeline  Plugin   Visualiza,on  of  chained  pipeline  jobs  
75. > gradle qa! :askQuestions ! ! BUILD SUCCESSFUL! ! Total time: 300 secs