Building a Continuous Delivery pipeline with Gradle and Jenkins

Building a Continuous Delivery pipeline with Gradle and Jenkins

Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.

With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.

8f2248c6bfcc6df39a2cd8edf4267cb5?s=128

Benjamin Muschko

June 13, 2013
Tweet

Transcript

  1. Building  a  Con,nuous  Delivery  pipeline    with  Gradle  and  Jenkins

      Benjamin  Muschko  
  2. bmuschko   bmuschko   benjamin.muschko@gmail.com   Gradle  in  Ac,on  

    Java/Groovy  developer   Gradle  contributor   Open  Source  enthusiast   Benjamin  Muschko   Presenta,on  will  be  available  on   hFps://speakerdeck.com/bmuschko  
  3. Releases  don’t  have  to  be  painful  

  4. Con,nuous  Delivery   Deliver  soKware  fast  and  frequently  

  5. #1  Every  commit  can  result  in  a  release   #2

     Automate  everything!         #3  Automated  tests  are  essen,al   #4  Done  means  released  
  6. Build  pipeline   Automated  manifesta,on  of  delivery  process  

  7. Build  quality  in!   Establish  automated  quality  gates  

  8. Test   Compile/Unit  Tests   Integra,on  Tests   Code  Analysis

      Package/Deploy   UAT   Prod   Acceptance  Tests  
  9. !    But     how?  

  10. The  “revolu,onary”  sample  applica,on   Internet To Do application Browser

    Data store Writes Reads
  11. Mul,-­‐project  dependencies   Model Web Repository depend depend depend

  12. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  13. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  14. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Define  project-­‐specific                            behavior  
  15. Project  hierarchy   include 'model' include 'repository' include 'web' settings.gradle

    Defines  which  projects  are        taking  part  in  the  build   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  16. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Always  use  Wrapper   to  execute  the  build!  
  17. Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat    Externalize  concerns  into  script  plugins   and  organize  them  in  dedicated  directory   Examples:     ü  Versioning  strategy   ü  Integra,on  and  func,onal  test  setup   ü  Deployment  func,onality   ü  ...        
  18. Project  ar,facts   root project model project repository project web

    project JAR   JAR   WAR   Deployable  ar,fact  
  19. Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works                at  a  technical  level  
  20. Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works  on  a   func,onal/non-­‐func,onal  level  
  21. Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Trigger  manually   Trigger  manually  
  22. Commit  stage:  Compile/unit  tests   Rapid  feedback  (<  5  mins)

      Run  on  every  VCS  check-­‐in   Priority:  fix  broken  build   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  23. Commit  stage:  Integra,on  tests   Long  running  tests   Require

     environment  setup   Hard  to  maintain   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  24. Separate  tests  in  project  layout   todo model repository web

    src integTest java main java test java Integra,on  test  Java  sources   Produc,on  Java  sources   Unit  test  Java  sources  
  25. Separate  tests  with  SourceSets   
   sourceSets {
 integrationTest

    {
 java.srcDir file('src/integTest/java')
 resources.srcDir file('src/integTest/resources')
 compileClasspath = sourceSets.main.output + configurations.testRuntime
 runtimeClasspath = output + compileClasspath
 }
 } task integrationTest(type: Test) {
 description = 'Runs the integration tests.'
 group = 'verification'
 testClassesDir = sourceSets.integrationTest.output.classesDir
 classpath = sourceSets.integrationTest.runtimeClasspath
 testResultsDir = file("$testResultsDir/integration")
 }
          Custom  test     results  directory   Set  source  and  resources  directory    Set  compile  and     run,me  classpath   gradlew integrationTest
  26. Database  integra,on  tests   stop Database test start Database build

    Schema integration Test check build ...
  27. Database  integra,on  tests   
   apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn

    startAndPrepareDatabase
 stopDatabase.dependsOn integrationTest
 task databaseIntegrationTest(dependsOn: stopDatabase) check.dependsOn databaseIntegrationTests   stop Database test start Database build Schema integration Test check build ... Separate  complex  setup  logic  into  script  plugin   Integrate  tasks  into  build  lifecycle  
  28. Picking  the  “right”  code  coverage  tool   Cobertura   Offline

     bytecode      instrumenta,on          Source  code      instrumenta,on   Offline  bytecode      instrumenta,on   On-­‐the-­‐fly  bytecode            instrumenta,on  
  29. On-­‐the-­‐fly  bytecode  instrumenta,on   No  modifica,on  to  source  or  bytecode

     
  30. Code  coverage  with  JaCoCo   
   buildscript {
 repositories

    {
 mavenCentral()
 }
 
 dependencies {
 classpath 'org.ajoberstar:gradle-jacoco:0.3.0'
 }
 }
 
 apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin
 
 jacoco {
 integrationTestTaskName = 'integrationTest'
 }   Configures  instrumenta,on     for  integra,on  tests  as  well        Add  plugin  to  the   buildscript’s  classpath   Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  
  31. Genera,ng  coverage  reports   test integration Test ... ... .exec

      jacoco TestReport jacocoIntegration TestReport .exec   .html   .html  
  32. Commit  stage:  Code  analysis   Publish Binaries Commit stage Compile

    Unit Tests Integration Tests Code Analysis Assemble Distribution Perform  code  health  check   Fail  build  for  low  quality   Record  progress  over  ,me  
  33. Sta,c  code  analysis  tools   Checkstyle   FindBugs   apply

    plugin: 'pmd'
 
 pmd {
 ignoreFailures = true
 }
 
 tasks.withType(Pmd) {
 reports {
 xml.enabled = false
 html.enabled = true
 }
 }   apply plugin: 'jdepend’
 
 jdepend {
 toolVersion = '2.9.1'
 ignoreFailures = true
 }   gradlew check
  34. Measure  quality  over  ,me  with  Sonar   Sonar database Gradle

    Sonar Runner root model repo. web analyzes publishes reads / writes Gradle project
  35. Applying  the  Sonar  Runner  plugin   apply plugin: 'sonar-runner'
 


    sonarRunner {
 sonarProperties {
 property 'sonar.projectName', 'todo'
 property 'sonar.projectDescription', 'A task management application'
 }
 }
 
 subprojects {
 sonarRunner {
 sonarProperties {
 property 'sonar.sourceEncoding', 'UTF-8'
 }
 }
 }   gradlew sonarRunner
  36. None
  37. Commit  stage:  Assemble  distribu,on   Exclude  env.  configura,on   Include

     build  informa,on   Choose  versioning  strategy   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  38. Versioning  strategy   1.0-­‐SNAPSHOT   1.0   during  development  

    when  released   …the  Maven  way   Change  version  with  Maven  Release  plugin  
  39. Versioning  strategy   1.0.134   1.0.134   during  development  

    when  released   …the  Con,nuous  Delivery  way   1.0.134   Jenkins  build  number   Project  version  number  
  40. Versioning  strategy   …implemented  with  Gradle   allprojects {
 apply

    from: "$rootDir/gradle/versioning.gradle"
 }   todo model repository build.gradle settings.gradle web gradle versioning.gradle Contains  version  implementa,on  
  41. Versioning  strategy   …implemented  with  Gradle   ext.buildTimestamp = new

    Date().format('yyyy-MM-dd HH:mm:ss')
 
 version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)
 
 class ProjectVersion {
 Integer major
 Integer minor
 String build
 
 ProjectVersion(Integer major, Integer minor, String build) {
 this.major = major
 this.minor = minor
 this.build = build
 }
 
 @Override
 String toString() { String fullVersion = "$major.$minor"
 if(build) {
 fullVersion += ".$build" }
 fullVersion
 } }   Jenkins  Build  Number                Builds  version     String  representa,on  
  42. Packaging  the  deployable  ar,fact   project(':web') { apply plugin: 'war'

    
 task createBuildInfoFile << {
 def buildInfoFile = new File("$buildDir/build-info.properties")
 Properties props = new Properties()
 props.setProperty('version', project.version.toString())
 props.setProperty('timestamp', project.buildTimestamp)
 props.store(buildInfoFile.newWriter(), null)
 }
 
 war {
 dependsOn createBuildInfoFile
 baseName = 'todo'
 
 from(buildDir) {
 include 'build-info.properties'
 into('WEB-INF/classes')
 }
 } }   Creates  file  containing   build  informa,on   Include  build  info  file   Into  WAR  distribu,on   gradlew assemble
  43. Commit  stage:  Publish  binaries   Version  ar,fact(s)   Use  binary

     repository   Publish  once,  then  reuse   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  44. Publishing  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33

      1.0.34  
  45. Defining  build  configura,on   binaryRepository {
 url = 'http://mycompany.bin.repo:8081/artifactory'
 username

    = 'admin'
 password = 'password'
 name = 'libs-release-local'
 }
 
 environments {
 test {
 server {
 hostname = 'mycompany.test'
 port = 8099
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 }
 
 uat {
 server {
 hostname = 'mycompany.uat'
 port = 8199
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 } 
 ...
 }   Environment-­‐specific        configura,on        Common    configura,on            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es  
  46. Reading  build  configura,on   task loadConfiguration {
 def env =

    project.hasProperty('env') ? project.getProperty('env') : 'test'
 logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy")
 def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())
 
 allprojects {
 ext.config = parsedConfig
 }
 }   Initialization phase Conguration phase Execution phase gradlew –Penv=uat ... Assign  configura,on  to                  extra  property  
  47. Using  the  Maven  Publishing  plugin   apply plugin: 'maven-publish'
 ext.fullRepoUrl

    = "$config.binaryRepository.url/$config.binaryRepository.name”
 
 publishing {
 publications {
 webApp(MavenPublication) {
 from components.web }
 }
 
 repositories {
 maven {
 url fullRepoUrl
 
 credentials {
 username = config.binaryRepository.username
 password = config.binaryRepository.password
 }
 }
 }
 }   Build  repository  URL   from  configura,on   gradlew publish Assign  publica,on  name              and  component  
  48. None
  49. Acceptance  stage:  Retrieve  binaries   Request  versioned  ar,fact   Store

     in  temp.  directory   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries
  50. Downloading  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33

      1.0.34   1.0.34   Test   UAT   Prod  
  51. Task  for  downloading  ar,fact   ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 
 task

    downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {
 ext {
 repoPath = project.group.replaceAll('\\.', '/')
 repoBaseArtifactName = war.baseName
 repoVersion = project.version.toString()
 repoArtifact = war.archiveName
 binaryUrl = "${config.binaryRepository.url}/simple/ ${config.binaryRepository.name}/${repoPath}/ ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"
 }
 
 sourceUrl = binaryUrl
 targetBinary = downloadedArtifact
 }   Target  loca,on  for   downloaded  ar,fact   gradlew downloadBinaryArchive Build  exposed     binary  URL  from   configura,on  
  52. Acceptance  stage:  Deploy  binaries   Deployment  on  request   Make

     it  a  reliable  process   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Use  process  for  all  envs.  
  53. Deploying  to  mul,ple  environments   Test   UAT   Prod

      –Penv=prod –Penv=uat –Penv=test
  54. Deployment  with  the  Cargo  plugin   cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote
 cargoUndeployRemote

    {
 onlyIf appContextStatus
 } 
 cargo {
 containerId = 'tomcat7x'
 port = config.server.port
 
 deployable {
 file = downloadedArtifact
 context = config.server.context
 }
 
 remote {
 hostname = config.server.hostname
 username = config.server.username
 password = config.server.password
 }
 }              Download  ar,fact  from  binary   repository  and  undeploy  exis,ng  one    Only  undeploy  if   URL  context  exists   Use  environment-­‐ specific  configura,on   gradlew –Penv=uat cargoDeploy
  55. None
  56. Acceptance  stage:  Func,onal  tests   Test  all  UI  permuta,ons  

    Test  important  use  cases   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Run  against  different  envs.  
  57. In-­‐container  func,onal  tests   functional JettyStop functional TestClasses functional JettyRun

    functional Test check ... ... 
   task functionalTest(type: Test) {
 ...
 }
 
 task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {
 httpPort = functionalJettyHttpPort
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 contextPath = functionalJettyContextPath
 daemon = true
 }
 
 task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 }
 
 functionalJettyRun.dependsOn functionalTestClasses
 functionalTest.dependsOn functionalJettyRun
 functionalJettyStop.dependsOn functionalTest
 task inContainerFunctionalTest(dependsOn: functionalJettyStop)   Func,onal  test  task   Custom  JeFy  Run  task   Custom  JeFy  Stop  task  
  58. Execu,ng  remote  func,onal  tests   ext {
 functionalTestReportDir = file("$testReportDir/functional")


    functionalTestResultsDir = file("$testResultsDir/functional")
 functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")]
 } task remoteFunctionalTest(type: Test) {
 testClassesDir = sourceSets.functionalTest.output.classesDir
 classpath = sourceSets.functionalTest.runtimeClasspath
 testReportDir = functionalTestReportDir
 testResultsDir = functionalTestResultsDir
 systemProperties functionalCommonSystemProperties
 systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/"
 }
   gradlew –Penv=test remoteFunctionalTest Build  URL  from     env.  configura,on   Reuse  setup   proper,es  
  59. Going  further:  Capacity  tes,ng   buildscript {
 repositories {
 mavenCentral()


    } 
 dependencies {
 classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"
 }
 }
 
 apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin   ext.loadTestResources = "$projectDir/src/loadTest/resources"
 
 jmeterRun.configure {
 jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]
 jmeterPropertyFile = file("$loadTestResources/jmeter.properties")
 jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"]
 logging.captureStandardError LogLevel.INFO }   gradlew –Penv=uat jmeterRun
  60. Let’s  bring  Jenkins  into  play!   Test   UAT  

    Prod        Deployment   Acceptance  Tests   Publish   Download   Trigger  Build   Pull  Source  Code  
  61. Model  pipeline  as  series  of  jobs   Triggered  job  when

     change              to  SCM  is  detected  
  62. Ini,al  Jenkins  Build  Job  

  63. Build  Name  SeFer  Plugin  

  64. JaCoCo  Plugin  

  65. Parameterized  Trigger  Plugin  

  66. Always  use  the  Wrapper!   Gradle  Plugin   Run  clean

     task  to  remove              exis,ng  ar,facts  
  67. Use  Jenkins  environment  variable   Clearly  iden,fy  a  build  

    through  an  expressive                    build  name   Build  Name  SeFer  Plugin  
  68. Next  job  to  trigger  if          

     build  is  stable   Defines  build  number   parameter  provided  to   subsequent  jobs   Parameterized  Trigger  Plugin  
  69. Archive  all  files   Only  archive  if  build    

         was  successful   Clone  Workspace  SCM  Plugin  
  70. Point  to  separated  test  results   Fail  build  of  quality

     gate          criteria  are  not  met   JaCoCo  Plugin   Point  to  JaCoCo  files  as  well        as  source  and  class  files  
  71. Reuse  ini,al  workspace   Reuse  ini,al  build  number   Clone

     Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  
  72. Define  the  target  environment   Downstream  job  that  requires  manual

     execu,on   Build  Pipeline  Plugin  
  73. Build  Pipeline  Plugin   Visualiza,on  of  chained  pipeline  jobs  

  74. None
  75. > gradle qa! :askQuestions ! ! BUILD SUCCESSFUL! ! Total

    time: 300 secs