Building a Continuous Delivery pipeline with Gradle and Jenkins

Building a Continuous Delivery pipeline with Gradle and Jenkins

Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.

With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.

8f2248c6bfcc6df39a2cd8edf4267cb5?s=128

Benjamin Muschko

June 13, 2013
Tweet

Transcript

  1. 2.

    bmuschko   bmuschko   benjamin.muschko@gmail.com   Gradle  in  Ac,on  

    Java/Groovy  developer   Gradle  contributor   Open  Source  enthusiast   Benjamin  Muschko   Presenta,on  will  be  available  on   hFps://speakerdeck.com/bmuschko  
  2. 5.

    #1  Every  commit  can  result  in  a  release   #2

     Automate  everything!         #3  Automated  tests  are  essen,al   #4  Done  means  released  
  3. 8.

    Test   Compile/Unit  Tests   Integra,on  Tests   Code  Analysis

      Package/Deploy   UAT   Prod   Acceptance  Tests  
  4. 12.

    Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  5. 13.

    Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  6. 14.

    Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Define  project-­‐specific                            behavior  
  7. 15.

    Project  hierarchy   include 'model' include 'repository' include 'web' settings.gradle

    Defines  which  projects  are        taking  part  in  the  build   todo model repository build.gradle settings.gradle web build.gradle build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat
  8. 16.

    Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat Always  use  Wrapper   to  execute  the  build!  
  9. 17.

    Project  hierarchy   todo model repository build.gradle settings.gradle web build.gradle

    build.gradle build.gradle gradle wrapper xyz.gradle gradlew gradlew.bat    Externalize  concerns  into  script  plugins   and  organize  them  in  dedicated  directory   Examples:     ü  Versioning  strategy   ü  Integra,on  and  func,onal  test  setup   ü  Deployment  func,onality   ü  ...        
  10. 18.

    Project  ar,facts   root project model project repository project web

    project JAR   JAR   WAR   Deployable  ar,fact  
  11. 19.

    Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works                at  a  technical  level  
  12. 20.

    Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Asserts  that  system  works  on  a   func,onal/non-­‐func,onal  level  
  13. 21.

    Stages  in  build  pipeline   Acceptance stage Functional tests Publish

    Binaries Commit stage Deploy Binaries UAT Deploy Binaries Production Integration Tests Code Analysis Assemble Distribution Compile Unit Tests Retrieve Binaries Deploy Binaries Trigger  manually   Trigger  manually  
  14. 22.

    Commit  stage:  Compile/unit  tests   Rapid  feedback  (<  5  mins)

      Run  on  every  VCS  check-­‐in   Priority:  fix  broken  build   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  15. 23.

    Commit  stage:  Integra,on  tests   Long  running  tests   Require

     environment  setup   Hard  to  maintain   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  16. 24.

    Separate  tests  in  project  layout   todo model repository web

    src integTest java main java test java Integra,on  test  Java  sources   Produc,on  Java  sources   Unit  test  Java  sources  
  17. 25.

    Separate  tests  with  SourceSets   
   sourceSets {
 integrationTest

    {
 java.srcDir file('src/integTest/java')
 resources.srcDir file('src/integTest/resources')
 compileClasspath = sourceSets.main.output + configurations.testRuntime
 runtimeClasspath = output + compileClasspath
 }
 } task integrationTest(type: Test) {
 description = 'Runs the integration tests.'
 group = 'verification'
 testClassesDir = sourceSets.integrationTest.output.classesDir
 classpath = sourceSets.integrationTest.runtimeClasspath
 testResultsDir = file("$testResultsDir/integration")
 }
          Custom  test     results  directory   Set  source  and  resources  directory    Set  compile  and     run,me  classpath   gradlew integrationTest
  18. 27.

    Database  integra,on  tests   
   apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn

    startAndPrepareDatabase
 stopDatabase.dependsOn integrationTest
 task databaseIntegrationTest(dependsOn: stopDatabase) check.dependsOn databaseIntegrationTests   stop Database test start Database build Schema integration Test check build ... Separate  complex  setup  logic  into  script  plugin   Integrate  tasks  into  build  lifecycle  
  19. 28.

    Picking  the  “right”  code  coverage  tool   Cobertura   Offline

     bytecode      instrumenta,on          Source  code      instrumenta,on   Offline  bytecode      instrumenta,on   On-­‐the-­‐fly  bytecode            instrumenta,on  
  20. 30.

    Code  coverage  with  JaCoCo   
   buildscript {
 repositories

    {
 mavenCentral()
 }
 
 dependencies {
 classpath 'org.ajoberstar:gradle-jacoco:0.3.0'
 }
 }
 
 apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin
 
 jacoco {
 integrationTestTaskName = 'integrationTest'
 }   Configures  instrumenta,on     for  integra,on  tests  as  well        Add  plugin  to  the   buildscript’s  classpath   Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  
  21. 31.

    Genera,ng  coverage  reports   test integration Test ... ... .exec

      jacoco TestReport jacocoIntegration TestReport .exec   .html   .html  
  22. 32.

    Commit  stage:  Code  analysis   Publish Binaries Commit stage Compile

    Unit Tests Integration Tests Code Analysis Assemble Distribution Perform  code  health  check   Fail  build  for  low  quality   Record  progress  over  ,me  
  23. 33.

    Sta,c  code  analysis  tools   Checkstyle   FindBugs   apply

    plugin: 'pmd'
 
 pmd {
 ignoreFailures = true
 }
 
 tasks.withType(Pmd) {
 reports {
 xml.enabled = false
 html.enabled = true
 }
 }   apply plugin: 'jdepend’
 
 jdepend {
 toolVersion = '2.9.1'
 ignoreFailures = true
 }   gradlew check
  24. 34.

    Measure  quality  over  ,me  with  Sonar   Sonar database Gradle

    Sonar Runner root model repo. web analyzes publishes reads / writes Gradle project
  25. 35.

    Applying  the  Sonar  Runner  plugin   apply plugin: 'sonar-runner'
 


    sonarRunner {
 sonarProperties {
 property 'sonar.projectName', 'todo'
 property 'sonar.projectDescription', 'A task management application'
 }
 }
 
 subprojects {
 sonarRunner {
 sonarProperties {
 property 'sonar.sourceEncoding', 'UTF-8'
 }
 }
 }   gradlew sonarRunner
  26. 36.
  27. 37.

    Commit  stage:  Assemble  distribu,on   Exclude  env.  configura,on   Include

     build  informa,on   Choose  versioning  strategy   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  28. 38.

    Versioning  strategy   1.0-­‐SNAPSHOT   1.0   during  development  

    when  released   …the  Maven  way   Change  version  with  Maven  Release  plugin  
  29. 39.

    Versioning  strategy   1.0.134   1.0.134   during  development  

    when  released   …the  Con,nuous  Delivery  way   1.0.134   Jenkins  build  number   Project  version  number  
  30. 40.

    Versioning  strategy   …implemented  with  Gradle   allprojects {
 apply

    from: "$rootDir/gradle/versioning.gradle"
 }   todo model repository build.gradle settings.gradle web gradle versioning.gradle Contains  version  implementa,on  
  31. 41.

    Versioning  strategy   …implemented  with  Gradle   ext.buildTimestamp = new

    Date().format('yyyy-MM-dd HH:mm:ss')
 
 version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)
 
 class ProjectVersion {
 Integer major
 Integer minor
 String build
 
 ProjectVersion(Integer major, Integer minor, String build) {
 this.major = major
 this.minor = minor
 this.build = build
 }
 
 @Override
 String toString() { String fullVersion = "$major.$minor"
 if(build) {
 fullVersion += ".$build" }
 fullVersion
 } }   Jenkins  Build  Number                Builds  version     String  representa,on  
  32. 42.

    Packaging  the  deployable  ar,fact   project(':web') { apply plugin: 'war'

    
 task createBuildInfoFile << {
 def buildInfoFile = new File("$buildDir/build-info.properties")
 Properties props = new Properties()
 props.setProperty('version', project.version.toString())
 props.setProperty('timestamp', project.buildTimestamp)
 props.store(buildInfoFile.newWriter(), null)
 }
 
 war {
 dependsOn createBuildInfoFile
 baseName = 'todo'
 
 from(buildDir) {
 include 'build-info.properties'
 into('WEB-INF/classes')
 }
 } }   Creates  file  containing   build  informa,on   Include  build  info  file   Into  WAR  distribu,on   gradlew assemble
  33. 43.

    Commit  stage:  Publish  binaries   Version  ar,fact(s)   Use  binary

     repository   Publish  once,  then  reuse   Publish Binaries Commit stage Compile Unit Tests Integration Tests Code Analysis Assemble Distribution
  34. 45.

    Defining  build  configura,on   binaryRepository {
 url = 'http://mycompany.bin.repo:8081/artifactory'
 username

    = 'admin'
 password = 'password'
 name = 'libs-release-local'
 }
 
 environments {
 test {
 server {
 hostname = 'mycompany.test'
 port = 8099
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 }
 
 uat {
 server {
 hostname = 'mycompany.uat'
 port = 8199
 context = 'todo'
 username = 'manager'
 password = 'manager'
 }
 } 
 ...
 }   Environment-­‐specific        configura,on        Common    configura,on            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es            Read  creden,als    from  gradle.proper,es  
  35. 46.

    Reading  build  configura,on   task loadConfiguration {
 def env =

    project.hasProperty('env') ? project.getProperty('env') : 'test'
 logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy")
 def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())
 
 allprojects {
 ext.config = parsedConfig
 }
 }   Initialization phase Conguration phase Execution phase gradlew –Penv=uat ... Assign  configura,on  to                  extra  property  
  36. 47.

    Using  the  Maven  Publishing  plugin   apply plugin: 'maven-publish'
 ext.fullRepoUrl

    = "$config.binaryRepository.url/$config.binaryRepository.name”
 
 publishing {
 publications {
 webApp(MavenPublication) {
 from components.web }
 }
 
 repositories {
 maven {
 url fullRepoUrl
 
 credentials {
 username = config.binaryRepository.username
 password = config.binaryRepository.password
 }
 }
 }
 }   Build  repository  URL   from  configura,on   gradlew publish Assign  publica,on  name              and  component  
  37. 48.
  38. 49.

    Acceptance  stage:  Retrieve  binaries   Request  versioned  ar,fact   Store

     in  temp.  directory   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries
  39. 50.

    Downloading  the  deployable  ar,fact   1.0.34   1.0.32   1.0.33

      1.0.34   1.0.34   Test   UAT   Prod  
  40. 51.

    Task  for  downloading  ar,fact   ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 
 task

    downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {
 ext {
 repoPath = project.group.replaceAll('\\.', '/')
 repoBaseArtifactName = war.baseName
 repoVersion = project.version.toString()
 repoArtifact = war.archiveName
 binaryUrl = "${config.binaryRepository.url}/simple/ ${config.binaryRepository.name}/${repoPath}/ ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"
 }
 
 sourceUrl = binaryUrl
 targetBinary = downloadedArtifact
 }   Target  loca,on  for   downloaded  ar,fact   gradlew downloadBinaryArchive Build  exposed     binary  URL  from   configura,on  
  41. 52.

    Acceptance  stage:  Deploy  binaries   Deployment  on  request   Make

     it  a  reliable  process   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Use  process  for  all  envs.  
  42. 53.

    Deploying  to  mul,ple  environments   Test   UAT   Prod

      –Penv=prod –Penv=uat –Penv=test
  43. 54.

    Deployment  with  the  Cargo  plugin   cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote
 cargoUndeployRemote

    {
 onlyIf appContextStatus
 } 
 cargo {
 containerId = 'tomcat7x'
 port = config.server.port
 
 deployable {
 file = downloadedArtifact
 context = config.server.context
 }
 
 remote {
 hostname = config.server.hostname
 username = config.server.username
 password = config.server.password
 }
 }              Download  ar,fact  from  binary   repository  and  undeploy  exis,ng  one    Only  undeploy  if   URL  context  exists   Use  environment-­‐ specific  configura,on   gradlew –Penv=uat cargoDeploy
  44. 55.
  45. 56.

    Acceptance  stage:  Func,onal  tests   Test  all  UI  permuta,ons  

    Test  important  use  cases   Acceptance stage Deploy Binaries Functional Tests Retrieve Binaries Run  against  different  envs.  
  46. 57.

    In-­‐container  func,onal  tests   functional JettyStop functional TestClasses functional JettyRun

    functional Test check ... ... 
   task functionalTest(type: Test) {
 ...
 }
 
 task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {
 httpPort = functionalJettyHttpPort
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 contextPath = functionalJettyContextPath
 daemon = true
 }
 
 task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {
 stopPort = functionalJettyStopPort
 stopKey = functionalJettyStopKey
 }
 
 functionalJettyRun.dependsOn functionalTestClasses
 functionalTest.dependsOn functionalJettyRun
 functionalJettyStop.dependsOn functionalTest
 task inContainerFunctionalTest(dependsOn: functionalJettyStop)   Func,onal  test  task   Custom  JeFy  Run  task   Custom  JeFy  Stop  task  
  47. 58.

    Execu,ng  remote  func,onal  tests   ext {
 functionalTestReportDir = file("$testReportDir/functional")


    functionalTestResultsDir = file("$testResultsDir/functional")
 functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")]
 } task remoteFunctionalTest(type: Test) {
 testClassesDir = sourceSets.functionalTest.output.classesDir
 classpath = sourceSets.functionalTest.runtimeClasspath
 testReportDir = functionalTestReportDir
 testResultsDir = functionalTestResultsDir
 systemProperties functionalCommonSystemProperties
 systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/"
 }
   gradlew –Penv=test remoteFunctionalTest Build  URL  from     env.  configura,on   Reuse  setup   proper,es  
  48. 59.

    Going  further:  Capacity  tes,ng   buildscript {
 repositories {
 mavenCentral()


    } 
 dependencies {
 classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"
 }
 }
 
 apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin   ext.loadTestResources = "$projectDir/src/loadTest/resources"
 
 jmeterRun.configure {
 jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]
 jmeterPropertyFile = file("$loadTestResources/jmeter.properties")
 jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"]
 logging.captureStandardError LogLevel.INFO }   gradlew –Penv=uat jmeterRun
  49. 60.

    Let’s  bring  Jenkins  into  play!   Test   UAT  

    Prod        Deployment   Acceptance  Tests   Publish   Download   Trigger  Build   Pull  Source  Code  
  50. 61.

    Model  pipeline  as  series  of  jobs   Triggered  job  when

     change              to  SCM  is  detected  
  51. 66.

    Always  use  the  Wrapper!   Gradle  Plugin   Run  clean

     task  to  remove              exis,ng  ar,facts  
  52. 67.

    Use  Jenkins  environment  variable   Clearly  iden,fy  a  build  

    through  an  expressive                    build  name   Build  Name  SeFer  Plugin  
  53. 68.

    Next  job  to  trigger  if          

     build  is  stable   Defines  build  number   parameter  provided  to   subsequent  jobs   Parameterized  Trigger  Plugin  
  54. 69.

    Archive  all  files   Only  archive  if  build    

         was  successful   Clone  Workspace  SCM  Plugin  
  55. 70.

    Point  to  separated  test  results   Fail  build  of  quality

     gate          criteria  are  not  met   JaCoCo  Plugin   Point  to  JaCoCo  files  as  well        as  source  and  class  files  
  56. 71.

    Reuse  ini,al  workspace   Reuse  ini,al  build  number   Clone

     Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  
  57. 74.