Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building a Continuous Delivery pipeline with Gradle and Jenkins

Building a Continuous Delivery pipeline with Gradle and Jenkins

Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.

With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.

Benjamin Muschko

June 13, 2013
Tweet

More Decks by Benjamin Muschko

Other Decks in Programming

Transcript

  1. Building  a  Con,nuous  Delivery  pipeline  
     with  Gradle  and  Jenkins  
    Benjamin  Muschko  

    View full-size slide

  2. bmuschko  
    bmuschko  
    [email protected]  
    Gradle  in  Ac,on  
    Java/Groovy  developer  
    Gradle  contributor  
    Open  Source  enthusiast  
    Benjamin  Muschko  
    Presenta,on  will  be  available  on  
    hFps://speakerdeck.com/bmuschko  

    View full-size slide

  3. Releases  don’t  have  to  be  painful  

    View full-size slide

  4. Con,nuous  Delivery  
    Deliver  soKware  fast  and  frequently  

    View full-size slide

  5. #1  Every  commit  can  result  in  a  release  
    #2  Automate  everything!    
       
    #3  Automated  tests  are  essen,al  
    #4  Done  means  released  

    View full-size slide

  6. Build  pipeline  
    Automated  manifesta,on  of  delivery  process  

    View full-size slide

  7. Build  quality  in!  
    Establish  automated  quality  gates  

    View full-size slide

  8. Test  
    Compile/Unit  Tests  
    Integra,on  Tests  
    Code  Analysis  
    Package/Deploy  
    UAT   Prod  
    Acceptance  Tests  

    View full-size slide

  9. !    But    
    how?  

    View full-size slide

  10. The  “revolu,onary”  sample  applica,on  
    Internet
    To Do
    application
    Browser Data
    store
    Writes
    Reads

    View full-size slide

  11. Mul,-­‐project  dependencies  
    Model
    Web Repository
    depend
    depend
    depend

    View full-size slide

  12. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat

    View full-size slide

  13. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat

    View full-size slide

  14. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    Define  project-­‐specific  
                             behavior  

    View full-size slide

  15. Project  hierarchy  
    include 'model'
    include 'repository'
    include 'web'
    settings.gradle
    Defines  which  projects  are  
         taking  part  in  the  build  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat

    View full-size slide

  16. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    Always  use  Wrapper  
    to  execute  the  build!  

    View full-size slide

  17. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
       Externalize  concerns  into  script  plugins  
    and  organize  them  in  dedicated  directory  
    Examples:  
     
    ü  Versioning  strategy  
    ü  Integra,on  and  func,onal  test  setup  
    ü  Deployment  func,onality  
    ü  ...  
     
     
     

    View full-size slide

  18. Project  ar,facts  
    root
    project
    model
    project
    repository
    project
    web
    project
    JAR  
    JAR  
    WAR  
    Deployable  ar,fact  

    View full-size slide

  19. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Asserts  that  system  works  
                 at  a  technical  level  

    View full-size slide

  20. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Asserts  that  system  works  on  a  
    func,onal/non-­‐func,onal  level  

    View full-size slide

  21. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Trigger  manually  
    Trigger  manually  

    View full-size slide

  22. Commit  stage:  Compile/unit  tests  
    Rapid  feedback  (<  5  mins)  
    Run  on  every  VCS  check-­‐in  
    Priority:  fix  broken  build  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution

    View full-size slide

  23. Commit  stage:  Integra,on  tests  
    Long  running  tests  
    Require  environment  setup  
    Hard  to  maintain  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution

    View full-size slide

  24. Separate  tests  in  project  layout  
    todo
    model
    repository
    web
    src
    integTest
    java
    main
    java
    test
    java
    Integra,on  test  Java  sources  
    Produc,on  Java  sources  
    Unit  test  Java  sources  

    View full-size slide

  25. Separate  tests  with  SourceSets  

     
    sourceSets {

    integrationTest {

    java.srcDir file('src/integTest/java')

    resources.srcDir file('src/integTest/resources')

    compileClasspath = sourceSets.main.output + configurations.testRuntime

    runtimeClasspath = output + compileClasspath

    }

    }
    task integrationTest(type: Test) {

    description = 'Runs the integration tests.'

    group = 'verification'

    testClassesDir = sourceSets.integrationTest.output.classesDir

    classpath = sourceSets.integrationTest.runtimeClasspath

    testResultsDir = file("$testResultsDir/integration")

    }

     
           Custom  test    
    results  directory  
    Set  source  and  resources  directory  
     Set  compile  and    
    run,me  classpath  
    gradlew integrationTest

    View full-size slide

  26. Database  integra,on  tests  
    stop
    Database
    test
    start
    Database
    build
    Schema
    integration
    Test
    check build
    ...

    View full-size slide

  27. Database  integra,on  tests  

     
    apply from: "$rootDir/gradle/databaseSetup.gradle"
    integrationTest.dependsOn startAndPrepareDatabase

    stopDatabase.dependsOn integrationTest

    task databaseIntegrationTest(dependsOn: stopDatabase)
    check.dependsOn databaseIntegrationTests  
    stop
    Database
    test
    start
    Database
    build
    Schema
    integration
    Test
    check build
    ...
    Separate  complex  setup  logic  into  script  plugin  
    Integrate  tasks  into  build  lifecycle  

    View full-size slide

  28. Picking  the  “right”  code  coverage  tool  
    Cobertura   Offline  bytecode    
     instrumenta,on  
           Source  code    
     instrumenta,on  
    Offline  bytecode    
     instrumenta,on  
    On-­‐the-­‐fly  bytecode    
           instrumenta,on  

    View full-size slide

  29. On-­‐the-­‐fly  bytecode  instrumenta,on  
    No  modifica,on  to  source  or  bytecode  

    View full-size slide

  30. Code  coverage  with  JaCoCo  

     
    buildscript {

    repositories {

    mavenCentral()

    }


    dependencies {

    classpath 'org.ajoberstar:gradle-jacoco:0.3.0'

    }

    }


    apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin


    jacoco {

    integrationTestTaskName = 'integrationTest'

    }  
    Configures  instrumenta,on    
    for  integra,on  tests  as  well  
         Add  plugin  to  the  
    buildscript’s  classpath  
    Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  

    View full-size slide

  31. Genera,ng  coverage  reports  
    test
    integration
    Test
    ... ...
    .exec  
    jacoco
    TestReport
    jacocoIntegration
    TestReport
    .exec   .html  
    .html  

    View full-size slide

  32. Commit  stage:  Code  analysis  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Perform  code  health  check  
    Fail  build  for  low  quality  
    Record  progress  over  ,me  

    View full-size slide

  33. Sta,c  code  analysis  tools  
    Checkstyle  
    FindBugs  
    apply plugin: 'pmd'


    pmd {

    ignoreFailures = true

    }


    tasks.withType(Pmd) {

    reports {

    xml.enabled = false

    html.enabled = true

    }

    }  
    apply plugin: 'jdepend’


    jdepend {

    toolVersion = '2.9.1'

    ignoreFailures = true

    }  
    gradlew check

    View full-size slide

  34. Measure  quality  over  ,me  with  Sonar  
    Sonar
    database
    Gradle
    Sonar Runner
    root
    model
    repo.
    web
    analyzes
    publishes reads / writes
    Gradle project

    View full-size slide

  35. Applying  the  Sonar  Runner  plugin  
    apply plugin: 'sonar-runner'


    sonarRunner {

    sonarProperties {

    property 'sonar.projectName', 'todo'

    property 'sonar.projectDescription', 'A task management application'

    }

    }


    subprojects {

    sonarRunner {

    sonarProperties {

    property 'sonar.sourceEncoding', 'UTF-8'

    }

    }

    }  
    gradlew sonarRunner

    View full-size slide

  36. Commit  stage:  Assemble  distribu,on  
    Exclude  env.  configura,on  
    Include  build  informa,on  
    Choose  versioning  strategy  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution

    View full-size slide

  37. Versioning  strategy  
    1.0-­‐SNAPSHOT   1.0  
    during  development   when  released  
    …the  Maven  way  
    Change  version  with  Maven  Release  plugin  

    View full-size slide

  38. Versioning  strategy  
    1.0.134   1.0.134  
    during  development   when  released  
    …the  Con,nuous  Delivery  way  
    1.0.134  
    Jenkins  build  number  
    Project  version  number  

    View full-size slide

  39. Versioning  strategy  
    …implemented  with  Gradle  
    allprojects {

    apply from: "$rootDir/gradle/versioning.gradle"

    }  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    gradle
    versioning.gradle Contains  version  implementa,on  

    View full-size slide

  40. Versioning  strategy  
    …implemented  with  Gradle  
    ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss')


    version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)


    class ProjectVersion {

    Integer major

    Integer minor

    String build


    ProjectVersion(Integer major, Integer minor, String build) {

    this.major = major

    this.minor = minor

    this.build = build

    }


    @Override

    String toString() {
    String fullVersion = "$major.$minor"

    if(build) {

    fullVersion += ".$build"
    }

    fullVersion

    }
    }  
    Jenkins  Build  Number  
                 Builds  version    
    String  representa,on  

    View full-size slide

  41. Packaging  the  deployable  ar,fact  
    project(':web') {
    apply plugin: 'war'

    task createBuildInfoFile << {

    def buildInfoFile = new File("$buildDir/build-info.properties")

    Properties props = new Properties()

    props.setProperty('version', project.version.toString())

    props.setProperty('timestamp', project.buildTimestamp)

    props.store(buildInfoFile.newWriter(), null)

    }


    war {

    dependsOn createBuildInfoFile

    baseName = 'todo'


    from(buildDir) {

    include 'build-info.properties'

    into('WEB-INF/classes')

    }

    }
    }  
    Creates  file  containing  
    build  informa,on  
    Include  build  info  file  
    Into  WAR  distribu,on  
    gradlew assemble

    View full-size slide

  42. Commit  stage:  Publish  binaries  
    Version  ar,fact(s)  
    Use  binary  repository  
    Publish  once,  then  reuse  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution

    View full-size slide

  43. Publishing  the  deployable  ar,fact  
    1.0.34  
    1.0.32   1.0.33   1.0.34  

    View full-size slide

  44. Defining  build  configura,on  
    binaryRepository {

    url = 'http://mycompany.bin.repo:8081/artifactory'

    username = 'admin'

    password = 'password'

    name = 'libs-release-local'

    }


    environments {

    test {

    server {

    hostname = 'mycompany.test'

    port = 8099

    context = 'todo'

    username = 'manager'

    password = 'manager'

    }

    }


    uat {

    server {

    hostname = 'mycompany.uat'

    port = 8199

    context = 'todo'

    username = 'manager'

    password = 'manager'

    }

    }

    ...

    }  
    Environment-­‐specific  
         configura,on  
         Common  
     configura,on  
             Read  creden,als  
     from  gradle.proper,es  
             Read  creden,als  
     from  gradle.proper,es  
             Read  creden,als  
     from  gradle.proper,es  

    View full-size slide

  45. Reading  build  configura,on  
    task loadConfiguration {

    def env = project.hasProperty('env') ? project.getProperty('env') : 'test'

    logger.quiet "Loading configuration for environment '$env’."
    def configFile = file("$rootDir/gradle/config/buildConfig.groovy")

    def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())


    allprojects {

    ext.config = parsedConfig

    }

    }  
    Initialization
    phase
    Conguration
    phase
    Execution
    phase
    gradlew –Penv=uat ...
    Assign  configura,on  to  
                   extra  property  

    View full-size slide

  46. Using  the  Maven  Publishing  plugin  
    apply plugin: 'maven-publish'

    ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name”


    publishing {

    publications {

    webApp(MavenPublication) {

    from components.web
    }

    }


    repositories {

    maven {

    url fullRepoUrl


    credentials {

    username = config.binaryRepository.username

    password = config.binaryRepository.password

    }

    }

    }

    }  
    Build  repository  URL  
    from  configura,on  
    gradlew publish
    Assign  publica,on  name  
               and  component  

    View full-size slide

  47. Acceptance  stage:  Retrieve  binaries  
    Request  versioned  ar,fact  
    Store  in  temp.  directory  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries

    View full-size slide

  48. Downloading  the  deployable  ar,fact  
    1.0.34  
    1.0.32   1.0.33   1.0.34  
    1.0.34  
    Test  
    UAT  
    Prod  

    View full-size slide

  49. Task  for  downloading  ar,fact  
    ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 

    task downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {

    ext {

    repoPath = project.group.replaceAll('\\.', '/')

    repoBaseArtifactName = war.baseName

    repoVersion = project.version.toString()

    repoArtifact = war.archiveName

    binaryUrl = "${config.binaryRepository.url}/simple/
    ${config.binaryRepository.name}/${repoPath}/
    ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"

    }


    sourceUrl = binaryUrl

    targetBinary = downloadedArtifact

    }  
    Target  loca,on  for  
    downloaded  ar,fact  
    gradlew downloadBinaryArchive
    Build  exposed    
    binary  URL  from  
    configura,on  

    View full-size slide

  50. Acceptance  stage:  Deploy  binaries  
    Deployment  on  request  
    Make  it  a  reliable  process  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries
    Use  process  for  all  envs.  

    View full-size slide

  51. Deploying  to  mul,ple  environments  
    Test  
    UAT  
    Prod  
    –Penv=prod
    –Penv=uat
    –Penv=test

    View full-size slide

  52. Deployment  with  the  Cargo  plugin  
    cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote

    cargoUndeployRemote {

    onlyIf appContextStatus

    }

    cargo {

    containerId = 'tomcat7x'

    port = config.server.port


    deployable {

    file = downloadedArtifact

    context = config.server.context

    }


    remote {

    hostname = config.server.hostname

    username = config.server.username

    password = config.server.password

    }

    }  
               Download  ar,fact  from  binary  
    repository  and  undeploy  exis,ng  one  
     Only  undeploy  if  
    URL  context  exists  
    Use  environment-­‐
    specific  configura,on  
    gradlew –Penv=uat cargoDeploy

    View full-size slide

  53. Acceptance  stage:  Func,onal  tests  
    Test  all  UI  permuta,ons  
    Test  important  use  cases  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries
    Run  against  different  envs.  

    View full-size slide

  54. In-­‐container  func,onal  tests  
    functional
    JettyStop
    functional
    TestClasses
    functional
    JettyRun
    functional
    Test
    check
    ... ...

     
    task functionalTest(type: Test) {

    ...

    }


    task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {

    httpPort = functionalJettyHttpPort

    stopPort = functionalJettyStopPort

    stopKey = functionalJettyStopKey

    contextPath = functionalJettyContextPath

    daemon = true

    }


    task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {

    stopPort = functionalJettyStopPort

    stopKey = functionalJettyStopKey

    }


    functionalJettyRun.dependsOn functionalTestClasses

    functionalTest.dependsOn functionalJettyRun

    functionalJettyStop.dependsOn functionalTest

    task inContainerFunctionalTest(dependsOn: functionalJettyStop)  
    Func,onal  test  task  
    Custom  JeFy  Run  task  
    Custom  JeFy  Stop  task  

    View full-size slide

  55. Execu,ng  remote  func,onal  tests  
    ext {

    functionalTestReportDir = file("$testReportDir/functional")

    functionalTestResultsDir = file("$testResultsDir/functional")

    functionalCommonSystemProperties =
    ['geb.env': 'firefox',
    'geb.build.reportsDir': reporting.file("$name/geb")]

    }
    task remoteFunctionalTest(type: Test) {

    testClassesDir = sourceSets.functionalTest.output.classesDir

    classpath = sourceSets.functionalTest.runtimeClasspath

    testReportDir = functionalTestReportDir

    testResultsDir = functionalTestResultsDir

    systemProperties functionalCommonSystemProperties

    systemProperty 'geb.build.baseUrl',
    "http://$config.server.hostname:$config.server.port/$config.server.context/"

    }

     
    gradlew –Penv=test remoteFunctionalTest
    Build  URL  from    
    env.  configura,on  
    Reuse  setup  
    proper,es  

    View full-size slide

  56. Going  further:  Capacity  tes,ng  
    buildscript {

    repositories {

    mavenCentral()

    }

    dependencies {

    classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"

    }

    }


    apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin  
    ext.loadTestResources = "$projectDir/src/loadTest/resources"


    jmeterRun.configure {

    jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]

    jmeterPropertyFile = file("$loadTestResources/jmeter.properties")

    jmeterUserProperties = ["hostname=${config.server.hostname},
    "port=${config.server.port}",
    "context=${config.server.context}"]

    logging.captureStandardError LogLevel.INFO
    }  
    gradlew –Penv=uat jmeterRun

    View full-size slide

  57. Let’s  bring  Jenkins  into  play!  
    Test  
    UAT  
    Prod  
         Deployment  
    Acceptance  Tests  
    Publish  
    Download  
    Trigger  Build  
    Pull  Source  Code  

    View full-size slide

  58. Model  pipeline  as  series  of  jobs  
    Triggered  job  when  change  
               to  SCM  is  detected  

    View full-size slide

  59. Ini,al  Jenkins  Build  Job  

    View full-size slide

  60. Build  Name  SeFer  Plugin  

    View full-size slide

  61. JaCoCo  Plugin  

    View full-size slide

  62. Parameterized  Trigger  Plugin  

    View full-size slide

  63. Always  use  the  Wrapper!  
    Gradle  Plugin  
    Run  clean  task  to  remove  
               exis,ng  ar,facts  

    View full-size slide

  64. Use  Jenkins  environment  variable  
    Clearly  iden,fy  a  build  
    through  an  expressive  
                     build  name  
    Build  Name  SeFer  Plugin  

    View full-size slide

  65. Next  job  to  trigger  if  
             build  is  stable  
    Defines  build  number  
    parameter  provided  to  
    subsequent  jobs  
    Parameterized  Trigger  Plugin  

    View full-size slide

  66. Archive  all  files  
    Only  archive  if  build  
           was  successful  
    Clone  Workspace  SCM  Plugin  

    View full-size slide

  67. Point  to  separated  test  results  
    Fail  build  of  quality  gate    
         criteria  are  not  met  
    JaCoCo  Plugin  
    Point  to  JaCoCo  files  as  well  
         as  source  and  class  files  

    View full-size slide

  68. Reuse  ini,al  workspace  
    Reuse  ini,al  build  number  
    Clone  Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  

    View full-size slide

  69. Define  the  target  environment  
    Downstream  job  that  requires  manual  execu,on  
    Build  Pipeline  Plugin  

    View full-size slide

  70. Build  Pipeline  Plugin  
    Visualiza,on  of  chained  pipeline  jobs  

    View full-size slide

  71. > gradle qa!
    :askQuestions !
    !
    BUILD SUCCESSFUL!
    !
    Total time: 300 secs

    View full-size slide