Building a Continuous Delivery pipeline with Gradle and Jenkins

Getting software from a developer's machine to a production environment without a fully automated process is time-consuming and error-prone. Continuous Delivery enables building, testing and deploying of software through build pipelines with well-defined quality gates. In this session, we will discuss how to build an exemplary pipeline with the help of Gradle and Jenkins.

With Jenkins as the centerpiece of our build pipeline, we'll model our way from build to deployment. We'll start by introducing a case study application and learn how to build it with Gradle. Step by step, we will touch on topics like developing unit, integration and functional tests, incorporating popular OSS code quality tools as well as packaging, publishing and deploying the deliverable.

Transcript

1. Building  a  Con,nuous  Delivery  pipeline  
    with  Gradle  and  Jenkins  
   Benjamin  Muschko  
   

   View Slide

2. bmuschko  
   bmuschko  
   benjamin.muschko@gmail.com  
   Gradle  in  Ac,on  
   Java/Groovy  developer  
   Gradle  contributor  
   Open  Source  enthusiast  
   Benjamin  Muschko  
   Presenta,on  will  be  available  on  
   hFps://speakerdeck.com/bmuschko  
   

   View Slide

3. Releases  don’t  have  to  be  painful  
   

   View Slide

4. Con,nuous  Delivery  
   Deliver  soKware  fast  and  frequently  
   

   View Slide

5. #1  Every  commit  can  result  in  a  release  
   #2  Automate  everything!    
      
   #3  Automated  tests  are  essen,al  
   #4  Done  means  released  
   

   View Slide

6. Build  pipeline  
   Automated  manifesta,on  of  delivery  process  
   

   View Slide

7. Build  quality  in!  
   Establish  automated  quality  gates  
   

   View Slide

8. Test  
   Compile/Unit  Tests  
   Integra,on  Tests  
   Code  Analysis  
   Package/Deploy  
   UAT   Prod  
   Acceptance  Tests  
   

   View Slide

9. !    But    
   how?  
   

   View Slide

10. The  “revolu,onary”  sample  applica,on  
    Internet
    To Do
    application
    Browser Data
    store
    Writes
    Reads
    

    View Slide

11. Mul,-­‐project  dependencies  
    Model
    Web Repository
    depend
    depend
    depend
    

    View Slide

12. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    

    View Slide

13. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    

    View Slide

14. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    Define  project-­‐specific  
                             behavior  
    

    View Slide

15. Project  hierarchy  
    include 'model'
    include 'repository'
    include 'web'
    settings.gradle
    Defines  which  projects  are  
         taking  part  in  the  build  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    

    View Slide

16. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
    Always  use  Wrapper  
    to  execute  the  build!  
    

    View Slide

17. Project  hierarchy  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    build.gradle
    build.gradle
    build.gradle
    gradle
    wrapper
    xyz.gradle
    gradlew
    gradlew.bat
       Externalize  concerns  into  script  plugins  
    and  organize  them  in  dedicated  directory  
    Examples:  
     
    ü  Versioning  strategy  
    ü  Integra,on  and  func,onal  test  setup  
    ü  Deployment  func,onality  
    ü  ...  
     
     
     
    

    View Slide

18. Project  ar,facts  
    root
    project
    model
    project
    repository
    project
    web
    project
    JAR  
    JAR  
    WAR  
    Deployable  ar,fact  
    

    View Slide

19. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Asserts  that  system  works  
                 at  a  technical  level  
    

    View Slide

20. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Asserts  that  system  works  on  a  
    func,onal/non-­‐func,onal  level  
    

    View Slide

21. Stages  in  build  pipeline  
    Acceptance stage
    Functional
    tests
    Publish
    Binaries
    Commit stage
    Deploy
    Binaries
    UAT
    Deploy
    Binaries
    Production
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Compile
    Unit Tests
    Retrieve
    Binaries
    Deploy
    Binaries
    Trigger  manually  
    Trigger  manually  
    

    View Slide

22. Commit  stage:  Compile/unit  tests  
    Rapid  feedback  (<  5  mins)  
    Run  on  every  VCS  check-­‐in  
    Priority:  fix  broken  build  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    

    View Slide

23. Commit  stage:  Integra,on  tests  
    Long  running  tests  
    Require  environment  setup  
    Hard  to  maintain  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    

    View Slide

24. Separate  tests  in  project  layout  
    todo
    model
    repository
    web
    src
    integTest
    java
    main
    java
    test
    java
    Integra,on  test  Java  sources  
    Produc,on  Java  sources  
    Unit  test  Java  sources  
    

    View Slide

25. Separate  tests  with  SourceSets  
    

     
    sourceSets {

    integrationTest {

    java.srcDir file('src/integTest/java')

    resources.srcDir file('src/integTest/resources')

    compileClasspath = sourceSets.main.output + configurations.testRuntime

    runtimeClasspath = output + compileClasspath

    }

    }
    task integrationTest(type: Test) {

    description = 'Runs the integration tests.'

    group = 'verification'

    testClassesDir = sourceSets.integrationTest.output.classesDir

    classpath = sourceSets.integrationTest.runtimeClasspath

    testResultsDir = file("$testResultsDir/integration")

    }

     
           Custom  test    
    results  directory  
    Set  source  and  resources  directory  
     Set  compile  and    
    run,me  classpath  
    gradlew integrationTest
    

    View Slide

26. Database  integra,on  tests  
    stop
    Database
    test
    start
    Database
    build
    Schema
    integration
    Test
    check build
    ...
    

    View Slide

27. Database  integra,on  tests  
    

     
    apply from: "$rootDir/gradle/databaseSetup.gradle"
    integrationTest.dependsOn startAndPrepareDatabase

    stopDatabase.dependsOn integrationTest

    task databaseIntegrationTest(dependsOn: stopDatabase)
    check.dependsOn databaseIntegrationTests  
    stop
    Database
    test
    start
    Database
    build
    Schema
    integration
    Test
    check build
    ...
    Separate  complex  setup  logic  into  script  plugin  
    Integrate  tasks  into  build  lifecycle  
    

    View Slide

28. Picking  the  “right”  code  coverage  tool  
    Cobertura   Offline  bytecode    
     instrumenta,on  
           Source  code    
     instrumenta,on  
    Offline  bytecode    
     instrumenta,on  
    On-­‐the-­‐fly  bytecode    
           instrumenta,on  
    

    View Slide

29. On-­‐the-­‐fly  bytecode  instrumenta,on  
    No  modifica,on  to  source  or  bytecode  
    

    View Slide

30. Code  coverage  with  JaCoCo  
    

     
    buildscript {

    repositories {

    mavenCentral()

    }

    

    dependencies {

    classpath 'org.ajoberstar:gradle-jacoco:0.3.0'

    }

    }

    

    apply plugin: org.ajoberstar.gradle.jacoco.plugins.JacocoPlugin

    

    jacoco {

    integrationTestTaskName = 'integrationTest'

    }  
    Configures  instrumenta,on    
    for  integra,on  tests  as  well  
         Add  plugin  to  the  
    buildscript’s  classpath  
    Gradle  1.6  includes  this  plugin  as  incuba,ng  feature  
    

    View Slide

31. Genera,ng  coverage  reports  
    test
    integration
    Test
    ... ...
    .exec  
    jacoco
    TestReport
    jacocoIntegration
    TestReport
    .exec   .html  
    .html  
    

    View Slide

32. Commit  stage:  Code  analysis  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    Perform  code  health  check  
    Fail  build  for  low  quality  
    Record  progress  over  ,me  
    

    View Slide

33. Sta,c  code  analysis  tools  
    Checkstyle  
    FindBugs  
    apply plugin: 'pmd'

    

    pmd {

    ignoreFailures = true

    }

    

    tasks.withType(Pmd) {

    reports {

    xml.enabled = false

    html.enabled = true

    }

    }  
    apply plugin: 'jdepend’

    

    jdepend {

    toolVersion = '2.9.1'

    ignoreFailures = true

    }  
    gradlew check
    

    View Slide

34. Measure  quality  over  ,me  with  Sonar  
    Sonar
    database
    Gradle
    Sonar Runner
    root
    model
    repo.
    web
    analyzes
    publishes reads / writes
    Gradle project
    

    View Slide

35. Applying  the  Sonar  Runner  plugin  
    apply plugin: 'sonar-runner'

    

    sonarRunner {

    sonarProperties {

    property 'sonar.projectName', 'todo'

    property 'sonar.projectDescription', 'A task management application'

    }

    }

    

    subprojects {

    sonarRunner {

    sonarProperties {

    property 'sonar.sourceEncoding', 'UTF-8'

    }

    }

    }  
    gradlew sonarRunner
    

    View Slide

36. View Slide

37. Commit  stage:  Assemble  distribu,on  
    Exclude  env.  configura,on  
    Include  build  informa,on  
    Choose  versioning  strategy  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    

    View Slide

38. Versioning  strategy  
    1.0-­‐SNAPSHOT   1.0  
    during  development   when  released  
    …the  Maven  way  
    Change  version  with  Maven  Release  plugin  
    

    View Slide

39. Versioning  strategy  
    1.0.134   1.0.134  
    during  development   when  released  
    …the  Con,nuous  Delivery  way  
    1.0.134  
    Jenkins  build  number  
    Project  version  number  
    

    View Slide

40. Versioning  strategy  
    …implemented  with  Gradle  
    allprojects {

    apply from: "$rootDir/gradle/versioning.gradle"

    }  
    todo
    model
    repository
    build.gradle
    settings.gradle
    web
    gradle
    versioning.gradle Contains  version  implementa,on  
    

    View Slide

41. Versioning  strategy  
    …implemented  with  Gradle  
    ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss')

    

    version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER)

    

    class ProjectVersion {

    Integer major

    Integer minor

    String build

    

    ProjectVersion(Integer major, Integer minor, String build) {

    this.major = major

    this.minor = minor

    this.build = build

    }

    

    @Override

    String toString() {
    String fullVersion = "$major.$minor"

    if(build) {

    fullVersion += ".$build"
    }

    fullVersion

    }
    }  
    Jenkins  Build  Number  
                 Builds  version    
    String  representa,on  
    

    View Slide

42. Packaging  the  deployable  ar,fact  
    project(':web') {
    apply plugin: 'war'
    

    task createBuildInfoFile << {

    def buildInfoFile = new File("$buildDir/build-info.properties")

    Properties props = new Properties()

    props.setProperty('version', project.version.toString())

    props.setProperty('timestamp', project.buildTimestamp)

    props.store(buildInfoFile.newWriter(), null)

    }

    

    war {

    dependsOn createBuildInfoFile

    baseName = 'todo'

    

    from(buildDir) {

    include 'build-info.properties'

    into('WEB-INF/classes')

    }

    }
    }  
    Creates  file  containing  
    build  informa,on  
    Include  build  info  file  
    Into  WAR  distribu,on  
    gradlew assemble
    

    View Slide

43. Commit  stage:  Publish  binaries  
    Version  ar,fact(s)  
    Use  binary  repository  
    Publish  once,  then  reuse  
    Publish
    Binaries
    Commit stage
    Compile
    Unit Tests
    Integration
    Tests
    Code
    Analysis
    Assemble
    Distribution
    

    View Slide

44. Publishing  the  deployable  ar,fact  
    1.0.34  
    1.0.32   1.0.33   1.0.34  
    

    View Slide

45. Defining  build  configura,on  
    binaryRepository {

    url = 'http://mycompany.bin.repo:8081/artifactory'

    username = 'admin'

    password = 'password'

    name = 'libs-release-local'

    }

    

    environments {

    test {

    server {

    hostname = 'mycompany.test'

    port = 8099

    context = 'todo'

    username = 'manager'

    password = 'manager'

    }

    }

    

    uat {

    server {

    hostname = 'mycompany.uat'

    port = 8199

    context = 'todo'

    username = 'manager'

    password = 'manager'

    }

    }
    

    ...

    }  
    Environment-­‐specific  
         configura,on  
         Common  
     configura,on  
             Read  creden,als  
     from  gradle.proper,es  
             Read  creden,als  
     from  gradle.proper,es  
             Read  creden,als  
     from  gradle.proper,es  
    

    View Slide

46. Reading  build  configura,on  
    task loadConfiguration {

    def env = project.hasProperty('env') ? project.getProperty('env') : 'test'

    logger.quiet "Loading configuration for environment '$env’."
    def configFile = file("$rootDir/gradle/config/buildConfig.groovy")

    def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL())

    

    allprojects {

    ext.config = parsedConfig

    }

    }  
    Initialization
    phase
    Conguration
    phase
    Execution
    phase
    gradlew –Penv=uat ...
    Assign  configura,on  to  
                   extra  property  
    

    View Slide

47. Using  the  Maven  Publishing  plugin  
    apply plugin: 'maven-publish'

    ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name”

    

    publishing {

    publications {

    webApp(MavenPublication) {

    from components.web
    }

    }

    

    repositories {

    maven {

    url fullRepoUrl

    

    credentials {

    username = config.binaryRepository.username

    password = config.binaryRepository.password

    }

    }

    }

    }  
    Build  repository  URL  
    from  configura,on  
    gradlew publish
    Assign  publica,on  name  
               and  component  
    

    View Slide

48. View Slide

49. Acceptance  stage:  Retrieve  binaries  
    Request  versioned  ar,fact  
    Store  in  temp.  directory  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries
    

    View Slide

50. Downloading  the  deployable  ar,fact  
    1.0.34  
    1.0.32   1.0.33   1.0.34  
    1.0.34  
    Test  
    UAT  
    Prod  
    

    View Slide

51. Task  for  downloading  ar,fact  
    ext.downloadedArtifact = file("$buildDir/download/$war.archiveName") 

    task downloadBinaryArchive(type: com.manning.gia.BinaryDownload) {

    ext {

    repoPath = project.group.replaceAll('\\.', '/')

    repoBaseArtifactName = war.baseName

    repoVersion = project.version.toString()

    repoArtifact = war.archiveName

    binaryUrl = "${config.binaryRepository.url}/simple/
    ${config.binaryRepository.name}/${repoPath}/
    ${repoBaseArtifactName}/${repoVersion}/${repoArtifact}"

    }

    

    sourceUrl = binaryUrl

    targetBinary = downloadedArtifact

    }  
    Target  loca,on  for  
    downloaded  ar,fact  
    gradlew downloadBinaryArchive
    Build  exposed    
    binary  URL  from  
    configura,on  
    

    View Slide

52. Acceptance  stage:  Deploy  binaries  
    Deployment  on  request  
    Make  it  a  reliable  process  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries
    Use  process  for  all  envs.  
    

    View Slide

53. Deploying  to  mul,ple  environments  
    Test  
    UAT  
    Prod  
    –Penv=prod
    –Penv=uat
    –Penv=test
    

    View Slide

54. Deployment  with  the  Cargo  plugin  
    cargoDeployRemote.dependsOn downloadBinaryArchive, cargoUndeployRemote

    cargoUndeployRemote {

    onlyIf appContextStatus

    }
    

    cargo {

    containerId = 'tomcat7x'

    port = config.server.port

    

    deployable {

    file = downloadedArtifact

    context = config.server.context

    }

    

    remote {

    hostname = config.server.hostname

    username = config.server.username

    password = config.server.password

    }

    }  
               Download  ar,fact  from  binary  
    repository  and  undeploy  exis,ng  one  
     Only  undeploy  if  
    URL  context  exists  
    Use  environment-­‐
    specific  configura,on  
    gradlew –Penv=uat cargoDeploy
    

    View Slide

55. View Slide

56. Acceptance  stage:  Func,onal  tests  
    Test  all  UI  permuta,ons  
    Test  important  use  cases  
    Acceptance stage
    Deploy
    Binaries
    Functional
    Tests
    Retrieve
    Binaries
    Run  against  different  envs.  
    

    View Slide

57. In-­‐container  func,onal  tests  
    functional
    JettyStop
    functional
    TestClasses
    functional
    JettyRun
    functional
    Test
    check
    ... ...
    

     
    task functionalTest(type: Test) {

    ...

    }

    

    task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) {

    httpPort = functionalJettyHttpPort

    stopPort = functionalJettyStopPort

    stopKey = functionalJettyStopKey

    contextPath = functionalJettyContextPath

    daemon = true

    }

    

    task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) {

    stopPort = functionalJettyStopPort

    stopKey = functionalJettyStopKey

    }

    

    functionalJettyRun.dependsOn functionalTestClasses

    functionalTest.dependsOn functionalJettyRun

    functionalJettyStop.dependsOn functionalTest

    task inContainerFunctionalTest(dependsOn: functionalJettyStop)  
    Func,onal  test  task  
    Custom  JeFy  Run  task  
    Custom  JeFy  Stop  task  
    

    View Slide

58. Execu,ng  remote  func,onal  tests  
    ext {

    functionalTestReportDir = file("$testReportDir/functional")

    functionalTestResultsDir = file("$testResultsDir/functional")

    functionalCommonSystemProperties =
    ['geb.env': 'firefox',
    'geb.build.reportsDir': reporting.file("$name/geb")]

    }
    task remoteFunctionalTest(type: Test) {

    testClassesDir = sourceSets.functionalTest.output.classesDir

    classpath = sourceSets.functionalTest.runtimeClasspath

    testReportDir = functionalTestReportDir

    testResultsDir = functionalTestResultsDir

    systemProperties functionalCommonSystemProperties

    systemProperty 'geb.build.baseUrl',
    "http://$config.server.hostname:$config.server.port/$config.server.context/"

    }

     
    gradlew –Penv=test remoteFunctionalTest
    Build  URL  from    
    env.  configura,on  
    Reuse  setup  
    proper,es  
    

    View Slide

59. Going  further:  Capacity  tes,ng  
    buildscript {

    repositories {

    mavenCentral()

    }
    

    dependencies {

    classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9"

    }

    }

    

    apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin  
    ext.loadTestResources = "$projectDir/src/loadTest/resources"

    

    jmeterRun.configure {

    jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")]

    jmeterPropertyFile = file("$loadTestResources/jmeter.properties")

    jmeterUserProperties = ["hostname=${config.server.hostname},
    "port=${config.server.port}",
    "context=${config.server.context}"]

    logging.captureStandardError LogLevel.INFO
    }  
    gradlew –Penv=uat jmeterRun
    

    View Slide

60. Let’s  bring  Jenkins  into  play!  
    Test  
    UAT  
    Prod  
         Deployment  
    Acceptance  Tests  
    Publish  
    Download  
    Trigger  Build  
    Pull  Source  Code  
    

    View Slide

61. Model  pipeline  as  series  of  jobs  
    Triggered  job  when  change  
               to  SCM  is  detected  
    

    View Slide

62. Ini,al  Jenkins  Build  Job  
    

    View Slide

63. Build  Name  SeFer  Plugin  
    

    View Slide

64. JaCoCo  Plugin  
    

    View Slide

65. Parameterized  Trigger  Plugin  
    

    View Slide

66. Always  use  the  Wrapper!  
    Gradle  Plugin  
    Run  clean  task  to  remove  
               exis,ng  ar,facts  
    

    View Slide

67. Use  Jenkins  environment  variable  
    Clearly  iden,fy  a  build  
    through  an  expressive  
                     build  name  
    Build  Name  SeFer  Plugin  
    

    View Slide

68. Next  job  to  trigger  if  
             build  is  stable  
    Defines  build  number  
    parameter  provided  to  
    subsequent  jobs  
    Parameterized  Trigger  Plugin  
    

    View Slide

69. Archive  all  files  
    Only  archive  if  build  
           was  successful  
    Clone  Workspace  SCM  Plugin  
    

    View Slide

70. Point  to  separated  test  results  
    Fail  build  of  quality  gate    
         criteria  are  not  met  
    JaCoCo  Plugin  
    Point  to  JaCoCo  files  as  well  
         as  source  and  class  files  
    

    View Slide

71. Reuse  ini,al  workspace  
    Reuse  ini,al  build  number  
    Clone  Workspace  SCM  Plugin   Build  Name  SeFer  Plugin  
    

    View Slide

72. Define  the  target  environment  
    Downstream  job  that  requires  manual  execu,on  
    Build  Pipeline  Plugin  
    

    View Slide

73. Build  Pipeline  Plugin  
    Visualiza,on  of  chained  pipeline  jobs  
    

    View Slide

74. View Slide

75. > gradle qa!
    :askQuestions !
    !
    BUILD SUCCESSFUL!
    !
    Total time: 300 secs
    

    View Slide