Building a Continuous Delivery Pipeline with Gradle and Jenkins

Slide 1

Slide 1 text

Building a Continuous Delivery Pipeline with Gradle and Jenkins Gary Hale

Slide 2

Slide 2 text

ghale [email protected] Java/Groovy developer Gradleware employee Compulsive Automator Gary Hale Presentation will be available on https://speakerdeck.com/ghale

Slide 3

Slide 3 text

Original content by Ben Muschko Modified and used with permission

Slide 4

Slide 4 text

Releases don’t have to be painful

Slide 5

Slide 5 text

Continuous Delivery Deliver software fast and frequently

Slide 6

Slide 6 text

#4 Done means released #3 Automated tests are essential #1 Every commit can result in a release #2 Automate everything! Principles

Slide 7

Slide 7 text

The Build as a Pipeline Automated manifestation of delivery process

Slide 8

Slide 8 text

Establish automated quality gates Build Quality In!

Slide 9

Slide 9 text

Test Compile/Unit Tests Integration Tests Code Analysis Package/Deploy UAT Prod Acceptance Tests

Slide 10

Slide 10 text

OK, so what now?

Slide 11

Slide 11 text

The “revolutionary” sample application

Slide 12

Slide 12 text

Multi-project dependencies

Slide 13

Slide 13 text

Project hierarchy

Slide 14

Slide 14 text

Project hierarchy

Slide 15

Slide 15 text

Project hierarchy Define project- specific behavior

Slide 16

Slide 16 text

Project hierarchy Defines which projects are taking part in the build

Slide 17

Slide 17 text

Project hierarchy Always use Wrapper to execute the build!

Slide 18

Slide 18 text

Project hierarchy Examples: ✓ Versioning strategy ✓ Integration and functional test setup ✓ Deployment functionality ✓ ... Externalize concerns into script plugins and organize them in a dedicated directory

Slide 19

Slide 19 text

Project artifacts JAR JAR WAR

Slide 20

Slide 20 text

Project artifacts JAR JAR WAR Deployable artifact

Slide 21

Slide 21 text

Stages in build pipeline Asserts that system works at a technical level

Slide 22

Slide 22 text

Stages in build pipeline Asserts that system works on a functional/non-functional level

Slide 23

Slide 23 text

Stages in build pipeline Trigger manually Trigger manually

Slide 24

Slide 24 text

Commit stage: Compile/unit tests Rapid feedback (< 5 mins) Run on every VCS check-in Priority: fix broken build

Slide 25

Slide 25 text

Commit stage: Integration tests Long running tests Environment setup Maintenance cost

Slide 26

Slide 26 text

Separate tests in project layout Integration test Java sources Production Java sources Unit test Java sources

Slide 27

Slide 27 text

Separate tests with SourceSets sourceSets { integrationTest { java.srcDir file('src/integTest/java') resources.srcDir file('src/integTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task integrationTest(type: Test) { description = 'Runs the integration tests.' group = 'verification' testClassesDir = sourceSets.integrationTest.output.classesDir classpath = sourceSets.integrationTest.runtimeClasspath testResultsDir = file("$testResultsDir/integration") } Custom test results Directory Set compile and runtime classpath gradlew integrationTest Set source and resources directory

Slide 28

Slide 28 text

Database integration tests

Slide 29

Slide 29 text

Database integration tests apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn startAndPrepareDatabase integrationTest.finalizedBy stopDatabase check.dependsOn integrationTest Separate complex setup logic into script plugin Integrate tasks into build lifecycle

Slide 30

Slide 30 text

Picking the “right” code coverage tool Cobertura Offline bytecode instrumentation Source code instrumentation Offline bytecode instrumentation On-the-fly bytecode instrumentation

Slide 31

Slide 31 text

On-the-fly bytecode instrumentation No modification to source or bytecode

Slide 32

Slide 32 text

apply plugin: “jacoco” test { jacoco { append = false destinationFile = file “$buildDir/jacoco/jacocoTest.exec” classDumpFile = file “$buildDir/jacoco/testDumpFile” } } integrationTest { jacoco { append = false destinationFile = file “$buildDir/jacoco/jacocoInt.exec” classDumpFile = file “$buildDir/jacoco/intDumpFile” } } Code coverage with JaCoCo Configures instrumentation for integration tests as well Jacoco extension is added to all tasks with type Test

Slide 33

Slide 33 text

Generating coverage reports .exec .exec .html .html

Slide 34

Slide 34 text

Commit stage: Code analysis Perform code health check Fail build for low quality Record progress over time

Slide 35

Slide 35 text

Static code analysis tools Checkstyle FindBugs apply plugin: 'pmd' pmd { ignoreFailures = true } tasks.withType(Pmd) { reports { xml.enabled = false html.enabled = true } } apply plugin: 'jdepend’ jdepend { toolVersion = '2.9.1' ignoreFailures = true } gradlew check

Slide 36

Slide 36 text

Measure quality over time with Sonar

Slide 37

Slide 37 text

Applying the Sonar Runner plugin apply plugin: 'sonar-runner' sonarRunner { sonarProperties { property 'sonar.projectName', 'todo' property 'sonar.projectDescription', 'A task management application' } } subprojects { sonarRunner { sonarProperties { property 'sonar.sourceEncoding', 'UTF-8' } } } gradlew sonarRunner

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Commit stage: Assemble distribution Exclude env. configuration Include build information Versioning strategy

Slide 40

Slide 40 text

Versioning strategy 1.0-SNAPSHOT 1.0 …the Maven way Change version with Maven Release plugin during development when released

Slide 41

Slide 41 text

Versioning strategy 1.0.134 1.0.134 during development …the Continuous Delivery way 1.0.134 Project version number Jenkins build number when released

Slide 42

Slide 42 text

Versioning strategy …implemented with Gradle allprojects { apply from: "$rootDir/gradle/versioning.gradle" } Contains version implementation

Slide 43

Slide 43 text

Versioning strategy …implemented with Gradle ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss') version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER) class ProjectVersion { Integer major Integer minor String build ProjectVersion(Integer major, Integer minor, String build) { this.major = major this.minor = minor this.build = build } @Override String toString() { String fullVersion = "$major.$minor" if(build) { fullVersion += ".$build” } fullVersion } } Jenkins Build Number Builds version String representation

Slide 44

Slide 44 text

Packaging the deployable artifact project(':web') { apply plugin: 'war' task createBuildInfoFile << { def buildInfoFile = new File("$buildDir/build-info.properties") Properties props = new Properties() props.setProperty('version', project.version.toString()) props.setProperty('timestamp', project.buildTimestamp) props.store(buildInfoFile.newWriter(), null) } war { dependsOn createBuildInfoFile baseName = 'todo' from(buildDir) { include 'build-info.properties' into('WEB-INF/classes') } } } Creates file containing build information Include build info file Into WAR distribution gradlew assemble

Slide 45

Slide 45 text

Commit stage: Publish binaries Version artifact(s) Use binary repository Publish once, then reuse Repository

Slide 46

Slide 46 text

Publishing the deployable artifact 1.0.34 1.0.32 1.0.33 1.0.34

Slide 47

Slide 47 text

Defining build configuration binaryRepository { url = 'http://mycompany.bin.repo:8081/artifactory' username = 'admin' password = 'password' name = 'libs-release-local' } environments { test { server { hostname = 'mycompany.test' port = 8099 context = 'todo' username = 'manager' password = 'manager' } } uat { server { hostname = 'mycompany.uat' port = 8199 context = 'todo' username = 'manager' password = 'manager' } } ... } Env-specific configuration Common configuration Read credentials from gradle.properties Read credentials from gradle.properties Read credentials from gradle.properties

Slide 48

Slide 48 text

Reading build configuration def env = project.hasProperty('env') ? project.getProperty('env') : 'test' logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy") def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL()) allprojects { ext.config = parsedConfig } gradlew –Penv=uat ... Assign configuration to extra property

Slide 49

Slide 49 text

Using the Maven Publishing plugin apply plugin: 'maven-publish' ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name” publishing { publications { webApp(MavenPublication) { from components.web } } repositories { maven { url fullRepoUrl credentials { username = config.binaryRepository.username password = config.binaryRepository.password } } } } Build repository URL from configuration gradlew publish

Slide 50

Slide 50 text

Acceptance stage: Retrieve binaries Request versioned artifact Store in temp. directory Repository

Slide 51

Slide 51 text

Downloading the deployable artifact 1.0.34 1.0.32 1.0.33 1.0.34 1.0.34 Test UAT Prod

Slide 52

Slide 52 text

Task for downloading artifact configurations { todo } dependencies { todo group: project.group, name: project.name, version: project.version.toString(), ext: 'war' } ext.downloadDir = file("$buildDir/download/artifacts") task fetchToDoWar(type: Copy) { from configurations.todo into downloadDir } gradlew fetchToDoWar Declare a Dependency on the WAR Copy it to the download directory

Slide 53

Slide 53 text

Acceptance stage: Deploy binaries Deployment on request Make it a reliable process Use process for all envs.

Slide 54

Slide 54 text

Deploying to multiple environments Test UAT Prod –Penv=prod –Penv=uat –Penv=test

Slide 55

Slide 55 text

Deployment with the Cargo plugin cargoDeployRemote.dependsOn fetchToDoWar, cargoUndeployRemote cargoUndeployRemote { onlyIf appContextStatus } cargo { containerId = 'tomcat7x' port = config.server.port deployable { file = downloadedArtifact context = config.server.context } remote { hostname = config.server.hostname username = config.server.username password = config.server.password } } Download artifact from binary repository and undeploy existing Only undeploy if URL context exists gradlew –Penv=uat cargoDeploy Use environment-specific configuration

Slide 56

Slide 56 text

Acceptance stage: Functional tests Test all UI permutations Test important use cases Run against different envs.

Slide 57

Slide 57 text

In-container functional tests

Slide 58

Slide 58 text

Local functional tests task functionalTest(type: Test) { ... } task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) { httpPort = functionalJettyHttpPort stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey contextPath = functionalJettyContextPath daemon = true } task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) { stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey } functionalJettyRun.dependsOn functionalTestClasses functionalTest.dependsOn functionalJettyRun functionalTest.finalizedBy functionalJettyStop Functional test task Custom Jetty Run task Custom Jetty Stop task

Slide 59

Slide 59 text

Executing remote functional tests ext { functionalTestReportDir = file("$testReportDir/functional") functionalTestResultsDir = file("$testResultsDir/functional") functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")] } task remoteFunctionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath testReportDir = functionalTestReportDir testResultsDir = functionalTestResultsDir systemProperties functionalCommonSystemProperties systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/" } gradlew –Penv=test remoteFunctionalTest Build URL from env. configuration Reuse setup properties

Slide 60

Slide 60 text

Going further: Capacity testing buildscript { repositories { mavenCentral() } dependencies { classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9" } } apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin ext.loadTestResources = "$projectDir/src/loadTest/resources" jmeterRun.configure { jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")] jmeterPropertyFile = file("$loadTestResources/jmeter.properties") jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"] logging.captureStandardError LogLevel.INFO } gradlew –Penv=uat jmeterRun

Slide 61

Slide 61 text

Let’s bring Jenkins into play! Test UAT Prod Deployment Acceptance Tests Publish Download Trigger Build Pull Source Code

Slide 62

Slide 62 text

Model pipeline as series of jobs Trigger job when SCM change is detected

Slide 63

Slide 63 text

Initial Jenkins Build Job

Slide 64

Slide 64 text

Build Name Setter Plugin

Slide 65

Slide 65 text

JaCoCo Plugin

Slide 66

Slide 66 text

Parameterized Trigger Plugin

Slide 67

Slide 67 text

Gradle Plugin

Slide 68

Slide 68 text

Gradle Plugin Clean task removes existing artifacts Always use the Wrapper!

Slide 69

Slide 69 text

Jenkins environment variable Expressive build name Build Name Setter Plugin

Slide 70

Slide 70 text

Next job to trigger if build is stable build number parameter provided to subsequent jobs Parameterized Trigger Plugin

Slide 71

Slide 71 text

Archive all files Only archive if build was successful Clone Workspace SCM Plugin

Slide 72

Slide 72 text

Point to separated test results Fail build if quality gate criteria are not met JaCoCo Plugin Point to JaCoCo files as well as source and class files

Slide 73

Slide 73 text

Reuse initial workspace Reuse initial build number Clone Workspace SCM Plugin Build Name Setter Plugin

Slide 74

Slide 74 text

Define the target environment Downstream job that requires manual execution Build Pipeline Plugin