Building a Continuous Delivery Pipeline with Gradle and Jenkins

From DevNexus 2015

Transcript

1. Building a Continuous Delivery Pipeline with Gradle and Jenkins Gary Hale
2. ghale gary.hale@gradleware.com Java/Groovy developer Gradleware employee Compulsive Automator Gary Hale Presentation will be available on https://speakerdeck.com/ghale
3. Original content by Ben Muschko Modified and used with permission
4. Releases don’t have to be painful
5. Continuous Delivery Deliver software fast and frequently
6. #4 Done means released #3 Automated tests are essential #1 Every commit can result in a release #2 Automate everything! Principles
7. The Build as a Pipeline Automated manifestation of delivery process
8. Establish automated quality gates Build Quality In!
9. Test Compile/Unit Tests Integration Tests Code Analysis Package/Deploy UAT Prod Acceptance Tests
10. OK, so what now?
11. The “revolutionary” sample application
12. Multi-project dependencies
13. Project hierarchy
14. Project hierarchy
15. Project hierarchy Define project- specific behavior
16. Project hierarchy Defines which projects are taking part in the build
17. Project hierarchy Always use Wrapper to execute the build!
18. Project hierarchy Examples: ✓ Versioning strategy ✓ Integration and functional test setup ✓ Deployment functionality ✓ ... Externalize concerns into script plugins and organize them in a dedicated directory
19. Project artifacts JAR JAR WAR
20. Project artifacts JAR JAR WAR Deployable artifact
21. Stages in build pipeline Asserts that system works at a technical level
22. Stages in build pipeline Asserts that system works on a functional/non-functional level
23. Stages in build pipeline Trigger manually Trigger manually
24. Commit stage: Compile/unit tests Rapid feedback (< 5 mins) Run on every VCS check-in Priority: fix broken build
25. Commit stage: Integration tests Long running tests Environment setup Maintenance cost
26. Separate tests in project layout Integration test Java sources Production Java sources Unit test Java sources
27. Separate tests with SourceSets sourceSets { integrationTest { java.srcDir file('src/integTest/java') resources.srcDir file('src/integTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task integrationTest(type: Test) { description = 'Runs the integration tests.' group = 'verification' testClassesDir = sourceSets.integrationTest.output.classesDir classpath = sourceSets.integrationTest.runtimeClasspath testResultsDir = file("$testResultsDir/integration") } Custom test results Directory Set compile and runtime classpath gradlew integrationTest Set source and resources directory
28. Database integration tests
29. Database integration tests apply from: "$rootDir/gradle/databaseSetup.gradle" integrationTest.dependsOn startAndPrepareDatabase integrationTest.finalizedBy stopDatabase check.dependsOn integrationTest Separate complex setup logic into script plugin Integrate tasks into build lifecycle
30. Picking the “right” code coverage tool Cobertura Offline bytecode instrumentation Source code instrumentation Offline bytecode instrumentation On-the-fly bytecode instrumentation
31. On-the-fly bytecode instrumentation No modification to source or bytecode
32. apply plugin: “jacoco” test { jacoco { append = false destinationFile = file “$buildDir/jacoco/jacocoTest.exec” classDumpFile = file “$buildDir/jacoco/testDumpFile” } } integrationTest { jacoco { append = false destinationFile = file “$buildDir/jacoco/jacocoInt.exec” classDumpFile = file “$buildDir/jacoco/intDumpFile” } } Code coverage with JaCoCo Configures instrumentation for integration tests as well Jacoco extension is added to all tasks with type Test
33. Generating coverage reports .exec .exec .html .html
34. Commit stage: Code analysis Perform code health check Fail build for low quality Record progress over time
35. Static code analysis tools Checkstyle FindBugs apply plugin: 'pmd' pmd { ignoreFailures = true } tasks.withType(Pmd) { reports { xml.enabled = false html.enabled = true } } apply plugin: 'jdepend’ jdepend { toolVersion = '2.9.1' ignoreFailures = true } gradlew check
36. Measure quality over time with Sonar
37. Applying the Sonar Runner plugin apply plugin: 'sonar-runner' sonarRunner { sonarProperties { property 'sonar.projectName', 'todo' property 'sonar.projectDescription', 'A task management application' } } subprojects { sonarRunner { sonarProperties { property 'sonar.sourceEncoding', 'UTF-8' } } } gradlew sonarRunner
39. Commit stage: Assemble distribution Exclude env. configuration Include build information Versioning strategy
40. Versioning strategy 1.0-SNAPSHOT 1.0 …the Maven way Change version with Maven Release plugin during development when released
41. Versioning strategy 1.0.134 1.0.134 during development …the Continuous Delivery way 1.0.134 Project version number Jenkins build number when released
42. Versioning strategy …implemented with Gradle allprojects { apply from: "$rootDir/gradle/versioning.gradle" } Contains version implementation
43. Versioning strategy …implemented with Gradle ext.buildTimestamp = new Date().format('yyyy-MM-dd HH:mm:ss') version = new ProjectVersion(1, 0, System.env.SOURCE_BUILD_NUMBER) class ProjectVersion { Integer major Integer minor String build ProjectVersion(Integer major, Integer minor, String build) { this.major = major this.minor = minor this.build = build } @Override String toString() { String fullVersion = "$major.$minor" if(build) { fullVersion += ".$build” } fullVersion } } Jenkins Build Number Builds version String representation
44. Packaging the deployable artifact project(':web') { apply plugin: 'war' task createBuildInfoFile << { def buildInfoFile = new File("$buildDir/build-info.properties") Properties props = new Properties() props.setProperty('version', project.version.toString()) props.setProperty('timestamp', project.buildTimestamp) props.store(buildInfoFile.newWriter(), null) } war { dependsOn createBuildInfoFile baseName = 'todo' from(buildDir) { include 'build-info.properties' into('WEB-INF/classes') } } } Creates file containing build information Include build info file Into WAR distribution gradlew assemble
45. Commit stage: Publish binaries Version artifact(s) Use binary repository Publish once, then reuse Repository
46. Publishing the deployable artifact 1.0.34 1.0.32 1.0.33 1.0.34
47. Defining build configuration binaryRepository { url = 'http://mycompany.bin.repo:8081/artifactory' username = 'admin' password = 'password' name = 'libs-release-local' } environments { test { server { hostname = 'mycompany.test' port = 8099 context = 'todo' username = 'manager' password = 'manager' } } uat { server { hostname = 'mycompany.uat' port = 8199 context = 'todo' username = 'manager' password = 'manager' } } ... } Env-specific configuration Common configuration Read credentials from gradle.properties Read credentials from gradle.properties Read credentials from gradle.properties
48. Reading build configuration def env = project.hasProperty('env') ? project.getProperty('env') : 'test' logger.quiet "Loading configuration for environment '$env’." def configFile = file("$rootDir/gradle/config/buildConfig.groovy") def parsedConfig = new ConfigSlurper(env).parse(configFile.toURL()) allprojects { ext.config = parsedConfig } gradlew –Penv=uat ... Assign configuration to extra property
49. Using the Maven Publishing plugin apply plugin: 'maven-publish' ext.fullRepoUrl = "$config.binaryRepository.url/$config.binaryRepository.name” publishing { publications { webApp(MavenPublication) { from components.web } } repositories { maven { url fullRepoUrl credentials { username = config.binaryRepository.username password = config.binaryRepository.password } } } } Build repository URL from configuration gradlew publish
50. Acceptance stage: Retrieve binaries Request versioned artifact Store in temp. directory Repository
51. Downloading the deployable artifact 1.0.34 1.0.32 1.0.33 1.0.34 1.0.34 Test UAT Prod
52. Task for downloading artifact configurations { todo } dependencies { todo group: project.group, name: project.name, version: project.version.toString(), ext: 'war' } ext.downloadDir = file("$buildDir/download/artifacts") task fetchToDoWar(type: Copy) { from configurations.todo into downloadDir } gradlew fetchToDoWar Declare a Dependency on the WAR Copy it to the download directory
53. Acceptance stage: Deploy binaries Deployment on request Make it a reliable process Use process for all envs.
54. Deploying to multiple environments Test UAT Prod –Penv=prod –Penv=uat –Penv=test
55. Deployment with the Cargo plugin cargoDeployRemote.dependsOn fetchToDoWar, cargoUndeployRemote cargoUndeployRemote { onlyIf appContextStatus } cargo { containerId = 'tomcat7x' port = config.server.port deployable { file = downloadedArtifact context = config.server.context } remote { hostname = config.server.hostname username = config.server.username password = config.server.password } } Download artifact from binary repository and undeploy existing Only undeploy if URL context exists gradlew –Penv=uat cargoDeploy Use environment-specific configuration
56. Acceptance stage: Functional tests Test all UI permutations Test important use cases Run against different envs.
57. In-container functional tests
58. Local functional tests task functionalTest(type: Test) { ... } task functionalJettyRun(type: org.gradle.api.plugins.jetty.JettyRun) { httpPort = functionalJettyHttpPort stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey contextPath = functionalJettyContextPath daemon = true } task functionalJettyStop(type: org.gradle.api.plugins.jetty.JettyStop) { stopPort = functionalJettyStopPort stopKey = functionalJettyStopKey } functionalJettyRun.dependsOn functionalTestClasses functionalTest.dependsOn functionalJettyRun functionalTest.finalizedBy functionalJettyStop Functional test task Custom Jetty Run task Custom Jetty Stop task
59. Executing remote functional tests ext { functionalTestReportDir = file("$testReportDir/functional") functionalTestResultsDir = file("$testResultsDir/functional") functionalCommonSystemProperties = ['geb.env': 'firefox', 'geb.build.reportsDir': reporting.file("$name/geb")] } task remoteFunctionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath testReportDir = functionalTestReportDir testResultsDir = functionalTestResultsDir systemProperties functionalCommonSystemProperties systemProperty 'geb.build.baseUrl', "http://$config.server.hostname:$config.server.port/$config.server.context/" } gradlew –Penv=test remoteFunctionalTest Build URL from env. configuration Reuse setup properties
60. Going further: Capacity testing buildscript { repositories { mavenCentral() } dependencies { classpath "com.github.kulya:jmeter-gradle-plugin:1.3.1-2.9" } } apply plugin: com.github.kulya.gradle.plugins.jmeter.JmeterPlugin ext.loadTestResources = "$projectDir/src/loadTest/resources" jmeterRun.configure { jmeterTestFiles = [file("$loadTestResources/todo-test-plan.jmx")] jmeterPropertyFile = file("$loadTestResources/jmeter.properties") jmeterUserProperties = ["hostname=${config.server.hostname}, "port=${config.server.port}", "context=${config.server.context}"] logging.captureStandardError LogLevel.INFO } gradlew –Penv=uat jmeterRun
61. Let’s bring Jenkins into play! Test UAT Prod Deployment Acceptance Tests Publish Download Trigger Build Pull Source Code
62. Model pipeline as series of jobs Trigger job when SCM change is detected
63. Initial Jenkins Build Job
64. Build Name Setter Plugin
65. JaCoCo Plugin
66. Parameterized Trigger Plugin
67. Gradle Plugin
68. Gradle Plugin Clean task removes existing artifacts Always use the Wrapper!
69. Jenkins environment variable Expressive build name Build Name Setter Plugin
70. Next job to trigger if build is stable build number parameter provided to subsequent jobs Parameterized Trigger Plugin
71. Archive all files Only archive if build was successful Clone Workspace SCM Plugin
72. Point to separated test results Fail build if quality gate criteria are not met JaCoCo Plugin Point to JaCoCo files as well as source and class files
73. Reuse initial workspace Reuse initial build number Clone Workspace SCM Plugin Build Name Setter Plugin
74. Define the target environment Downstream job that requires manual execution Build Pipeline Plugin
75. Build Pipeline Plugin Visualization of chained pipeline jobs
76. > gradle qa :askQuestions BUILD SUCCESSFUL Total time: 300 secs