Provisioning virtualized infrastructure with Gradle

Server infrastructure and configuration plays a crucial role in every build pipeline. It provides the runtime environment for your application and is used as basis for automated acceptance and exploratory testing. Managing infrastructure manually is time-consuming and error-prone. Virtualization helps to provide a repeatable, "production-like" environment. In this talk, we'll discuss how Gradle integrates with popular tools for creating virtual machines (VMs) as part of a standardized deployment and release process.

The main focus of this session lies on the creation and integration of VMs in the context of Continuous Delivery. We'll look at issues such as creating VM images, bootstrapping a VM as part of your build lifecycle as well as executing functional and smoke tests against a running instance. We'll round out this talk by modeling these steps as orchestrated jobs on a Continuous Integration server.

Transcript

1. Provisioning virtualized infrastructure with Gradle Benjamin Muschko Principal Engineer, Gradleware @bmuschko
2. Manual infrastructure maintenance
3. Virtualization and "Infrastructure as Code" to the rescue
4. Continuous Delivery Demo Creation Environment Parity Dev Environment
5. Developer machine Infrastructure code edit commit pull Proper configuration management
6. Developer machine Infrastructure code VMs edit provision Standardized, consolidated environments
7. Easy to create and revert back to "good" known state Developer machine Initial provision Changed modify Destroyed tear down
8. Virtual Machines versus containers Hardware Host OS VM Guest OS Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries Hardware Host OS Binaries / Libraries Container Container Container
10. Vagrant prerequisites Configures and bootstraps virtual machine Virtualization runtime environment +
11. send command manages Vagrant architecture Provisioner Vagrantfile invokes VM Developer machine Public Vagrant box catalog pull box share box
12. Vagrant quickstart $ vagrant up $ vagrant init hashicorp/precise64 $ vagrant ssh 1 2 3 Creates Vagrantfile with initialized with box image Downloads box image and starts virtual machine SSH into running Vagrant virtual machine
13. Gradle Vagrant plugin composition vagrant vagrant-base ‣ adds custom task types ‣ validates Vagrant installation ‣ adds standard tasks to project ‣ configures default conventions ‣ exposes DSL applies
14. buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-vagrant-plugin:0.6' } } apply plugin: 'vagrant' vagrant { boxDir = file('~/dev/my-vagrant-box') } Using convention plugin gradlew vagrantUp vagrant Points to Vagrant box definition Adds Vagrant plugin to build
15. import org.gradle.api.plugins.vagrant.tasks.* ext { fusionBoxDir = file('fusion-box') fusionProvider = 'vmware_fusion' } task startFusionVm(type: VagrantUp) { description = 'Starts VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } task stopFusionVm(type: VagrantDestroy) { description = 'Stops VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } User-defined Vagrant tasks vagrant-base Stands up VM with VMware Fusion provider Tears down VM
16. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh task vagrantEcho(type: VagrantSsh) { description = 'Runs remote SSH command in Vagrant box.' sshCommand = "echo 'hello'" dependsOn vagrantUp finalizedBy vagrantDestroy } Executing SSH commands in VM gradlew vagrantEcho Definition of SSH command Vagrant SSH custom task Starts/stops VM Start VM Execute SSH Stop VM 1 2 3
17. vagrant { boxDir = file('~/dev/my-vagrant-box') environmentVariables { variable 'IP', '192.168.1.33' } } Controlling logic with env. variables Defines key/value pairs Vagrant.configure("2") do |config| config.vm.network "private_network", ip: ENV['IP'] end dynamic values via env. variables
18. Let there be smoke!
19. Test source code organization Production source Unit test source Smoke test source
20. ext { vmIp = '192.168.1.33' tomcatPort = 8080 } sourceSets { smokeTest { groovy.srcDir file('src/smokeTest/groovy') resources.srcDir file('src/smokeTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task smokeTest(type: Test) { testClassesDir = sourceSets.smokeTest.output.classesDir classpath = sourceSets.smokeTest.runtimeClasspath systemProperty 'vmIp', vmIp systemProperty 'tomcatPort', tomcatPort } Smoke tests definition Organizing test classes as dedicated source set Inject runtime parameters gradlew smokeTest
21. dependencies { compile 'org.codehaus.groovy:groovy-all:2.2.2' testCompile 'org.apache.httpcomponents:httpclient:4.3.3' testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' } smokeTest.mustRunAfter vagrantUp task startVmAndVerify { dependsOn vagrantUp, smokeTest finalizedBy vagrantDestroy } Smoke tests as part of build lifecycle Spock and HttpClient libraries used for tests Run smoke test after standing up VM gradlew startVmAndVerify Start VM Smoke tests Stop VM 1 2 3
22. package org.gradle.vm.smoke import org.apache.http.HttpResponse import spock.lang.Unroll class TomcatHttpSmokeTest extends AbstractHttpSmokeTest { @Unroll def "Can resolve #name with URL #url"() { when: HttpConnectionParams params = new HttpConnectionParams(retries: 10) HttpResponse httpResponse = get(url, params) then: assert httpResponse.statusLine.statusCode == HttpURLConnection.HTTP_OK, "Failed to reach '$url'. Reason: $httpResponse.statusLine" where: name | url 'dashboard' | tomcatBaseUrl 'hello world' | "$tomcatBaseUrl/examples/servlets/servlet/HelloWorldExample" } } Testing Tomcat availability with Spock Provides heavy lifting of making HTTP calls Data driven testing
23. package org.gradle.vm.smoke import org.apache.http.client.methods.* import org.apache.http.impl.client.* import spock.lang.Specification abstract class AbstractHttpSmokeTest extends Specification { protected HttpResponse get(String url, String contextPath = null, Map queryParams = null, HttpConnectionParams httpConnectionParams) { CloseableHttpClient httpClient try { httpClient = buildHttpClient(httpConnectionParams) URI uri = buildURI(url, contextPath, queryParams) HttpGet httpGet = new HttpGet(uri) return httpClient.execute(httpGet) } finally { httpClient?.close() } } } Abstract class for HTTP-based tests Use retry handler Apache HTTPClient highly stable and configurable
24. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh version = '1.5' task deployWebAppToVM(type: VagrantSsh) { description = 'Deploys web application.' sshCommand = "scp http://binrepo:9090/internal/myapp-${project.version}.war /opt/tomcat/webapps" dependsOn vagrantUp finalizedBy vagrantDestroy } Pull deployment of a web app gradlew deployWebAppToVM SCP web artifact from binary repository Start VM Deploy app Stop VM 1 2 3
25. Keep acceptance tests green!
26. task functionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath reports.html.destination = file("$reports.html.destination/functional") reports.junitXml.destination = file("$reports.junitXml.destination/functional") systemProperty 'geb.env', 'firefox' systemProperty 'geb.build.reportsDir', reporting.file("$name/geb") mustRunAfter deployWebAppToVM dependsOn smokeTest } Automate most important use cases as functional tests gradlew functionalTest Only run tests after environment is deemed "functional"
28. pull image send command manages Docker architecture build image Developer machine Public Docker image repository push image Local Repository Container Dockerfile uses image images
29. Docker quickstart $ docker run busybox /bin/echo hello world $ docker pull busybox 1 2 Pulls minimal Linux image from Docker repository Runs image in container Executes echo command
30. buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-docker-plugin:0.1' } } apply plugin: 'docker' docker { serverUrl = 'http://remote.docker.com:4243' } Using the Gradle Docker plugin docker Points to Docker server URL exposing remote API Adds Docker plugin to build
31. import org.gradle.api.plugins.docker.tasks.image.DockerBuildImage task buildMyAppImage(type: DockerBuildImage) { inputDir = file('docker/myapp') tag = 'bmuschko/myapp' } Building a Docker image Builds image from Dockerfile FROM ubuntu MAINTAINER Benjamin Muschko <benjamin.muschko@gmail.com> RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update ... Dockerfile instructions
32. import org.gradle.api.plugins.docker.tasks.container.* task createMyAppContainer(type: DockerCreateContainer) { dependsOn buildMyAppImage imageId = 'bmuschko/myapp' } task startMyAppContainer(type: DockerStartContainer) { dependsOn createMyAppContainer targetContainerId { createMyAppContainer.getContainerId() } } task stopMyAppContainer(type: DockerStopContainer) { targetContainerId { createMyAppContainer.getContainerId() } } Run container with image docker Creates container Use container ID for starting /stopping container
33. Virtualization in build pipelines
34. Exemplary build pipeline with Docker Commit stage Image creation Autom. accept. testing Manual testing Release pull image publish artifact build & push image download artifact run container run container run container pull image pull image
35. References Vagrant http://www.vagrantup.com Gradle Vagrant Plugin https://github.com/bmuschko/gradle-vagrant-plugin Docker https://www.docker.io Gradle Docker Plugin https://github.com/bmuschko/gradle-docker-plugin
36. > gradle askQuestions :askQuestions BUILD SUCCESSFUL Total time: 300 secs