Provisioning virtualized infrastructure with Gradle

Provisioning virtualized infrastructure with Gradle

Server infrastructure and configuration plays a crucial role in every build pipeline. It provides the runtime environment for your application and is used as basis for automated acceptance and exploratory testing. Managing infrastructure manually is time-consuming and error-prone. Virtualization helps to provide a repeatable, "production-like" environment. In this talk, we'll discuss how Gradle integrates with popular tools for creating virtual machines (VMs) as part of a standardized deployment and release process.

The main focus of this session lies on the creation and integration of VMs in the context of Continuous Delivery. We'll look at issues such as creating VM images, bootstrapping a VM as part of your build lifecycle as well as executing functional and smoke tests against a running instance. We'll round out this talk by modeling these steps as orchestrated jobs on a Continuous Integration server.


Benjamin Muschko

June 13, 2014


  1. Provisioning virtualized infrastructure with Gradle Benjamin Muschko Principal Engineer, Gradleware

  2. Manual infrastructure maintenance

  3. Virtualization and "Infrastructure as Code" to the rescue

  4. Continuous Delivery Demo Creation Environment Parity Dev Environment

  5. Developer machine Infrastructure code edit commit pull Proper configuration management

  6. Developer machine Infrastructure code VMs edit provision Standardized, consolidated environments

  7. Easy to create and revert back to "good" known state

    Developer machine Initial provision Changed modify Destroyed tear down
  8. Virtual Machines versus containers Hardware Host OS VM Guest OS

    Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries Hardware Host OS Binaries / Libraries Container Container Container
  9. None
  10. Vagrant prerequisites Configures and bootstraps virtual machine Virtualization runtime environment

  11. send command manages Vagrant architecture Provisioner Vagrantfile invokes VM Developer

    machine Public Vagrant box catalog pull box share box
  12. Vagrant quickstart $ vagrant up $ vagrant init hashicorp/precise64 $

    vagrant ssh 1 2 3 Creates Vagrantfile with initialized with box image Downloads box image and starts virtual machine SSH into running Vagrant virtual machine
  13. Gradle Vagrant plugin composition vagrant vagrant-base ‣ adds custom task

    types ‣ validates Vagrant installation ‣ adds standard tasks to project ‣ configures default conventions ‣ exposes DSL applies
  14. buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-vagrant-plugin:0.6'

    } } apply plugin: 'vagrant' vagrant { boxDir = file('~/dev/my-vagrant-box') } Using convention plugin gradlew vagrantUp vagrant Points to Vagrant box definition Adds Vagrant plugin to build
  15. import org.gradle.api.plugins.vagrant.tasks.* ext { fusionBoxDir = file('fusion-box') fusionProvider = 'vmware_fusion'

    } task startFusionVm(type: VagrantUp) { description = 'Starts VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } task stopFusionVm(type: VagrantDestroy) { description = 'Stops VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } User-defined Vagrant tasks vagrant-base Stands up VM with VMware Fusion provider Tears down VM
  16. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh task vagrantEcho(type: VagrantSsh) { description = 'Runs remote

    SSH command in Vagrant box.' sshCommand = "echo 'hello'" dependsOn vagrantUp finalizedBy vagrantDestroy } Executing SSH commands in VM gradlew vagrantEcho Definition of SSH command Vagrant SSH custom task Starts/stops VM Start VM Execute SSH Stop VM 1 2 3
  17. vagrant { boxDir = file('~/dev/my-vagrant-box') environmentVariables { variable 'IP', ''

    } } Controlling logic with env. variables Defines key/value pairs Vagrant.configure("2") do |config| "private_network", ip: ENV['IP'] end dynamic values via env. variables
  18. Let there be smoke!

  19. Test source code organization Production source Unit test source Smoke

    test source
  20. ext { vmIp = '' tomcatPort = 8080 } sourceSets

    { smokeTest { groovy.srcDir file('src/smokeTest/groovy') resources.srcDir file('src/smokeTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task smokeTest(type: Test) { testClassesDir = sourceSets.smokeTest.output.classesDir classpath = sourceSets.smokeTest.runtimeClasspath systemProperty 'vmIp', vmIp systemProperty 'tomcatPort', tomcatPort } Smoke tests definition Organizing test classes as dedicated source set Inject runtime parameters gradlew smokeTest
  21. dependencies { compile 'org.codehaus.groovy:groovy-all:2.2.2' testCompile 'org.apache.httpcomponents:httpclient:4.3.3' testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' } smokeTest.mustRunAfter

    vagrantUp task startVmAndVerify { dependsOn vagrantUp, smokeTest finalizedBy vagrantDestroy } Smoke tests as part of build lifecycle Spock and HttpClient libraries used for tests Run smoke test after standing up VM gradlew startVmAndVerify Start VM Smoke tests Stop VM 1 2 3
  22. package org.gradle.vm.smoke import org.apache.http.HttpResponse import spock.lang.Unroll class TomcatHttpSmokeTest extends AbstractHttpSmokeTest

    { @Unroll def "Can resolve #name with URL #url"() { when: HttpConnectionParams params = new HttpConnectionParams(retries: 10) HttpResponse httpResponse = get(url, params) then: assert httpResponse.statusLine.statusCode == HttpURLConnection.HTTP_OK, "Failed to reach '$url'. Reason: $httpResponse.statusLine" where: name | url 'dashboard' | tomcatBaseUrl 'hello world' | "$tomcatBaseUrl/examples/servlets/servlet/HelloWorldExample" } } Testing Tomcat availability with Spock Provides heavy lifting of making HTTP calls Data driven testing
  23. package org.gradle.vm.smoke import org.apache.http.client.methods.* import org.apache.http.impl.client.* import spock.lang.Specification abstract class

    AbstractHttpSmokeTest extends Specification { protected HttpResponse get(String url, String contextPath = null, Map queryParams = null, HttpConnectionParams httpConnectionParams) { CloseableHttpClient httpClient try { httpClient = buildHttpClient(httpConnectionParams) URI uri = buildURI(url, contextPath, queryParams) HttpGet httpGet = new HttpGet(uri) return httpClient.execute(httpGet) } finally { httpClient?.close() } } } Abstract class for HTTP-based tests Use retry handler Apache HTTPClient highly stable and configurable
  24. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh version = '1.5' task deployWebAppToVM(type: VagrantSsh) { description

    = 'Deploys web application.' sshCommand = "scp http://binrepo:9090/internal/myapp-${project.version}.war /opt/tomcat/webapps" dependsOn vagrantUp finalizedBy vagrantDestroy } Pull deployment of a web app gradlew deployWebAppToVM SCP web artifact from binary repository Start VM Deploy app Stop VM 1 2 3
  25. Keep acceptance tests green!

  26. task functionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath

    reports.html.destination = file("$reports.html.destination/functional") reports.junitXml.destination = file("$reports.junitXml.destination/functional") systemProperty 'geb.env', 'firefox' systemProperty '', reporting.file("$name/geb") mustRunAfter deployWebAppToVM dependsOn smokeTest } Automate most important use cases as functional tests gradlew functionalTest Only run tests after environment is deemed "functional"
  27. None
  28. pull image send command manages Docker architecture build image Developer

    machine Public Docker image repository push image Local Repository Container Dockerfile uses image images
  29. Docker quickstart $ docker run busybox /bin/echo hello world $

    docker pull busybox 1 2 Pulls minimal Linux image from Docker repository Runs image in container Executes echo command
  30. buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-docker-plugin:0.1'

    } } apply plugin: 'docker' docker { serverUrl = '' } Using the Gradle Docker plugin docker Points to Docker server URL exposing remote API Adds Docker plugin to build
  31. import org.gradle.api.plugins.docker.tasks.image.DockerBuildImage task buildMyAppImage(type: DockerBuildImage) { inputDir = file('docker/myapp') tag

    = 'bmuschko/myapp' } Building a Docker image Builds image from Dockerfile FROM ubuntu MAINTAINER Benjamin Muschko <> RUN echo "deb precise main universe" > /etc/apt/sources.list RUN apt-get update ... Dockerfile instructions
  32. import org.gradle.api.plugins.docker.tasks.container.* task createMyAppContainer(type: DockerCreateContainer) { dependsOn buildMyAppImage imageId =

    'bmuschko/myapp' } task startMyAppContainer(type: DockerStartContainer) { dependsOn createMyAppContainer targetContainerId { createMyAppContainer.getContainerId() } } task stopMyAppContainer(type: DockerStopContainer) { targetContainerId { createMyAppContainer.getContainerId() } } Run container with image docker Creates container Use container ID for starting /stopping container
  33. Virtualization in build pipelines

  34. Exemplary build pipeline with Docker Commit stage Image creation Autom.

    accept. testing Manual testing Release pull image publish artifact build & push image download artifact run container run container run container pull image pull image
  35. References Vagrant Gradle Vagrant Plugin Docker Gradle

    Docker Plugin
  36. > gradle askQuestions :askQuestions BUILD SUCCESSFUL Total time: 300 secs