Provisioning virtualized infrastructure with Gradle

Provisioning virtualized infrastructure with Gradle

Server infrastructure and configuration plays a crucial role in every build pipeline. It provides the runtime environment for your application and is used as basis for automated acceptance and exploratory testing. Managing infrastructure manually is time-consuming and error-prone. Virtualization helps to provide a repeatable, "production-like" environment. In this talk, we'll discuss how Gradle integrates with popular tools for creating virtual machines (VMs) as part of a standardized deployment and release process.

The main focus of this session lies on the creation and integration of VMs in the context of Continuous Delivery. We'll look at issues such as creating VM images, bootstrapping a VM as part of your build lifecycle as well as executing functional and smoke tests against a running instance. We'll round out this talk by modeling these steps as orchestrated jobs on a Continuous Integration server.

8f2248c6bfcc6df39a2cd8edf4267cb5?s=128

Benjamin Muschko

June 13, 2014
Tweet

Transcript

  1. 7.

    Easy to create and revert back to "good" known state

    Developer machine Initial provision Changed modify Destroyed tear down
  2. 8.

    Virtual Machines versus containers Hardware Host OS VM Guest OS

    Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries VM Guest OS Processes Binaries Hardware Host OS Binaries / Libraries Container Container Container
  3. 9.
  4. 11.
  5. 12.

    Vagrant quickstart $ vagrant up $ vagrant init hashicorp/precise64 $

    vagrant ssh 1 2 3 Creates Vagrantfile with initialized with box image Downloads box image and starts virtual machine SSH into running Vagrant virtual machine
  6. 13.

    Gradle Vagrant plugin composition vagrant vagrant-base ‣ adds custom task

    types ‣ validates Vagrant installation ‣ adds standard tasks to project ‣ configures default conventions ‣ exposes DSL applies
  7. 14.

    buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-vagrant-plugin:0.6'

    } } apply plugin: 'vagrant' vagrant { boxDir = file('~/dev/my-vagrant-box') } Using convention plugin gradlew vagrantUp vagrant Points to Vagrant box definition Adds Vagrant plugin to build
  8. 15.

    import org.gradle.api.plugins.vagrant.tasks.* ext { fusionBoxDir = file('fusion-box') fusionProvider = 'vmware_fusion'

    } task startFusionVm(type: VagrantUp) { description = 'Starts VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } task stopFusionVm(type: VagrantDestroy) { description = 'Stops VM machine running on VMware Fusion.' group = 'Fusion VM' boxDir = fusionBoxDir provider = fusionProvider } User-defined Vagrant tasks vagrant-base Stands up VM with VMware Fusion provider Tears down VM
  9. 16.

    import org.gradle.api.plugins.vagrant.tasks.VagrantSsh task vagrantEcho(type: VagrantSsh) { description = 'Runs remote

    SSH command in Vagrant box.' sshCommand = "echo 'hello'" dependsOn vagrantUp finalizedBy vagrantDestroy } Executing SSH commands in VM gradlew vagrantEcho Definition of SSH command Vagrant SSH custom task Starts/stops VM Start VM Execute SSH Stop VM 1 2 3
  10. 17.

    vagrant { boxDir = file('~/dev/my-vagrant-box') environmentVariables { variable 'IP', '192.168.1.33'

    } } Controlling logic with env. variables Defines key/value pairs Vagrant.configure("2") do |config| config.vm.network "private_network", ip: ENV['IP'] end dynamic values via env. variables
  11. 20.

    ext { vmIp = '192.168.1.33' tomcatPort = 8080 } sourceSets

    { smokeTest { groovy.srcDir file('src/smokeTest/groovy') resources.srcDir file('src/smokeTest/resources') compileClasspath = sourceSets.main.output + configurations.testRuntime runtimeClasspath = output + compileClasspath } } task smokeTest(type: Test) { testClassesDir = sourceSets.smokeTest.output.classesDir classpath = sourceSets.smokeTest.runtimeClasspath systemProperty 'vmIp', vmIp systemProperty 'tomcatPort', tomcatPort } Smoke tests definition Organizing test classes as dedicated source set Inject runtime parameters gradlew smokeTest
  12. 21.

    dependencies { compile 'org.codehaus.groovy:groovy-all:2.2.2' testCompile 'org.apache.httpcomponents:httpclient:4.3.3' testCompile 'org.spockframework:spock-core:0.7-groovy-2.0' } smokeTest.mustRunAfter

    vagrantUp task startVmAndVerify { dependsOn vagrantUp, smokeTest finalizedBy vagrantDestroy } Smoke tests as part of build lifecycle Spock and HttpClient libraries used for tests Run smoke test after standing up VM gradlew startVmAndVerify Start VM Smoke tests Stop VM 1 2 3
  13. 22.

    package org.gradle.vm.smoke import org.apache.http.HttpResponse import spock.lang.Unroll class TomcatHttpSmokeTest extends AbstractHttpSmokeTest

    { @Unroll def "Can resolve #name with URL #url"() { when: HttpConnectionParams params = new HttpConnectionParams(retries: 10) HttpResponse httpResponse = get(url, params) then: assert httpResponse.statusLine.statusCode == HttpURLConnection.HTTP_OK, "Failed to reach '$url'. Reason: $httpResponse.statusLine" where: name | url 'dashboard' | tomcatBaseUrl 'hello world' | "$tomcatBaseUrl/examples/servlets/servlet/HelloWorldExample" } } Testing Tomcat availability with Spock Provides heavy lifting of making HTTP calls Data driven testing
  14. 23.

    package org.gradle.vm.smoke import org.apache.http.client.methods.* import org.apache.http.impl.client.* import spock.lang.Specification abstract class

    AbstractHttpSmokeTest extends Specification { protected HttpResponse get(String url, String contextPath = null, Map queryParams = null, HttpConnectionParams httpConnectionParams) { CloseableHttpClient httpClient try { httpClient = buildHttpClient(httpConnectionParams) URI uri = buildURI(url, contextPath, queryParams) HttpGet httpGet = new HttpGet(uri) return httpClient.execute(httpGet) } finally { httpClient?.close() } } } Abstract class for HTTP-based tests Use retry handler Apache HTTPClient highly stable and configurable
  15. 24.

    import org.gradle.api.plugins.vagrant.tasks.VagrantSsh version = '1.5' task deployWebAppToVM(type: VagrantSsh) { description

    = 'Deploys web application.' sshCommand = "scp http://binrepo:9090/internal/myapp-${project.version}.war /opt/tomcat/webapps" dependsOn vagrantUp finalizedBy vagrantDestroy } Pull deployment of a web app gradlew deployWebAppToVM SCP web artifact from binary repository Start VM Deploy app Stop VM 1 2 3
  16. 26.

    task functionalTest(type: Test) { testClassesDir = sourceSets.functionalTest.output.classesDir classpath = sourceSets.functionalTest.runtimeClasspath

    reports.html.destination = file("$reports.html.destination/functional") reports.junitXml.destination = file("$reports.junitXml.destination/functional") systemProperty 'geb.env', 'firefox' systemProperty 'geb.build.reportsDir', reporting.file("$name/geb") mustRunAfter deployWebAppToVM dependsOn smokeTest } Automate most important use cases as functional tests gradlew functionalTest Only run tests after environment is deemed "functional"
  17. 27.
  18. 28.

    pull image send command manages Docker architecture build image Developer

    machine Public Docker image repository push image Local Repository Container Dockerfile uses image images
  19. 29.

    Docker quickstart $ docker run busybox /bin/echo hello world $

    docker pull busybox 1 2 Pulls minimal Linux image from Docker repository Runs image in container Executes echo command
  20. 30.

    buildscript { repositories { jcenter() } dependencies { classpath 'org.gradle.api.plugins:gradle-docker-plugin:0.1'

    } } apply plugin: 'docker' docker { serverUrl = 'http://remote.docker.com:4243' } Using the Gradle Docker plugin docker Points to Docker server URL exposing remote API Adds Docker plugin to build
  21. 31.

    import org.gradle.api.plugins.docker.tasks.image.DockerBuildImage task buildMyAppImage(type: DockerBuildImage) { inputDir = file('docker/myapp') tag

    = 'bmuschko/myapp' } Building a Docker image Builds image from Dockerfile FROM ubuntu MAINTAINER Benjamin Muschko <benjamin.muschko@gmail.com> RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list RUN apt-get update ... Dockerfile instructions
  22. 32.

    import org.gradle.api.plugins.docker.tasks.container.* task createMyAppContainer(type: DockerCreateContainer) { dependsOn buildMyAppImage imageId =

    'bmuschko/myapp' } task startMyAppContainer(type: DockerStartContainer) { dependsOn createMyAppContainer targetContainerId { createMyAppContainer.getContainerId() } } task stopMyAppContainer(type: DockerStopContainer) { targetContainerId { createMyAppContainer.getContainerId() } } Run container with image docker Creates container Use container ID for starting /stopping container
  23. 34.

    Exemplary build pipeline with Docker Commit stage Image creation Autom.

    accept. testing Manual testing Release pull image publish artifact build & push image download artifact run container run container run container pull image pull image