Provisioning virtualized infrastructure with Gradle

Server infrastructure and configuration plays a crucial role in every build pipeline. It provides the runtime environment for your application and is used as basis for automated acceptance and exploratory testing. Managing infrastructure manually is time-consuming and error-prone. Virtualization helps to provide a repeatable, "production-like" environment. In this talk, we'll discuss how Gradle integrates with popular tools for creating virtual machines (VMs) as part of a standardized deployment and release process.

The main focus of this session lies on the creation and integration of VMs in the context of Continuous Delivery. We'll look at issues such as creating VM images, bootstrapping a VM as part of your build lifecycle as well as executing functional and smoke tests against a running instance. We'll round out this talk by modeling these steps as orchestrated jobs on a Continuous Integration server.

Transcript

1. Provisioning virtualized
   infrastructure with Gradle
   Benjamin Muschko
   Principal Engineer, Gradleware
   @bmuschko
   

   View Slide

2. Manual infrastructure maintenance
   

   View Slide

3. Virtualization and "Infrastructure
   as Code" to the rescue
   

   View Slide

4. Continuous Delivery
   Demo Creation
   Environment Parity
   Dev Environment
   

   View Slide

5. Developer
   machine
   Infrastructure code
   edit
   commit
   pull
   Proper configuration management
   

   View Slide

6. Developer
   machine
   Infrastructure code
   VMs
   edit
   provision
   Standardized, consolidated
   environments
   

   View Slide

7. Easy to create and revert back to
   "good" known state
   Developer
   machine Initial
   provision
   Changed
   modify
   Destroyed
   tear down
   

   View Slide

8. Virtual Machines versus containers
   Hardware
   Host OS
   VM
   Guest OS
   Processes
   Binaries
   VM
   Guest OS
   Processes
   Binaries
   VM
   Guest OS
   Processes
   Binaries
   VM
   Guest OS
   Processes
   Binaries
   Hardware
   Host OS
   Binaries / Libraries
   Container
   Container
   Container
   

   View Slide

9. View Slide

10. Vagrant prerequisites
    Configures and bootstraps
    virtual machine
    Virtualization runtime
    environment
    +
    

    View Slide

11. send
    command manages
    Vagrant architecture
    Provisioner
    Vagrantfile
    invokes
    VM
    Developer
    machine
    Public Vagrant
    box catalog
    pull box share box
    

    View Slide

12. Vagrant quickstart
    $ vagrant up
    $ vagrant init hashicorp/precise64
    $ vagrant ssh
    1
    2
    3
    Creates Vagrantfile with
    initialized with box image
    Downloads box image and
    starts virtual machine
    SSH into running Vagrant
    virtual machine
    

    View Slide

13. Gradle Vagrant plugin composition
    vagrant
    vagrant-base
    ‣ adds custom task types
    ‣ validates Vagrant installation
    ‣ adds standard tasks to project
    ‣ configures default conventions
    ‣ exposes DSL
    applies
    

    View Slide

14. buildscript {
    repositories {
    jcenter()
    }
    dependencies {
    classpath 'org.gradle.api.plugins:gradle-vagrant-plugin:0.6'
    }
    }
    apply plugin: 'vagrant'
    vagrant {
    boxDir = file('~/dev/my-vagrant-box')
    }
    Using convention plugin
    gradlew vagrantUp
    vagrant
    Points to Vagrant box definition
    Adds Vagrant plugin to build
    

    View Slide

15. import org.gradle.api.plugins.vagrant.tasks.*
    ext {
    fusionBoxDir = file('fusion-box')
    fusionProvider = 'vmware_fusion'
    }
    task startFusionVm(type: VagrantUp) {
    description = 'Starts VM machine running on VMware Fusion.'
    group = 'Fusion VM'
    boxDir = fusionBoxDir
    provider = fusionProvider
    }
    task stopFusionVm(type: VagrantDestroy) {
    description = 'Stops VM machine running on VMware Fusion.'
    group = 'Fusion VM'
    boxDir = fusionBoxDir
    provider = fusionProvider
    }
    User-defined Vagrant tasks
    vagrant-base
    Stands up VM with VMware
    Fusion provider
    Tears down VM
    

    View Slide

16. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh
    task vagrantEcho(type: VagrantSsh) {
    description = 'Runs remote SSH command in Vagrant box.'
    sshCommand = "echo 'hello'"
    dependsOn vagrantUp
    finalizedBy vagrantDestroy
    }
    Executing SSH commands in VM
    gradlew vagrantEcho
    Definition of SSH command
    Vagrant SSH custom task
    Starts/stops VM
    Start VM Execute SSH Stop VM
    1 2 3
    

    View Slide

17. vagrant {
    boxDir = file('~/dev/my-vagrant-box')
    environmentVariables {
    variable 'IP', '192.168.1.33'
    }
    }
    Controlling logic with env. variables
    Defines key/value pairs
    Vagrant.configure("2") do |config|
    config.vm.network "private_network", ip: ENV['IP']
    end
    dynamic values via
    env. variables
    

    View Slide

18. Let there be smoke!
    

    View Slide

19. Test source code organization
    Production source
    Unit test source
    Smoke test source
    

    View Slide

20. ext {
    vmIp = '192.168.1.33'
    tomcatPort = 8080
    }
    sourceSets {
    smokeTest {
    groovy.srcDir file('src/smokeTest/groovy')
    resources.srcDir file('src/smokeTest/resources')
    compileClasspath = sourceSets.main.output + configurations.testRuntime
    runtimeClasspath = output + compileClasspath
    }
    }
    task smokeTest(type: Test) {
    testClassesDir = sourceSets.smokeTest.output.classesDir
    classpath = sourceSets.smokeTest.runtimeClasspath
    systemProperty 'vmIp', vmIp
    systemProperty 'tomcatPort', tomcatPort
    }
    Smoke tests definition
    Organizing test classes as
    dedicated source set
    Inject runtime parameters gradlew smokeTest
    

    View Slide

21. dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.2.2'
    testCompile 'org.apache.httpcomponents:httpclient:4.3.3'
    testCompile 'org.spockframework:spock-core:0.7-groovy-2.0'
    }
    smokeTest.mustRunAfter vagrantUp
    task startVmAndVerify {
    dependsOn vagrantUp, smokeTest
    finalizedBy vagrantDestroy
    }
    Smoke tests as part of build lifecycle
    Spock and HttpClient
    libraries used for tests
    Run smoke test after
    standing up VM
    gradlew startVmAndVerify
    Start VM Smoke tests Stop VM
    1 2 3
    

    View Slide

22. package org.gradle.vm.smoke
    import org.apache.http.HttpResponse
    import spock.lang.Unroll
    class TomcatHttpSmokeTest extends AbstractHttpSmokeTest {
    @Unroll
    def "Can resolve #name with URL #url"() {
    when:
    HttpConnectionParams params = new HttpConnectionParams(retries: 10)
    HttpResponse httpResponse = get(url, params)
    then:
    assert httpResponse.statusLine.statusCode == HttpURLConnection.HTTP_OK,
    "Failed to reach '$url'. Reason: $httpResponse.statusLine"
    where:
    name | url
    'dashboard' | tomcatBaseUrl
    'hello world' | "$tomcatBaseUrl/examples/servlets/servlet/HelloWorldExample"
    }
    }
    Testing Tomcat availability with Spock
    Provides heavy lifting of
    making HTTP calls
    Data driven testing
    

    View Slide

23. package org.gradle.vm.smoke
    import org.apache.http.client.methods.*
    import org.apache.http.impl.client.*
    import spock.lang.Specification
    abstract class AbstractHttpSmokeTest extends Specification {
    protected HttpResponse get(String url, String contextPath = null,
    Map queryParams = null, HttpConnectionParams httpConnectionParams) {
    CloseableHttpClient httpClient
    try {
    httpClient = buildHttpClient(httpConnectionParams)
    URI uri = buildURI(url, contextPath, queryParams)
    HttpGet httpGet = new HttpGet(uri)
    return httpClient.execute(httpGet)
    }
    finally {
    httpClient?.close()
    }
    }
    }
    Abstract class for HTTP-based tests
    Use retry handler
    Apache HTTPClient highly
    stable and configurable
    

    View Slide

24. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh
    version = '1.5'
    task deployWebAppToVM(type: VagrantSsh) {
    description = 'Deploys web application.'
    sshCommand = "scp http://binrepo:9090/internal/myapp-${project.version}.war
    /opt/tomcat/webapps"
    dependsOn vagrantUp
    finalizedBy vagrantDestroy
    }
    Pull deployment of a web app
    gradlew deployWebAppToVM
    SCP web artifact from
    binary repository
    Start VM Deploy app Stop VM
    1 2 3
    

    View Slide

25. Keep acceptance
    tests green!
    

    View Slide

26. task functionalTest(type: Test) {
    testClassesDir = sourceSets.functionalTest.output.classesDir
    classpath = sourceSets.functionalTest.runtimeClasspath
    reports.html.destination = file("$reports.html.destination/functional")
    reports.junitXml.destination = file("$reports.junitXml.destination/functional")
    systemProperty 'geb.env', 'firefox'
    systemProperty 'geb.build.reportsDir', reporting.file("$name/geb")
    mustRunAfter deployWebAppToVM
    dependsOn smokeTest
    }
    Automate most important use cases
    as functional tests
    gradlew functionalTest
    Only run tests after environment
    is deemed "functional"
    

    View Slide

27. View Slide

28. pull image
    send
    command manages
    Docker architecture
    build image
    Developer
    machine
    Public Docker
    image repository
    push image
    Local Repository
    Container
    Dockerfile
    uses image
    images
    

    View Slide

29. Docker quickstart
    $ docker run busybox /bin/echo hello world
    $ docker pull busybox
    1
    2
    Pulls minimal Linux image
    from Docker repository
    Runs image in container Executes echo command
    

    View Slide

30. buildscript {
    repositories {
    jcenter()
    }
    dependencies {
    classpath 'org.gradle.api.plugins:gradle-docker-plugin:0.1'
    }
    }
    apply plugin: 'docker'
    docker {
    serverUrl = 'http://remote.docker.com:4243'
    }
    Using the Gradle Docker plugin
    docker
    Points to Docker server URL exposing remote API
    Adds Docker plugin to build
    

    View Slide

31. import org.gradle.api.plugins.docker.tasks.image.DockerBuildImage
    task buildMyAppImage(type: DockerBuildImage) {
    inputDir = file('docker/myapp')
    tag = 'bmuschko/myapp'
    }
    Building a Docker image
    Builds image from
    Dockerfile
    FROM ubuntu
    MAINTAINER Benjamin Muschko
    RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
    RUN apt-get update
    ...
    Dockerfile
    instructions
    

    View Slide

32. import org.gradle.api.plugins.docker.tasks.container.*
    task createMyAppContainer(type: DockerCreateContainer) {
    dependsOn buildMyAppImage
    imageId = 'bmuschko/myapp'
    }
    task startMyAppContainer(type: DockerStartContainer) {
    dependsOn createMyAppContainer
    targetContainerId { createMyAppContainer.getContainerId() }
    }
    task stopMyAppContainer(type: DockerStopContainer) {
    targetContainerId { createMyAppContainer.getContainerId() }
    }
    Run container with image
    docker
    Creates container
    Use container ID for
    starting /stopping container
    

    View Slide

33. Virtualization in build pipelines
    

    View Slide

34. Exemplary build pipeline with Docker
    Commit
    stage
    Image
    creation
    Autom.
    accept.
    testing
    Manual
    testing
    Release
    pull
    image
    publish
    artifact
    build & push
    image
    download
    artifact
    run container run container run container
    pull
    image
    pull
    image
    

    View Slide

35. References
    Vagrant
    http://www.vagrantup.com
    Gradle Vagrant Plugin
    https://github.com/bmuschko/gradle-vagrant-plugin
    Docker
    https://www.docker.io
    Gradle Docker Plugin
    https://github.com/bmuschko/gradle-docker-plugin
    

    View Slide

36. > gradle askQuestions
    :askQuestions
    BUILD SUCCESSFUL
    Total time: 300 secs
    

    View Slide