$30 off During Our Annual Pro Sale. View Details »

Provisioning virtualized infrastructure with Gradle

Provisioning virtualized infrastructure with Gradle

Server infrastructure and configuration plays a crucial role in every build pipeline. It provides the runtime environment for your application and is used as basis for automated acceptance and exploratory testing. Managing infrastructure manually is time-consuming and error-prone. Virtualization helps to provide a repeatable, "production-like" environment. In this talk, we'll discuss how Gradle integrates with popular tools for creating virtual machines (VMs) as part of a standardized deployment and release process.

The main focus of this session lies on the creation and integration of VMs in the context of Continuous Delivery. We'll look at issues such as creating VM images, bootstrapping a VM as part of your build lifecycle as well as executing functional and smoke tests against a running instance. We'll round out this talk by modeling these steps as orchestrated jobs on a Continuous Integration server.

Benjamin Muschko

June 13, 2014
Tweet

More Decks by Benjamin Muschko

Other Decks in Programming

Transcript

  1. Provisioning virtualized
    infrastructure with Gradle
    Benjamin Muschko
    Principal Engineer, Gradleware
    @bmuschko

    View Slide

  2. Manual infrastructure maintenance

    View Slide

  3. Virtualization and "Infrastructure
    as Code" to the rescue

    View Slide

  4. Continuous Delivery
    Demo Creation
    Environment Parity
    Dev Environment

    View Slide

  5. Developer
    machine
    Infrastructure code
    edit
    commit
    pull
    Proper configuration management

    View Slide

  6. Developer
    machine
    Infrastructure code
    VMs
    edit
    provision
    Standardized, consolidated
    environments

    View Slide

  7. Easy to create and revert back to
    "good" known state
    Developer
    machine Initial
    provision
    Changed
    modify
    Destroyed
    tear down

    View Slide

  8. Virtual Machines versus containers
    Hardware
    Host OS
    VM
    Guest OS
    Processes
    Binaries
    VM
    Guest OS
    Processes
    Binaries
    VM
    Guest OS
    Processes
    Binaries
    VM
    Guest OS
    Processes
    Binaries
    Hardware
    Host OS
    Binaries / Libraries
    Container
    Container
    Container

    View Slide

  9. View Slide

  10. Vagrant prerequisites
    Configures and bootstraps
    virtual machine
    Virtualization runtime
    environment
    +

    View Slide

  11. send
    command manages
    Vagrant architecture
    Provisioner
    Vagrantfile
    invokes
    VM
    Developer
    machine
    Public Vagrant
    box catalog
    pull box share box

    View Slide

  12. Vagrant quickstart
    $ vagrant up
    $ vagrant init hashicorp/precise64
    $ vagrant ssh
    1
    2
    3
    Creates Vagrantfile with
    initialized with box image
    Downloads box image and
    starts virtual machine
    SSH into running Vagrant
    virtual machine

    View Slide

  13. Gradle Vagrant plugin composition
    vagrant
    vagrant-base
    ‣ adds custom task types
    ‣ validates Vagrant installation
    ‣ adds standard tasks to project
    ‣ configures default conventions
    ‣ exposes DSL
    applies

    View Slide

  14. buildscript {
    repositories {
    jcenter()
    }
    dependencies {
    classpath 'org.gradle.api.plugins:gradle-vagrant-plugin:0.6'
    }
    }
    apply plugin: 'vagrant'
    vagrant {
    boxDir = file('~/dev/my-vagrant-box')
    }
    Using convention plugin
    gradlew vagrantUp
    vagrant
    Points to Vagrant box definition
    Adds Vagrant plugin to build

    View Slide

  15. import org.gradle.api.plugins.vagrant.tasks.*
    ext {
    fusionBoxDir = file('fusion-box')
    fusionProvider = 'vmware_fusion'
    }
    task startFusionVm(type: VagrantUp) {
    description = 'Starts VM machine running on VMware Fusion.'
    group = 'Fusion VM'
    boxDir = fusionBoxDir
    provider = fusionProvider
    }
    task stopFusionVm(type: VagrantDestroy) {
    description = 'Stops VM machine running on VMware Fusion.'
    group = 'Fusion VM'
    boxDir = fusionBoxDir
    provider = fusionProvider
    }
    User-defined Vagrant tasks
    vagrant-base
    Stands up VM with VMware
    Fusion provider
    Tears down VM

    View Slide

  16. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh
    task vagrantEcho(type: VagrantSsh) {
    description = 'Runs remote SSH command in Vagrant box.'
    sshCommand = "echo 'hello'"
    dependsOn vagrantUp
    finalizedBy vagrantDestroy
    }
    Executing SSH commands in VM
    gradlew vagrantEcho
    Definition of SSH command
    Vagrant SSH custom task
    Starts/stops VM
    Start VM Execute SSH Stop VM
    1 2 3

    View Slide

  17. vagrant {
    boxDir = file('~/dev/my-vagrant-box')
    environmentVariables {
    variable 'IP', '192.168.1.33'
    }
    }
    Controlling logic with env. variables
    Defines key/value pairs
    Vagrant.configure("2") do |config|
    config.vm.network "private_network", ip: ENV['IP']
    end
    dynamic values via
    env. variables

    View Slide

  18. Let there be smoke!

    View Slide

  19. Test source code organization
    Production source
    Unit test source
    Smoke test source

    View Slide

  20. ext {
    vmIp = '192.168.1.33'
    tomcatPort = 8080
    }
    sourceSets {
    smokeTest {
    groovy.srcDir file('src/smokeTest/groovy')
    resources.srcDir file('src/smokeTest/resources')
    compileClasspath = sourceSets.main.output + configurations.testRuntime
    runtimeClasspath = output + compileClasspath
    }
    }
    task smokeTest(type: Test) {
    testClassesDir = sourceSets.smokeTest.output.classesDir
    classpath = sourceSets.smokeTest.runtimeClasspath
    systemProperty 'vmIp', vmIp
    systemProperty 'tomcatPort', tomcatPort
    }
    Smoke tests definition
    Organizing test classes as
    dedicated source set
    Inject runtime parameters gradlew smokeTest

    View Slide

  21. dependencies {
    compile 'org.codehaus.groovy:groovy-all:2.2.2'
    testCompile 'org.apache.httpcomponents:httpclient:4.3.3'
    testCompile 'org.spockframework:spock-core:0.7-groovy-2.0'
    }
    smokeTest.mustRunAfter vagrantUp
    task startVmAndVerify {
    dependsOn vagrantUp, smokeTest
    finalizedBy vagrantDestroy
    }
    Smoke tests as part of build lifecycle
    Spock and HttpClient
    libraries used for tests
    Run smoke test after
    standing up VM
    gradlew startVmAndVerify
    Start VM Smoke tests Stop VM
    1 2 3

    View Slide

  22. package org.gradle.vm.smoke
    import org.apache.http.HttpResponse
    import spock.lang.Unroll
    class TomcatHttpSmokeTest extends AbstractHttpSmokeTest {
    @Unroll
    def "Can resolve #name with URL #url"() {
    when:
    HttpConnectionParams params = new HttpConnectionParams(retries: 10)
    HttpResponse httpResponse = get(url, params)
    then:
    assert httpResponse.statusLine.statusCode == HttpURLConnection.HTTP_OK,
    "Failed to reach '$url'. Reason: $httpResponse.statusLine"
    where:
    name | url
    'dashboard' | tomcatBaseUrl
    'hello world' | "$tomcatBaseUrl/examples/servlets/servlet/HelloWorldExample"
    }
    }
    Testing Tomcat availability with Spock
    Provides heavy lifting of
    making HTTP calls
    Data driven testing

    View Slide

  23. package org.gradle.vm.smoke
    import org.apache.http.client.methods.*
    import org.apache.http.impl.client.*
    import spock.lang.Specification
    abstract class AbstractHttpSmokeTest extends Specification {
    protected HttpResponse get(String url, String contextPath = null,
    Map queryParams = null, HttpConnectionParams httpConnectionParams) {
    CloseableHttpClient httpClient
    try {
    httpClient = buildHttpClient(httpConnectionParams)
    URI uri = buildURI(url, contextPath, queryParams)
    HttpGet httpGet = new HttpGet(uri)
    return httpClient.execute(httpGet)
    }
    finally {
    httpClient?.close()
    }
    }
    }
    Abstract class for HTTP-based tests
    Use retry handler
    Apache HTTPClient highly
    stable and configurable

    View Slide

  24. import org.gradle.api.plugins.vagrant.tasks.VagrantSsh
    version = '1.5'
    task deployWebAppToVM(type: VagrantSsh) {
    description = 'Deploys web application.'
    sshCommand = "scp http://binrepo:9090/internal/myapp-${project.version}.war
    /opt/tomcat/webapps"
    dependsOn vagrantUp
    finalizedBy vagrantDestroy
    }
    Pull deployment of a web app
    gradlew deployWebAppToVM
    SCP web artifact from
    binary repository
    Start VM Deploy app Stop VM
    1 2 3

    View Slide

  25. Keep acceptance
    tests green!

    View Slide

  26. task functionalTest(type: Test) {
    testClassesDir = sourceSets.functionalTest.output.classesDir
    classpath = sourceSets.functionalTest.runtimeClasspath
    reports.html.destination = file("$reports.html.destination/functional")
    reports.junitXml.destination = file("$reports.junitXml.destination/functional")
    systemProperty 'geb.env', 'firefox'
    systemProperty 'geb.build.reportsDir', reporting.file("$name/geb")
    mustRunAfter deployWebAppToVM
    dependsOn smokeTest
    }
    Automate most important use cases
    as functional tests
    gradlew functionalTest
    Only run tests after environment
    is deemed "functional"

    View Slide

  27. View Slide

  28. pull image
    send
    command manages
    Docker architecture
    build image
    Developer
    machine
    Public Docker
    image repository
    push image
    Local Repository
    Container
    Dockerfile
    uses image
    images

    View Slide

  29. Docker quickstart
    $ docker run busybox /bin/echo hello world
    $ docker pull busybox
    1
    2
    Pulls minimal Linux image
    from Docker repository
    Runs image in container Executes echo command

    View Slide

  30. buildscript {
    repositories {
    jcenter()
    }
    dependencies {
    classpath 'org.gradle.api.plugins:gradle-docker-plugin:0.1'
    }
    }
    apply plugin: 'docker'
    docker {
    serverUrl = 'http://remote.docker.com:4243'
    }
    Using the Gradle Docker plugin
    docker
    Points to Docker server URL exposing remote API
    Adds Docker plugin to build

    View Slide

  31. import org.gradle.api.plugins.docker.tasks.image.DockerBuildImage
    task buildMyAppImage(type: DockerBuildImage) {
    inputDir = file('docker/myapp')
    tag = 'bmuschko/myapp'
    }
    Building a Docker image
    Builds image from
    Dockerfile
    FROM ubuntu
    MAINTAINER Benjamin Muschko
    RUN echo "deb http://archive.ubuntu.com/ubuntu precise main universe" > /etc/apt/sources.list
    RUN apt-get update
    ...
    Dockerfile
    instructions

    View Slide

  32. import org.gradle.api.plugins.docker.tasks.container.*
    task createMyAppContainer(type: DockerCreateContainer) {
    dependsOn buildMyAppImage
    imageId = 'bmuschko/myapp'
    }
    task startMyAppContainer(type: DockerStartContainer) {
    dependsOn createMyAppContainer
    targetContainerId { createMyAppContainer.getContainerId() }
    }
    task stopMyAppContainer(type: DockerStopContainer) {
    targetContainerId { createMyAppContainer.getContainerId() }
    }
    Run container with image
    docker
    Creates container
    Use container ID for
    starting /stopping container

    View Slide

  33. Virtualization in build pipelines

    View Slide

  34. Exemplary build pipeline with Docker
    Commit
    stage
    Image
    creation
    Autom.
    accept.
    testing
    Manual
    testing
    Release
    pull
    image
    publish
    artifact
    build & push
    image
    download
    artifact
    run container run container run container
    pull
    image
    pull
    image

    View Slide

  35. References
    Vagrant
    http://www.vagrantup.com
    Gradle Vagrant Plugin
    https://github.com/bmuschko/gradle-vagrant-plugin
    Docker
    https://www.docker.io
    Gradle Docker Plugin
    https://github.com/bmuschko/gradle-docker-plugin

    View Slide

  36. > gradle askQuestions
    :askQuestions
    BUILD SUCCESSFUL
    Total time: 300 secs

    View Slide