Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Continuous Integration for Infrastructure

Continuous Integration for Infrastructure

Lots of examples of emerging infrastructure testing patterns. Test Kitchen, Puppet, Docker, Serverspec, Packer, Bats and more. Presented at #craftconf in Budapest

Gareth Rushgrove

April 25, 2014
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Continuous
    Integration for
    Infrastructure
    Craft Conference, Budapest, 2014
    Gareth Rushgrove

    View Slide

  2. Who
    (Who is this person?)

    View Slide

  3. @garethr

    View Slide

  4. UK Government
    Digital Service

    View Slide

  5. View Slide

  6. View Slide

  7. What is Continuous
    Integration
    (And what has that to do with
    infrastructure)

    View Slide

  8. A common software
    development practice
    Gareth Rushgrove

    View Slide

  9. Test every commit
    Gareth Rushgrove

    View Slide

  10. Get fast feedback on
    changes
    Gareth Rushgrove

    View Slide

  11. Discover bugs early, when
    they are easier to fix
    Gareth Rushgrove

    View Slide

  12. Avoid working in isolation,
    encourage good team
    work
    Gareth Rushgrove

    View Slide

  13. Lots of good tools to start
    with
    Gareth Rushgrove

    View Slide

  14. Gareth Rushgrove
    jenkins-ci.org

    View Slide

  15. Gareth Rushgrove
    jetbrains.com/teamcity

    View Slide

  16. Gareth Rushgrove
    travis-ci.com

    View Slide

  17. Gareth Rushgrove
    wercker.com

    View Slide

  18. And lots more
    Gareth Rushgrove

    View Slide

  19. Infrastructure as code
    means your infrastructure
    is software
    Gareth Rushgrove

    View Slide

  20. So let’s apply traditional
    software engineering
    practices to infrastructure
    Gareth Rushgrove

    View Slide

  21. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove

    View Slide

  22. Configuration management!
    Acceptance testing!
    Verifying machine images !
    Load testing!
    Provisioning
    Gareth Rushgrove

    View Slide

  23. Configuration management!
    Acceptance testing!
    Verifying machine images !
    Load testing!
    Provisioning
    Gareth Rushgrove

    View Slide

  24. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove

    View Slide

  25. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove

    View Slide

  26. Infrastructure
    testing as code
    (A quick wrecker example)

    View Slide

  27. Define your continuous
    integration setup in code
    too
    Gareth Rushgrove

    View Slide

  28. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove

    View Slide

  29. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove
    1
    2
    3
    4

    View Slide

  30. View builds in a web
    browser
    Gareth Rushgrove

    View Slide

  31. Gareth Rushgrove

    View Slide

  32. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove

    View Slide

  33. Gareth Rushgrove

    View Slide

  34. You can do similar things
    with Jenkins Job Builder
    and with Travis
    Gareth Rushgrove

    View Slide

  35. Testing config
    management code
    (The easy bit)

    View Slide

  36. Gareth Rushgrove

    View Slide

  37. Both Chef and Puppet
    have good testing support
    Gareth Rushgrove

    View Slide

  38. Write unit tests for
    modules or cookbooks
    Gareth Rushgrove

    View Slide

  39. rspec-puppet or chefspec
    Gareth Rushgrove

    View Slide

  40. class sample {
    file { '/tmp/sample':
    ensure => present,
    }
    }
    sample.pp
    Gareth Rushgrove

    View Slide

  41. describe "sample" do
    it {
    should create_file('/tmp/sample')
    }
    end
    sample_spec.rb
    Gareth Rushgrove

    View Slide

  42. sample
    should contain File[/tmp/sample]
    !
    Finished in 0.3881 seconds
    1 example, 0 failures
    Gareth Rushgrove

    View Slide

  43. Linting, code coverage,
    syntax checking
    Gareth Rushgrove

    View Slide

  44. Lots of good material
    available on this subject
    Gareth Rushgrove

    View Slide

  45. Gareth Rushgrove

    View Slide

  46. Gareth Rushgrove

    View Slide

  47. Acceptance testing
    infrastructure code
    (Testing with real machines)

    View Slide

  48. You don’t need to be using
    config management tools
    to test your code
    Gareth Rushgrove

    View Slide

  49. Run code against short
    lived virtual machines and
    make assertions against
    the resulting state
    Gareth Rushgrove

    View Slide

  50. Gareth Rushgrove
    Test Kitchen

    View Slide

  51. First the code we want to
    test
    Gareth Rushgrove

    View Slide

  52. #!/bin/bash
    apt-get install ntp -y
    scripts/bootstrap.sh
    Gareth Rushgrove

    View Slide

  53. Then a simple test kitchen
    configuration
    Gareth Rushgrove

    View Slide

  54. ---
    driver:
    name: digitalocean
    !
    provisioner:
    name: shell
    !
    platforms:
    - name: ubuntu-12.04
    !
    suites:
    - name: default
    provisioner:
    script: scripts/bootstrap.sh
    .kitchen.yml
    Gareth Rushgrove

    View Slide

  55. You can provision
    machines anywhere with
    different drivers
    Gareth Rushgrove

    View Slide

  56. kitchen driver discover
    Gareth Rushgrove

    View Slide

  57. Gareth Rushgrove

    View Slide

  58. And have different
    platforms running in
    parallel
    Gareth Rushgrove

    View Slide

  59. platforms:
    - name: ubuntu-14.04
    - name: ubuntu-13.10
    - name: centos-6.5
    - name: centos-6.4
    - name: ubuntu-12.04
    - name: centos-6.4
    - name: centos-6.4
    - name: archlinux-2013.05
    .kitchen.yml
    Gareth Rushgrove

    View Slide

  60. And then write tests
    against the provisioned
    machines
    Gareth Rushgrove

    View Slide

  61. Gareth Rushgrove
    serverspec.org

    View Slide

  62. RSpec tests for your
    servers configured !
    by Puppet, Chef or
    anything else
    Gareth Rushgrove

    View Slide

  63. describe package('ntp') do
    it { should be_installed }
    end
    serverspec/sample_spec.rb
    Gareth Rushgrove

    View Slide

  64. describe service('ntp') do
    it { should be_enabled }
    it { should be_running }
    end
    serverspec/sample_spec.rb
    Gareth Rushgrove

    View Slide

  65. kitchen test -c
    Gareth Rushgrove

    View Slide

  66. Package "ntp"
    should be installed
    !
    Service "ntp"
    should be enabled
    should be running
    !
    Finished in 0.11191 seconds
    3 examples, 0 failures
    Finished verifying (0m7.39s).
    -----> Destroying ...
    Digital Ocean instance <1493500> destroyed.
    Finished destroying (0m4.47s).
    Finished testing (3m59.40s).
    -----> Kitchen is finished. (3m59.95s)
    Gareth Rushgrove

    View Slide

  67. Kitchen is finished. (3m59.95s)
    Gareth Rushgrove

    View Slide

  68. Gareth Rushgrove
    garethr/do-test-kitchen

    View Slide

  69. Gareth Rushgrove

    View Slide

  70. Verifying machine
    images
    (AMIs and other artefacts)

    View Slide

  71. Your virtual machine
    images need a pipeline
    too
    Gareth Rushgrove

    View Slide

  72. We’ll start with an example
    using AMIs on
    Gareth Rushgrove

    View Slide

  73. Gareth Rushgrove
    packer.io

    View Slide

  74. Packer is a tool for
    creating identical machine
    images for multiple
    platforms from a single
    source configuration
    Gareth Rushgrove

    View Slide

  75. Configuration is lots of
    JSON
    Gareth Rushgrove

    View Slide

  76. {
    "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
    },
    "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "eu-west-1",
    "source_ami": "ami-480bea3f",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-serverspec-example-{{timestamp}}"
    }],
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "file",
    template.json
    Gareth Rushgrove

    View Slide

  77. First build an AMI
    Gareth Rushgrove

    View Slide

  78. Then install some
    packages and related
    configuration
    Gareth Rushgrove

    View Slide

  79. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove

    View Slide

  80. First we’ll install puppet
    Gareth Rushgrove

    View Slide

  81. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove

    View Slide

  82. #!/bin/bash
    !
    # Install language locale as without can
    # interfere with package installation
    sudo apt-get install language-pack-en -y
    !
    # Install Ruby
    sudo apt-get update
    sudo apt-get install ruby1.9.1 -y
    !
    # Install puppet/facter
    sudo gem install puppet facter --no-ri --no-rdoc
    scripts/puppet.sh
    Gareth Rushgrove

    View Slide

  83. Then run puppet to
    configure the machine
    Gareth Rushgrove

    View Slide

  84. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove

    View Slide

  85. Packer supports a range
    of provisioners
    Gareth Rushgrove

    View Slide

  86. Shell, File upload, Puppet,
    Chef, Salt, Ansible
    Gareth Rushgrove

    View Slide

  87. How to test our built-but-
    not-saved image?
    Gareth Rushgrove

    View Slide

  88. Run serverspec tests from
    within the new image
    Gareth Rushgrove

    View Slide

  89. Only if the tests pass do
    we save the AMI
    Gareth Rushgrove

    View Slide

  90. Gareth Rushgrove
    garethr/packer-serverspec-example

    View Slide

  91. Gareth Rushgrove

    View Slide

  92. Linux containers are
    another good artefact
    example
    Gareth Rushgrove

    View Slide

  93. Testing!
    based on changes to the
    Dockerfile works in the
    same way
    Gareth Rushgrove

    View Slide

  94. Gareth Rushgrove
    garethr/docker-spec-example

    View Slide

  95. Gareth Rushgrove

    View Slide

  96. Continuous
    load testing
    (The power of aggregate assertions)

    View Slide

  97. Load testing is often!
    ad hoc
    Gareth Rushgrove

    View Slide

  98. Load testing is more than
    just an application concern
    Gareth Rushgrove

    View Slide

  99. Lots of open source tools
    Gareth Rushgrove

    View Slide

  100. JMeter, Funkload, Tsung,
    Grinder, HTTPerf, Siege,
    Gatling, Locust, Vegeta,
    Gor …
    Gareth Rushgrove

    View Slide

  101. Nothing is perfect for our
    purposes
    Gareth Rushgrove

    View Slide

  102. Gareth Rushgrove
    We’ll use Gatling as an
    example

    View Slide

  103. class ExampleSimulation extends Simulation {
    val httpConf = http.baseURL("http://www.example.com")
    !
    val sample = scenario("Simple example").exec(
    http("Homepage").get("/").check(status.is(200)))
    !
    setUp(sample.inject(ramp(3 users) over (10 seconds)))
    .protocols(httpConf)
    }
    ExampleSimulation.scala
    Gareth Rushgrove

    View Slide

  104. Aggregate assertions are
    the killer feature of Gatling
    Gareth Rushgrove

    View Slide

  105. .assertions(
    global.responseTime.mean.lessThan(750),
    )
    ExampleSimulation.scala
    Gareth Rushgrove

    View Slide

  106. .assertions(
    global.successfulRequests.percent.is(100),
    global.responseTime.max.lessThan(1000),
    global.responseTime.mean.lessThan(750),
    global.responseTime.percentile1.lessThan(800),
    global.requestsPerSec.greaterThan(10)
    )
    ExampleSimulation.scala
    Gareth Rushgrove

    View Slide

  107. Continuous load testing
    supplements rather than
    replaces ad hoc load
    testing
    Gareth Rushgrove

    View Slide

  108. Gareth Rushgrove
    garethr/gatling-demo

    View Slide

  109. Gareth Rushgrove

    View Slide

  110. Entire infrastructures
    as code
    (Testing out provisioning)

    View Slide

  111. So far we’ve tested units
    of infrastructure
    Gareth Rushgrove

    View Slide

  112. This next example tests
    an entire system
    Gareth Rushgrove

    View Slide

  113. As an example we’ll use
    Ansible
    Gareth Rushgrove

    View Slide

  114. Gareth Rushgrove
    ansible.com

    View Slide

  115. Specifically we’ll use the
    DigitalOcean module
    Gareth Rushgrove

    View Slide

  116. First we define our system
    in code
    Gareth Rushgrove

    View Slide

  117. ---
    droplets:
    - name: production.web.1
    - name: production.web.2
    - name: production.app.1
    size: 64
    - name: production.app.2
    size: 64
    - name: production.db.1
    size: 69
    vars/droplets.yml
    Gareth Rushgrove

    View Slide

  118. For testing we’ll use a
    combination of serverspec
    and bats
    Gareth Rushgrove

    View Slide

  119. Gareth Rushgrove
    sstephenson/bats

    View Slide

  120. bats is great for testing
    combinations of shell
    scripts
    Gareth Rushgrove

    View Slide

  121. #!/usr/bin/env bats
    !
    @test "addition using bc" {
    result="$(echo 2+2 | bc)"
    [ "$result" -eq 4 ]
    }
    !
    @test "addition using dc" {
    result="$(echo 2 2+p | dc)"
    [ "$result" -eq 4 ]
    }
    example.bats
    Gareth Rushgrove

    View Slide

  122. Run out provisioning code
    inside a bats test
    Gareth Rushgrove

    View Slide

  123. @test "no droplets exist" {
    run tugboat droplets
    [ "$status" -eq 0 ]
    [ "${lines[0]}" = "You don't appear to have any droplets." ]
    }
    !
    @test "provisioner runs successfully" {
    run ansible-playbook -i hosts provision_digital_ocean.yml
    [ "$status" -eq 0 ]
    }
    acceptance.bats
    Gareth Rushgrove

    View Slide

  124. Confirm we have started
    some machines
    Gareth Rushgrove

    View Slide

  125. @test “we can count the correct number of machines" {
    run bash -c "ansible -i hosts all --list-hosts | wc -
    l | tr -d ' '"
    [ "$status" -eq 0 ]
    [ "$output" -eq 5 ]
    }
    acceptance.bats
    Gareth Rushgrove

    View Slide

  126. Serverspec has a mode of
    operation where it runs
    against remote hosts
    Gareth Rushgrove

    View Slide

  127. @test "per host serverspec tests pass" {
    run rake spec
    [ "$status" -eq 0 ]
    }
    acceptance.bats
    Gareth Rushgrove

    View Slide

  128. bats acceptance.bats
    Gareth Rushgrove

    View Slide

  129. ✓ no droplets exist
    ✓ provisioner runs successfully
    ✓ we can count the correct number of hosts
    ✓ we can list the new hosts
    ✓ all host serverspec tests pass
    ✓ clean up all droplets
    ✓ droplets have been cleaned up
    !
    7 tests, 0 failures
    Gareth Rushgrove

    View Slide

  130. Gareth Rushgrove
    garethr/ansible-provisioner

    View Slide

  131. Other things to
    think about
    (This is just the start)

    View Slide

  132. Feature flags for
    infrastructure
    Gareth Rushgrove

    View Slide

  133. Testing software defined
    networks
    Gareth Rushgrove

    View Slide

  134. Building a complete
    infrastructure pipeline
    Gareth Rushgrove

    View Slide

  135. Moving from continuous
    integration to continuous
    deployment
    Gareth Rushgrove

    View Slide

  136. Questions?
    (And thanks for listening)

    View Slide