Continuous Integration for Infrastructure

Continuous Integration for Infrastructure

Lots of examples of emerging infrastructure testing patterns. Test Kitchen, Puppet, Docker, Serverspec, Packer, Bats and more. Presented at #craftconf in Budapest

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

April 25, 2014
Tweet

Transcript

  1. Continuous Integration for Infrastructure Craft Conference, Budapest, 2014 Gareth Rushgrove

  2. Who (Who is this person?)

  3. @garethr

  4. UK Government Digital Service

  5. None
  6. None
  7. What is Continuous Integration (And what has that to do

    with infrastructure)
  8. A common software development practice Gareth Rushgrove

  9. Test every commit Gareth Rushgrove

  10. Get fast feedback on changes Gareth Rushgrove

  11. Discover bugs early, when they are easier to fix Gareth

    Rushgrove
  12. Avoid working in isolation, encourage good team work Gareth Rushgrove

  13. Lots of good tools to start with Gareth Rushgrove

  14. Gareth Rushgrove jenkins-ci.org

  15. Gareth Rushgrove jetbrains.com/teamcity

  16. Gareth Rushgrove travis-ci.com

  17. Gareth Rushgrove wercker.com

  18. And lots more Gareth Rushgrove

  19. Infrastructure as code means your infrastructure is software Gareth Rushgrove

  20. So let’s apply traditional software engineering practices to infrastructure Gareth

    Rushgrove
  21. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning

    Gareth Rushgrove
  22. Configuration management! Acceptance testing! Verifying machine images ! Load testing!

    Provisioning Gareth Rushgrove
  23. Configuration management! Acceptance testing! Verifying machine images ! Load testing!

    Provisioning Gareth Rushgrove
  24. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning

    Gareth Rushgrove
  25. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning

    Gareth Rushgrove
  26. Infrastructure testing as code (A quick wrecker example)

  27. Define your continuous integration setup in code too Gareth Rushgrove

  28. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code:

    sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove
  29. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code:

    sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove 1 2 3 4
  30. View builds in a web browser Gareth Rushgrove

  31. Gareth Rushgrove

  32. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code:

    sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove
  33. Gareth Rushgrove

  34. You can do similar things with Jenkins Job Builder and

    with Travis Gareth Rushgrove
  35. Testing config management code (The easy bit)

  36. Gareth Rushgrove

  37. Both Chef and Puppet have good testing support Gareth Rushgrove

  38. Write unit tests for modules or cookbooks Gareth Rushgrove

  39. rspec-puppet or chefspec Gareth Rushgrove

  40. class sample { file { '/tmp/sample': ensure => present, }

    } sample.pp Gareth Rushgrove
  41. describe "sample" do it { should create_file('/tmp/sample') } end sample_spec.rb

    Gareth Rushgrove
  42. sample should contain File[/tmp/sample] ! Finished in 0.3881 seconds 1

    example, 0 failures Gareth Rushgrove
  43. Linting, code coverage, syntax checking Gareth Rushgrove

  44. Lots of good material available on this subject Gareth Rushgrove

  45. Gareth Rushgrove

  46. Gareth Rushgrove

  47. Acceptance testing infrastructure code (Testing with real machines)

  48. You don’t need to be using config management tools to

    test your code Gareth Rushgrove
  49. Run code against short lived virtual machines and make assertions

    against the resulting state Gareth Rushgrove
  50. Gareth Rushgrove Test Kitchen

  51. First the code we want to test Gareth Rushgrove

  52. #!/bin/bash apt-get install ntp -y scripts/bootstrap.sh Gareth Rushgrove

  53. Then a simple test kitchen configuration Gareth Rushgrove

  54. --- driver: name: digitalocean ! provisioner: name: shell ! platforms:

    - name: ubuntu-12.04 ! suites: - name: default provisioner: script: scripts/bootstrap.sh .kitchen.yml Gareth Rushgrove
  55. You can provision machines anywhere with different drivers Gareth Rushgrove

  56. kitchen driver discover Gareth Rushgrove

  57. Gareth Rushgrove

  58. And have different platforms running in parallel Gareth Rushgrove

  59. platforms: - name: ubuntu-14.04 - name: ubuntu-13.10 - name: centos-6.5

    - name: centos-6.4 - name: ubuntu-12.04 - name: centos-6.4 - name: centos-6.4 - name: archlinux-2013.05 .kitchen.yml Gareth Rushgrove
  60. And then write tests against the provisioned machines Gareth Rushgrove

  61. Gareth Rushgrove serverspec.org

  62. RSpec tests for your servers configured ! by Puppet, Chef

    or anything else Gareth Rushgrove
  63. describe package('ntp') do it { should be_installed } end serverspec/sample_spec.rb

    Gareth Rushgrove
  64. describe service('ntp') do it { should be_enabled } it {

    should be_running } end serverspec/sample_spec.rb Gareth Rushgrove
  65. kitchen test -c Gareth Rushgrove

  66. Package "ntp" should be installed ! Service "ntp" should be

    enabled should be running ! Finished in 0.11191 seconds 3 examples, 0 failures Finished verifying <default-ubuntu-1310> (0m7.39s). -----> Destroying <default-ubuntu-1310>... Digital Ocean instance <1493500> destroyed. Finished destroying <default-ubuntu-1310> (0m4.47s). Finished testing <default-ubuntu-1310> (3m59.40s). -----> Kitchen is finished. (3m59.95s) Gareth Rushgrove
  67. Kitchen is finished. (3m59.95s) Gareth Rushgrove

  68. Gareth Rushgrove garethr/do-test-kitchen

  69. Gareth Rushgrove

  70. Verifying machine images (AMIs and other artefacts)

  71. Your virtual machine images need a pipeline too Gareth Rushgrove

  72. We’ll start with an example using AMIs on Gareth Rushgrove

  73. Gareth Rushgrove packer.io

  74. Packer is a tool for creating identical machine images for

    multiple platforms from a single source configuration Gareth Rushgrove
  75. Configuration is lots of JSON Gareth Rushgrove

  76. { "variables": { "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}", "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}" },

    "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "eu-west-1", "source_ami": "ami-480bea3f", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-serverspec-example-{{timestamp}}" }], "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "file", template.json Gareth Rushgrove
  77. First build an AMI Gareth Rushgrove

  78. Then install some packages and related configuration Gareth Rushgrove

  79. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type":

    "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
  80. First we’ll install puppet Gareth Rushgrove

  81. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type":

    "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
  82. #!/bin/bash ! # Install language locale as without can #

    interfere with package installation sudo apt-get install language-pack-en -y ! # Install Ruby sudo apt-get update sudo apt-get install ruby1.9.1 -y ! # Install puppet/facter sudo gem install puppet facter --no-ri --no-rdoc scripts/puppet.sh Gareth Rushgrove
  83. Then run puppet to configure the machine Gareth Rushgrove

  84. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type":

    "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
  85. Packer supports a range of provisioners Gareth Rushgrove

  86. Shell, File upload, Puppet, Chef, Salt, Ansible Gareth Rushgrove

  87. How to test our built-but- not-saved image? Gareth Rushgrove

  88. Run serverspec tests from within the new image Gareth Rushgrove

  89. Only if the tests pass do we save the AMI

    Gareth Rushgrove
  90. Gareth Rushgrove garethr/packer-serverspec-example

  91. Gareth Rushgrove

  92. Linux containers are another good artefact example Gareth Rushgrove

  93. Testing! based on changes to the Dockerfile works in the

    same way Gareth Rushgrove
  94. Gareth Rushgrove garethr/docker-spec-example

  95. Gareth Rushgrove

  96. Continuous load testing (The power of aggregate assertions)

  97. Load testing is often! ad hoc Gareth Rushgrove

  98. Load testing is more than just an application concern Gareth

    Rushgrove
  99. Lots of open source tools Gareth Rushgrove

  100. JMeter, Funkload, Tsung, Grinder, HTTPerf, Siege, Gatling, Locust, Vegeta, Gor

    … Gareth Rushgrove
  101. Nothing is perfect for our purposes Gareth Rushgrove

  102. Gareth Rushgrove We’ll use Gatling as an example

  103. class ExampleSimulation extends Simulation { val httpConf = http.baseURL("http://www.example.com") !

    val sample = scenario("Simple example").exec( http("Homepage").get("/").check(status.is(200))) ! setUp(sample.inject(ramp(3 users) over (10 seconds))) .protocols(httpConf) } ExampleSimulation.scala Gareth Rushgrove
  104. Aggregate assertions are the killer feature of Gatling Gareth Rushgrove

  105. .assertions( global.responseTime.mean.lessThan(750), ) ExampleSimulation.scala Gareth Rushgrove

  106. .assertions( global.successfulRequests.percent.is(100), global.responseTime.max.lessThan(1000), global.responseTime.mean.lessThan(750), global.responseTime.percentile1.lessThan(800), global.requestsPerSec.greaterThan(10) ) ExampleSimulation.scala Gareth Rushgrove

  107. Continuous load testing supplements rather than replaces ad hoc load

    testing Gareth Rushgrove
  108. Gareth Rushgrove garethr/gatling-demo

  109. Gareth Rushgrove

  110. Entire infrastructures as code (Testing out provisioning)

  111. So far we’ve tested units of infrastructure Gareth Rushgrove

  112. This next example tests an entire system Gareth Rushgrove

  113. As an example we’ll use Ansible Gareth Rushgrove

  114. Gareth Rushgrove ansible.com

  115. Specifically we’ll use the DigitalOcean module Gareth Rushgrove

  116. First we define our system in code Gareth Rushgrove

  117. --- droplets: - name: production.web.1 - name: production.web.2 - name:

    production.app.1 size: 64 - name: production.app.2 size: 64 - name: production.db.1 size: 69 vars/droplets.yml Gareth Rushgrove
  118. For testing we’ll use a combination of serverspec and bats

    Gareth Rushgrove
  119. Gareth Rushgrove sstephenson/bats

  120. bats is great for testing combinations of shell scripts Gareth

    Rushgrove
  121. #!/usr/bin/env bats ! @test "addition using bc" { result="$(echo 2+2

    | bc)" [ "$result" -eq 4 ] } ! @test "addition using dc" { result="$(echo 2 2+p | dc)" [ "$result" -eq 4 ] } example.bats Gareth Rushgrove
  122. Run out provisioning code inside a bats test Gareth Rushgrove

  123. @test "no droplets exist" { run tugboat droplets [ "$status"

    -eq 0 ] [ "${lines[0]}" = "You don't appear to have any droplets." ] } ! @test "provisioner runs successfully" { run ansible-playbook -i hosts provision_digital_ocean.yml [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove
  124. Confirm we have started some machines Gareth Rushgrove

  125. @test “we can count the correct number of machines" {

    run bash -c "ansible -i hosts all --list-hosts | wc - l | tr -d ' '" [ "$status" -eq 0 ] [ "$output" -eq 5 ] } acceptance.bats Gareth Rushgrove
  126. Serverspec has a mode of operation where it runs against

    remote hosts Gareth Rushgrove
  127. @test "per host serverspec tests pass" { run rake spec

    [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove
  128. bats acceptance.bats Gareth Rushgrove

  129. ✓ no droplets exist ✓ provisioner runs successfully ✓ we

    can count the correct number of hosts ✓ we can list the new hosts ✓ all host serverspec tests pass ✓ clean up all droplets ✓ droplets have been cleaned up ! 7 tests, 0 failures Gareth Rushgrove
  130. Gareth Rushgrove garethr/ansible-provisioner

  131. Other things to think about (This is just the start)

  132. Feature flags for infrastructure Gareth Rushgrove

  133. Testing software defined networks Gareth Rushgrove

  134. Building a complete infrastructure pipeline Gareth Rushgrove

  135. Moving from continuous integration to continuous deployment Gareth Rushgrove

  136. Questions? (And thanks for listening)