Continuous Integration for Infrastructure

Lots of examples of emerging infrastructure testing patterns. Test Kitchen, Puppet, Docker, Serverspec, Packer, Bats and more. Presented at #craftconf in Budapest

Transcript

1. Continuous
   Integration for
   Infrastructure
   Craft Conference, Budapest, 2014
   Gareth Rushgrove
   

   View Slide

2. Who
   (Who is this person?)
   

   View Slide

3. @garethr
   

   View Slide

4. UK Government
   Digital Service
   

   View Slide

5. View Slide

6. View Slide

7. What is Continuous
   Integration
   (And what has that to do with
   infrastructure)
   

   View Slide

8. A common software
   development practice
   Gareth Rushgrove
   

   View Slide

9. Test every commit
   Gareth Rushgrove
   

   View Slide

10. Get fast feedback on
    changes
    Gareth Rushgrove
    

    View Slide

11. Discover bugs early, when
    they are easier to fix
    Gareth Rushgrove
    

    View Slide

12. Avoid working in isolation,
    encourage good team
    work
    Gareth Rushgrove
    

    View Slide

13. Lots of good tools to start
    with
    Gareth Rushgrove
    

    View Slide

14. Gareth Rushgrove
    jenkins-ci.org
    

    View Slide

15. Gareth Rushgrove
    jetbrains.com/teamcity
    

    View Slide

16. Gareth Rushgrove
    travis-ci.com
    

    View Slide

17. Gareth Rushgrove
    wercker.com
    

    View Slide

18. And lots more
    Gareth Rushgrove
    

    View Slide

19. Infrastructure as code
    means your infrastructure
    is software
    Gareth Rushgrove
    

    View Slide

20. So let’s apply traditional
    software engineering
    practices to infrastructure
    Gareth Rushgrove
    

    View Slide

21. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove
    

    View Slide

22. Configuration management!
    Acceptance testing!
    Verifying machine images !
    Load testing!
    Provisioning
    Gareth Rushgrove
    

    View Slide

23. Configuration management!
    Acceptance testing!
    Verifying machine images !
    Load testing!
    Provisioning
    Gareth Rushgrove
    

    View Slide

24. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove
    

    View Slide

25. Configuration management!
    Acceptance testing!
    Verifying machine images!
    Load testing!
    Provisioning
    Gareth Rushgrove
    

    View Slide

26. Infrastructure
    testing as code
    (A quick wrecker example)
    

    View Slide

27. Define your continuous
    integration setup in code
    too
    Gareth Rushgrove
    

    View Slide

28. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove
    

    View Slide

29. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove
    1
    2
    3
    4
    

    View Slide

30. View builds in a web
    browser
    Gareth Rushgrove
    

    View Slide

31. Gareth Rushgrove
    

    View Slide

32. - install-packages:
    packages: ruby1.9.1-dev
    - script:
    name: install bundler
    code: sudo gem install bundler
    - bundle-install
    - script:
    name: print ruby version
    code: ruby —version
    wercker.yaml
    Gareth Rushgrove
    

    View Slide

33. Gareth Rushgrove
    

    View Slide

34. You can do similar things
    with Jenkins Job Builder
    and with Travis
    Gareth Rushgrove
    

    View Slide

35. Testing config
    management code
    (The easy bit)
    

    View Slide

36. Gareth Rushgrove
    

    View Slide

37. Both Chef and Puppet
    have good testing support
    Gareth Rushgrove
    

    View Slide

38. Write unit tests for
    modules or cookbooks
    Gareth Rushgrove
    

    View Slide

39. rspec-puppet or chefspec
    Gareth Rushgrove
    

    View Slide

40. class sample {
    file { '/tmp/sample':
    ensure => present,
    }
    }
    sample.pp
    Gareth Rushgrove
    

    View Slide

41. describe "sample" do
    it {
    should create_file('/tmp/sample')
    }
    end
    sample_spec.rb
    Gareth Rushgrove
    

    View Slide

42. sample
    should contain File[/tmp/sample]
    !
    Finished in 0.3881 seconds
    1 example, 0 failures
    Gareth Rushgrove
    

    View Slide

43. Linting, code coverage,
    syntax checking
    Gareth Rushgrove
    

    View Slide

44. Lots of good material
    available on this subject
    Gareth Rushgrove
    

    View Slide

45. Gareth Rushgrove
    

    View Slide

46. Gareth Rushgrove
    

    View Slide

47. Acceptance testing
    infrastructure code
    (Testing with real machines)
    

    View Slide

48. You don’t need to be using
    config management tools
    to test your code
    Gareth Rushgrove
    

    View Slide

49. Run code against short
    lived virtual machines and
    make assertions against
    the resulting state
    Gareth Rushgrove
    

    View Slide

50. Gareth Rushgrove
    Test Kitchen
    

    View Slide

51. First the code we want to
    test
    Gareth Rushgrove
    

    View Slide

52. #!/bin/bash
    apt-get install ntp -y
    scripts/bootstrap.sh
    Gareth Rushgrove
    

    View Slide

53. Then a simple test kitchen
    configuration
    Gareth Rushgrove
    

    View Slide

54. ---
    driver:
    name: digitalocean
    !
    provisioner:
    name: shell
    !
    platforms:
    - name: ubuntu-12.04
    !
    suites:
    - name: default
    provisioner:
    script: scripts/bootstrap.sh
    .kitchen.yml
    Gareth Rushgrove
    

    View Slide

55. You can provision
    machines anywhere with
    different drivers
    Gareth Rushgrove
    

    View Slide

56. kitchen driver discover
    Gareth Rushgrove
    

    View Slide

57. Gareth Rushgrove
    

    View Slide

58. And have different
    platforms running in
    parallel
    Gareth Rushgrove
    

    View Slide

59. platforms:
    - name: ubuntu-14.04
    - name: ubuntu-13.10
    - name: centos-6.5
    - name: centos-6.4
    - name: ubuntu-12.04
    - name: centos-6.4
    - name: centos-6.4
    - name: archlinux-2013.05
    .kitchen.yml
    Gareth Rushgrove
    

    View Slide

60. And then write tests
    against the provisioned
    machines
    Gareth Rushgrove
    

    View Slide

61. Gareth Rushgrove
    serverspec.org
    

    View Slide

62. RSpec tests for your
    servers configured !
    by Puppet, Chef or
    anything else
    Gareth Rushgrove
    

    View Slide

63. describe package('ntp') do
    it { should be_installed }
    end
    serverspec/sample_spec.rb
    Gareth Rushgrove
    

    View Slide

64. describe service('ntp') do
    it { should be_enabled }
    it { should be_running }
    end
    serverspec/sample_spec.rb
    Gareth Rushgrove
    

    View Slide

65. kitchen test -c
    Gareth Rushgrove
    

    View Slide

66. Package "ntp"
    should be installed
    !
    Service "ntp"
    should be enabled
    should be running
    !
    Finished in 0.11191 seconds
    3 examples, 0 failures
    Finished verifying (0m7.39s).
    -----> Destroying ...
    Digital Ocean instance <1493500> destroyed.
    Finished destroying (0m4.47s).
    Finished testing (3m59.40s).
    -----> Kitchen is finished. (3m59.95s)
    Gareth Rushgrove
    

    View Slide

67. Kitchen is finished. (3m59.95s)
    Gareth Rushgrove
    

    View Slide

68. Gareth Rushgrove
    garethr/do-test-kitchen
    

    View Slide

69. Gareth Rushgrove
    

    View Slide

70. Verifying machine
    images
    (AMIs and other artefacts)
    

    View Slide

71. Your virtual machine
    images need a pipeline
    too
    Gareth Rushgrove
    

    View Slide

72. We’ll start with an example
    using AMIs on
    Gareth Rushgrove
    

    View Slide

73. Gareth Rushgrove
    packer.io
    

    View Slide

74. Packer is a tool for
    creating identical machine
    images for multiple
    platforms from a single
    source configuration
    Gareth Rushgrove
    

    View Slide

75. Configuration is lots of
    JSON
    Gareth Rushgrove
    

    View Slide

76. {
    "variables": {
    "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}"
    },
    "builders": [{
    "type": "amazon-ebs",
    "access_key": "{{user `aws_access_key`}}",
    "secret_key": "{{user `aws_secret_key`}}",
    "region": "eu-west-1",
    "source_ami": "ami-480bea3f",
    "instance_type": "t1.micro",
    "ssh_username": "ubuntu",
    "ami_name": "packer-serverspec-example-{{timestamp}}"
    }],
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "file",
    template.json
    Gareth Rushgrove
    

    View Slide

77. First build an AMI
    Gareth Rushgrove
    

    View Slide

78. Then install some
    packages and related
    configuration
    Gareth Rushgrove
    

    View Slide

79. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove
    

    View Slide

80. First we’ll install puppet
    Gareth Rushgrove
    

    View Slide

81. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove
    

    View Slide

82. #!/bin/bash
    !
    # Install language locale as without can
    # interfere with package installation
    sudo apt-get install language-pack-en -y
    !
    # Install Ruby
    sudo apt-get update
    sudo apt-get install ruby1.9.1 -y
    !
    # Install puppet/facter
    sudo gem install puppet facter --no-ri --no-rdoc
    scripts/puppet.sh
    Gareth Rushgrove
    

    View Slide

83. Then run puppet to
    configure the machine
    Gareth Rushgrove
    

    View Slide

84. !
    "provisioners": [{
    "type": "shell",
    "script": "scripts/puppet.sh"
    }, {
    "type": "puppet-masterless",
    "manifest_file": "manifests/site.pp",
    "hiera_config_path": "hiera.yaml",
    "module_paths": ["modules"]
    }
    template.json
    Gareth Rushgrove
    

    View Slide

85. Packer supports a range
    of provisioners
    Gareth Rushgrove
    

    View Slide

86. Shell, File upload, Puppet,
    Chef, Salt, Ansible
    Gareth Rushgrove
    

    View Slide

87. How to test our built-but-
    not-saved image?
    Gareth Rushgrove
    

    View Slide

88. Run serverspec tests from
    within the new image
    Gareth Rushgrove
    

    View Slide

89. Only if the tests pass do
    we save the AMI
    Gareth Rushgrove
    

    View Slide

90. Gareth Rushgrove
    garethr/packer-serverspec-example
    

    View Slide

91. Gareth Rushgrove
    

    View Slide

92. Linux containers are
    another good artefact
    example
    Gareth Rushgrove
    

    View Slide

93. Testing!
    based on changes to the
    Dockerfile works in the
    same way
    Gareth Rushgrove
    

    View Slide

94. Gareth Rushgrove
    garethr/docker-spec-example
    

    View Slide

95. Gareth Rushgrove
    

    View Slide

96. Continuous
    load testing
    (The power of aggregate assertions)
    

    View Slide

97. Load testing is often!
    ad hoc
    Gareth Rushgrove
    

    View Slide

98. Load testing is more than
    just an application concern
    Gareth Rushgrove
    

    View Slide

99. Lots of open source tools
    Gareth Rushgrove
    

    View Slide

100. JMeter, Funkload, Tsung,
     Grinder, HTTPerf, Siege,
     Gatling, Locust, Vegeta,
     Gor …
     Gareth Rushgrove
     

     View Slide

101. Nothing is perfect for our
     purposes
     Gareth Rushgrove
     

     View Slide

102. Gareth Rushgrove
     We’ll use Gatling as an
     example
     

     View Slide

103. class ExampleSimulation extends Simulation {
     val httpConf = http.baseURL("http://www.example.com")
     !
     val sample = scenario("Simple example").exec(
     http("Homepage").get("/").check(status.is(200)))
     !
     setUp(sample.inject(ramp(3 users) over (10 seconds)))
     .protocols(httpConf)
     }
     ExampleSimulation.scala
     Gareth Rushgrove
     

     View Slide

104. Aggregate assertions are
     the killer feature of Gatling
     Gareth Rushgrove
     

     View Slide

105. .assertions(
     global.responseTime.mean.lessThan(750),
     )
     ExampleSimulation.scala
     Gareth Rushgrove
     

     View Slide

106. .assertions(
     global.successfulRequests.percent.is(100),
     global.responseTime.max.lessThan(1000),
     global.responseTime.mean.lessThan(750),
     global.responseTime.percentile1.lessThan(800),
     global.requestsPerSec.greaterThan(10)
     )
     ExampleSimulation.scala
     Gareth Rushgrove
     

     View Slide

107. Continuous load testing
     supplements rather than
     replaces ad hoc load
     testing
     Gareth Rushgrove
     

     View Slide

108. Gareth Rushgrove
     garethr/gatling-demo
     

     View Slide

109. Gareth Rushgrove
     

     View Slide

110. Entire infrastructures
     as code
     (Testing out provisioning)
     

     View Slide

111. So far we’ve tested units
     of infrastructure
     Gareth Rushgrove
     

     View Slide

112. This next example tests
     an entire system
     Gareth Rushgrove
     

     View Slide

113. As an example we’ll use
     Ansible
     Gareth Rushgrove
     

     View Slide

114. Gareth Rushgrove
     ansible.com
     

     View Slide

115. Specifically we’ll use the
     DigitalOcean module
     Gareth Rushgrove
     

     View Slide

116. First we define our system
     in code
     Gareth Rushgrove
     

     View Slide

117. ---
     droplets:
     - name: production.web.1
     - name: production.web.2
     - name: production.app.1
     size: 64
     - name: production.app.2
     size: 64
     - name: production.db.1
     size: 69
     vars/droplets.yml
     Gareth Rushgrove
     

     View Slide

118. For testing we’ll use a
     combination of serverspec
     and bats
     Gareth Rushgrove
     

     View Slide

119. Gareth Rushgrove
     sstephenson/bats
     

     View Slide

120. bats is great for testing
     combinations of shell
     scripts
     Gareth Rushgrove
     

     View Slide

121. #!/usr/bin/env bats
     !
     @test "addition using bc" {
     result="$(echo 2+2 | bc)"
     [ "$result" -eq 4 ]
     }
     !
     @test "addition using dc" {
     result="$(echo 2 2+p | dc)"
     [ "$result" -eq 4 ]
     }
     example.bats
     Gareth Rushgrove
     

     View Slide

122. Run out provisioning code
     inside a bats test
     Gareth Rushgrove
     

     View Slide

123. @test "no droplets exist" {
     run tugboat droplets
     [ "$status" -eq 0 ]
     [ "${lines[0]}" = "You don't appear to have any droplets." ]
     }
     !
     @test "provisioner runs successfully" {
     run ansible-playbook -i hosts provision_digital_ocean.yml
     [ "$status" -eq 0 ]
     }
     acceptance.bats
     Gareth Rushgrove
     

     View Slide

124. Confirm we have started
     some machines
     Gareth Rushgrove
     

     View Slide

125. @test “we can count the correct number of machines" {
     run bash -c "ansible -i hosts all --list-hosts | wc -
     l | tr -d ' '"
     [ "$status" -eq 0 ]
     [ "$output" -eq 5 ]
     }
     acceptance.bats
     Gareth Rushgrove
     

     View Slide

126. Serverspec has a mode of
     operation where it runs
     against remote hosts
     Gareth Rushgrove
     

     View Slide

127. @test "per host serverspec tests pass" {
     run rake spec
     [ "$status" -eq 0 ]
     }
     acceptance.bats
     Gareth Rushgrove
     

     View Slide

128. bats acceptance.bats
     Gareth Rushgrove
     

     View Slide

129. ✓ no droplets exist
     ✓ provisioner runs successfully
     ✓ we can count the correct number of hosts
     ✓ we can list the new hosts
     ✓ all host serverspec tests pass
     ✓ clean up all droplets
     ✓ droplets have been cleaned up
     !
     7 tests, 0 failures
     Gareth Rushgrove
     

     View Slide

130. Gareth Rushgrove
     garethr/ansible-provisioner
     

     View Slide

131. Other things to
     think about
     (This is just the start)
     

     View Slide

132. Feature flags for
     infrastructure
     Gareth Rushgrove
     

     View Slide

133. Testing software defined
     networks
     Gareth Rushgrove
     

     View Slide

134. Building a complete
     infrastructure pipeline
     Gareth Rushgrove
     

     View Slide

135. Moving from continuous
     integration to continuous
     deployment
     Gareth Rushgrove
     

     View Slide

136. Questions?
     (And thanks for listening)
     

     View Slide