Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Test-Driven Infrastructure

Test-Driven Infrastructure

Talk introducing Test-Driven Infrastructure tools with a focus on Chef. Given at Bristol DevOps 18th March 2015.

Andy Gale

March 18, 2015
Tweet

More Decks by Andy Gale

Other Decks in Technology

Transcript

  1. Andy Gale @andygale on Twitter Bristol Rovers fan #UTG Bristol

    DevOps Organiser What do you do? Owner and DevOps Consultant at Hello Future
  2. Hello Future • @hellofutur3 on Twitter • DevOps, Continuous Delivery,

    Chef and cloud automation consultancy • Web application development • If you like what you see today we’re hiring Web Developer and a Junior DevOps Consultant/ System Administrator What do we do?
  3. Infrastructure • Servers • Cloud configuration (security groups, load balancers,

    Cloud DNS, IaaS etc) • Switches and other configurable hardware What do you mean by that?
  4. Topics • What is testing? • Stuff everyone can use

    today • Chef & Test-Driven Infrastructure example • Behaviour-driven testing What we’re covering today
  5. Traditional sysadmin Setting up a website… Test website in browser

    Test website in browser Setup website in text editor Fix that issue Fails Success
  6. Configuration Management Setting up a website… Test website in browser

    Test website in browser Create website with configuration management Update DNS
  7. Test-Driven Infrastructure Run test in code to prove website is

    up Setting up a website… Run website test to prove website is down Write test to see if website is up Create website with configuration management
  8. Test-Driven Infrastructure • With configuration management now you’re writing code

    and you should have tests • Tests can be used alongside monitoring to ensure your services remain functioning • Continuous Delivery of your infrastructure code Okay - what are the benefits?
  9. So how can I test? • Serverspec • Cucumber •

    Linting tools: Foodcritic, Rubocop • Behaviour-Driven: Chefspec, rspec-puppet • Integration Tests: Test Kitchen/Kitchen CI Tools you can use to help you test
  10. • Allows us to connect to existing servers (or localhost)

    and check things are there. • Things can be packages, files, processes listening on ports etc.
  11. $ serverspec-init Select OS type: 1) UN*X 2) Windows Select

    number: 1 Select a backend type: 1) SSH 2) Exec (local) Select number: 1 Vagrant instance y/n: n Input target host name: bilbo.hellofutu.re + spec/ + spec/bilbo.hellofutu.re/ + spec/bilbo.hellofutu.re/sample_spec.rb + spec/spec_helper.rb + Rakefile + .rspec Getting started
  12. require 'spec_helper' describe package('httpd'), :if => os[:family] == 'redhat' do

    it { should be_installed } end describe package('apache2'), :if => os[:family] == 'ubuntu' do it { should be_installed } end describe service('httpd'), :if => os[:family] == 'redhat' do it { should be_enabled } it { should be_running } end describe service('apache2'), :if => os[:family] == 'ubuntu' do it { should be_enabled } it { should be_running } end describe service('org.apache.httpd'), :if => os[:family] == 'darwin' do it { should be_enabled } it { should be_running } end describe port(80) do it { should be_listening } end Getting started - Apache example 1/serverspec/spec/bilbo.hellofutu.re/sample_spec.rb
  13. /Users/andy/.rvm/rubies/ruby-2.1.5/bin/ruby -I/Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec-support-3.2.2/lib:/Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec- core-3.2.2/lib /Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec-core-3.2.2/exe/rspec --pattern spec/bilbo.hellofutu.re/\*_spec.rb Package "apache2" should be

    installed (FAILED - 1) Service "apache2" should be enabled (FAILED - 2) should be running (FAILED - 3) Port "80" should be listening (FAILED - 4) Failures: 1) Package "apache2" should be installed On host `bilbo.hellofutu.re' Failure/Error: it { should be_installed } expected Package "apache2" to be installed sudo -p 'Password: ' /bin/sh -c dpkg-query\ -f\ \'\$\{Status\}\'\ -W\ apache2\ \|\ grep\ -E\ \'\^\(install\|hold\)\ ok\ installed\$\' # ./spec/bilbo.hellofutu.re/sample_spec.rb:8:in `block (2 levels) in <top (required)>' 2) Service "apache2" should be enabled On host `bilbo.hellofutu.re' Failure/Error: it { should be_enabled } expected Service "apache2" to be enabled sudo -p 'Password: ' /bin/sh -c ls\ /etc/rc3.d/\ \|\ grep\ --\ \'\^S..apache2\'\ \|\|\ grep\ \'start\ on\'\ /etc/init/apache2.conf # ./spec/bilbo.hellofutu.re/sample_spec.rb:17:in `block (2 levels) in <top (required)>' 3) Service "apache2" should be running On host `bilbo.hellofutu.re' Failure/Error: it { should be_running } expected Service "apache2" to be running sudo -p 'Password: ' /bin/sh -c ps\ aux\ \|\ grep\ -w\ --\ apache2\ \|\ grep\ -qv\ grep # ./spec/bilbo.hellofutu.re/sample_spec.rb:18:in `block (2 levels) in <top (required)>' 4) Port "80" should be listening On host `bilbo.hellofutu.re' Failure/Error: it { should be_listening } expected Port "80" to be listening sudo -p 'Password: ' /bin/sh -c netstat\ -tunl\ \|\ grep\ --\ :80\\\ # ./spec/bilbo.hellofutu.re/sample_spec.rb:27:in `block (2 levels) in <top (required)>' Finished in 0.49838 seconds (files took 2.56 seconds to load) 4 examples, 4 failures Failed examples: rspec ./spec/bilbo.hellofutu.re/sample_spec.rb:8 # Package "apache2" should be installed rspec ./spec/bilbo.hellofutu.re/sample_spec.rb:17 # Service "apache2" should be enabled rspec ./spec/bilbo.hellofutu.re/sample_spec.rb:18 # Service "apache2" should be running rspec ./spec/bilbo.hellofutu.re/sample_spec.rb:27 # Port "80" should be listening /Users/andy/.rvm/rubies/ruby-2.1.5/bin/ruby -I/Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec-support-3.2.2/lib:/Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec- core-3.2.2/lib /Users/andy/.rvm/gems/ruby-2.1.5@tdi/gems/rspec-core-3.2.2/exe/rspec --pattern spec/bilbo.hellofutu.re/\*_spec.rb failed
  14. $ rake spec /Users/andy/.rvm/rubies/ruby-2.1.5/bin/ruby -I/Users/andy/.rvm/gems/ ruby-2.1.5@tdi/gems/rspec-support-3.2.2/lib:/Users/andy/.rvm/gems/ ruby-2.1.5@tdi/gems/rspec-core-3.2.2/lib /Users/andy/.rvm/gems/ ruby-2.1.5@tdi/gems/rspec-core-3.2.2/exe/rspec --pattern

    spec/ bilbo.hellofutu.re/\*_spec.rb Package "apache2" should be installed Service "apache2" should be enabled should be running Port "80" should be listening Finished in 0.41582 seconds (files took 3.65 seconds to load) 4 examples, 0 failures Getting started
  15. Things you can test for… Getting started bond, bridge, cgroup,

    command, cron, default_gateway, file, group, host, iis_app_pool, iis_website, interface, ipfilter, ipnat, iptables, kernel_module, linux_kernel_parameter, lxc, mail_alias, package, php_config, port, ppa, process, routing_table, selinux, service, user, x509_certificate, x509_private_key, windows_feature, windows_registry_key, yumrepo, zfs http://serverspec.org/resource_types.html
  16. • Behaviour-Driven Development • Ruby ❤️ • Allows more complex

    testing • Human readable so perfect for dealing with non technical people and/or customers
  17. 1. Describe behaviour in plain text 2. Write a step

    definition in Ruby 3. Run and watch it fail 4. Write code to make the step pass 5. Run again and see the step pass 6. Repeat 2-5 until green like a cuke 7. Repeat 1-6 until the money runs out Steps for success… as the Cumber website says…
  18. Example - our feature Feature: Web redirects Scenario Outline: Given

    the http URL "<http_url>" When a request is made Then the response http_code should be "<http_code>" And I should be redirected to "<redirect_url>" Examples: | http_url | http_code | redirect_url | | http://www.scredible.com/ | 302 | https://scredible.com/ | | https://www.scredible.com/ | 302 | https://scredible.com/ | | http://scredible.com/ | 302 | https://scredible.com/ | | https://scredible.com/ | 200 | <nowhere> | 2/cucumber/features/http.feature
  19. Example - The Ruby require 'net/http' No need to write

    HTTP code in Ruby we’ll just require this
  20. Example - The Ruby Given the http URL "<http_url>" Given(/^the

    http URL "(.*?)"$/) do |http_url| @http_url = http_url end This feature…. Matches this Ruby step definition
  21. Example - The Ruby When(/^a request is made$/) do @http_code

    = 0 uri = URI.parse(@http_url) http = Net::HTTP.new(uri.host, uri.port) if uri.port == 443 http.use_ssl = true end req = Net::HTTP::Get.new(uri.request_uri) res = http.request(req) res_hash = res.to_hash if res_hash.key?('location') @redirect_url = res_hash['location'][0] else @redirect_url = '<nowhere>' end @http_code = res.code end
  22. Example - The Ruby Then the response http_code should be

    "<http_code>" Then(/^the response http_code should be "(.*?)"$/) do | expected_http_code| expect(@http_code).to eq(expected_http_code) end This part of our feature…. Matches this part of our Ruby step definition
  23. Example - The Ruby And I should be redirected to

    "<redirect_url>" And(/^I should be redirected to "(.*?)"$/) do |expected| expect(@redirect_url).to eq(expected) end This part of our feature…. Matches this part of our Ruby step definition And of course we loop through each of the examples 2/cucumber/features/step_definitions/http_features.rb
  24. Example - The output $ cucumber Feature: Web redirects Scenario

    Outline: Given the http URL "<http_url>" When a request is made Then the response http_code should be "<http_code>" And I should be redirected to "<redirect_url>" Examples: | http_url | http_code | redirect_url | | http://www.scredible.com/ | 302 | https://scredible.com/ | | https://www.scredible.com/ | 302 | https://scredible.com/ | | http://scredible.com/ | 302 | https://scredible.com/ | | https://scredible.com/ | 200 | <nowhere> | 4 scenarios (4 passed) 16 steps (16 passed) 0m2.727s
  25. • SSL testing (POODLE, Heartbleed.. the new one coming tomorrow)

    • SSL certificate expiry • DNS records and Domain expiry • …. and testing complex configurations At Hello Future we also use Cucumber tests for simple customer readable tests
  26. The Chef development pipeline we use and recommend at Hello

    Future Foodcritic Chefspec Rubocop Test Kitchen
  27. Foodcritic • Style • Correctness • Syntax • Best practices

    • Common mistakes • Deprecations Foodcritic checks your cookbooks for the following
  28. Rubocop • Consistent style • Syntax • Best practices •

    Cyclomatic complexity, etc Rubocop checks your Ruby for the following
  29. Chefspec • Test messages instead of actions • Test over

    many different types of platforms and versions extremely quickly • Super fast feedback Allows you to test the behaviour of your cookbooks
  30. Test Kitchen • Supports Chef, Puppet, Ansible etc etc •

    Supports Vagrant, EC2, LXC, Docker etc etc • Testing with Serverspec, Bats etc Test you recipe against operating systems
  31. • A cookbook to install Varnish • We’ll use all

    the tools I’ve mentioned TDI Example
  32. • Install ChefDK because life is too short to piss

    around with RVM and Gemfiles when you don’t have to
 
 https://downloads.chef.io/chef-dk/ TDI Example
  33. TDI Example $ chef exec knife cookbook create hf-varnish -o

    . Make a skeleton cookbook WARNING: No knife configuration file found ** Creating cookbook hf-varnish in /Users/andy/Work/tdi ** Creating README for cookbook: hf-varnish ** Creating CHANGELOG for cookbook: hf-varnish ** Creating metadata for cookbook: hf-varnish
  34. TDI Example $ chef exec foodcritic -f any . Let’s

    run Foodcritic The empty output from Foodcritic means everything is fine. And you would hope so as there isn’t any code to critique yet!
  35. $ chef exec rubocop Inspecting 4 files C... Offenses: metadata.rb:1:5:

    C: Put one space between the method name and the first argument. name 'hf-varnish' ^^^^^^^^^^^^^ metadata.rb:2:11: C: Put one space between the method name and the first argument. maintainer 'Hello Future' ^^^^^^^ metadata.rb:4:8: C: Put one space between the method name and the first argument. license 'All rights reserved' ^^^^^^^^^^ metadata.rb:5:12: C: Put one space between the method name and the first argument. description 'Installs/Configures hf-varnish' ^^^^^^ metadata.rb:7:8: C: Put one space between the method name and the first argument. version '0.1.0' ^^^^^^^^^^ 4 files inspected, 5 offenses detected Let’s run Rubocop
  36. 3/hf-varnish/.rubocop.yml Create .rubocop.yml in your text editor AlignParameters: Enabled: false

    Encoding: Enabled: false HashSyntax: Enabled: false LineLength: Enabled: false MethodLength: Max: 30 SingleSpaceBeforeFirstArg: Enabled: false
  37. $ chef exec rubocop Inspecting 4 files .... 4 files

    inspected, no offenses detected Let’s run Rubocop
  38. require 'chefspec' describe 'hf-varnish::default' do let(:chef_run) do ChefSpec::SoloRunner.new(platform: 'ubuntu', version:

    '14.04') end it 'should install the varnish package' do chef_run.converge(described_recipe) expect(chef_run).to install_package('varnish') end end 3/hf-varnish/spec/default_spec.rb Create spec/default_spec.rb in your text editor
  39. $ chef exec rspec --color --format progress . F Failures:

    1) hf-varnish::default should install the varnish package Failure/Error: expect(chef_run).to install_package('varnish') expected "package[varnish]" with action :install to be in Chef run. Other package resources: # ./spec/default_spec.rb:10:in `block (2 levels) in <top (required)>' Finished in 0.01204 seconds (files took 2.4 seconds to load) 1 example, 1 failure Failed examples: rspec ./spec/default_spec.rb:8 # hf-varnish::default should install the varnish package We’ll now run Chefspec and confirm our test fails
  40. require 'serverspec' set :backend, :exec describe package('varnish') do it {

    should be_installed } end 3/hf-varnish/test/integration/default/serverspec/default_spec.rb Create test/integration/default/serverspec/default_spec.rb in your text editor
  41. 3/hf-varnish/.kitchen.yml Create .kitchen.yml in your text editor --- driver: name:

    vagrant provisioner: name: chef_solo platforms: - name: ubuntu-14.04 suites: - name: default run_list: - recipe[hf-varnish::default]
  42. $ chef exec kitchen converge We’ll now run Test Kitchen

    and confirm our test fails Transferring files to <default-ubuntu-1404> Starting Chef Client, version 12.1.1 Compiling Cookbooks... Converging 0 resources Running handlers: Running handlers complete Chef Client finished, 0/0 resources updated in 6.452326869 seconds Finished converging <default-ubuntu-1404> (0m58.86s). -----> Kitchen is finished. (3m0.46s)
  43. $ chef exec kitchen verify We’ll now run Test Kitchen

    and confirm our test fails -----> serverspec installed (version 2.10.1) /opt/chef/embedded/bin/ruby -I/tmp/busser/suites/serverspec -I/tmp/busser/ gems/gems/rspec-support-3.2.2/lib:/tmp/busser/gems/gems/rspec-core-3.2.2/lib /opt/ chef/embedded/bin/rspec --pattern /tmp/busser/suites/serverspec/\*\*/\*_spec.rb -- color --format documentation --default-path /tmp/busser/suites/serverspec Package "varnish" should be installed (FAILED - 1) Failures: 1) Package "varnish" should be installed Failure/Error: it { should be_installed } expected Package "varnish" to be installed /bin/sh -c dpkg-query\ -f\ \'\$\{Status\}\'\ -W\ varnish\ \|\ grep\ - \ \'\^\(install\|hold\)\ ok\ installed\$\' dpkg-query: no packages found matching varnish # /tmp/busser/suites/serverspec/default_spec.rb:6:in `block (2 levels) in <top (required)>' Finished in 0.12083 seconds (files took 0.29726 seconds to load) 1 example, 1 failure
  44. # # Cookbook Name:: hf-varnish # Recipe:: default # package

    'varnish' do action :install end 3/hf-varnish/recipes/default.rb Create recipes/default.rb in your text editor
  45. # # Cookbook Name:: hf-varnish # Recipe:: default # package

    'varnish' Create recipes/default.rb in your text editor
  46. $ chef exec foodcritic -f any . Let’s test our

    cookbook $ chef exec rubocop Inspecting 4 files .... 4 files inspected, no offenses detected $ chef exec rspec --color --format progress . Finished in 0.01458 seconds (files took 1.92 seconds to load) 1 example, 0 failures
  47. $ chef exec kitchen test Let’s test our cookbook Recipe:

    hf-varnish::default * apt_package[varnish] action install - install version 3.0.5-2 of package varnish Running handlers: Running handlers complete Chef Client finished, 1/1 resources updated in 20.503556761 seconds serverspec installed (version 2.10.1) Package "varnish" should be installed Finished in 0.19874 seconds (files took 0.50069 seconds to load) 1 example, 0 failures Finished verifying <default-ubuntu-1404> (0m18.79s). …
  48. Test Kitchen workflow $ chef exec kitchen converge $ chef

    exec kitchen verify $ chef exec kitchen converge $ chef exec kitchen verify $ chef exec kitchen test
  49. require 'chefspec' describe 'hf-varnish::default' do %w(12.04 14.04).each do |version| let(:chef_run)

    do ChefSpec::SoloRunner.new(platform: 'ubuntu', version: version) end it 'should install the varnish package' do chef_run.converge(described_recipe) expect(chef_run).to install_package('varnish') end end end 4/hf-varnish/spec/default_spec.rb Create spec/default_spec.rb in your text editor
  50. 4/hf-varnish/.kitchen.yml Create .kitchen.yml in your text editor --- driver: name:

    vagrant provisioner: name: chef_solo platforms: - name: ubuntu-12.04 - name: ubuntu-14.04 suites: - name: default run_list: - recipe[hf-varnish::default]
  51. $ chef exec foodcritic -f any . Let’s test our

    cookbook again! $ chef exec rubocop Inspecting 4 files .... 4 files inspected, no offenses detected $ chef exec rspec --color --format progress .. Finished in 0.03363 seconds (files took 3.21 seconds to load) 2 examples, 0 failures
  52. $ chef exec kitchen test Let’s test our cookbook Recipe:

    hf-varnish::default * apt_package[varnish] action install - install version 3.0.2-1ubuntu0.1 of package varnish Package "varnish" should be installed Finished in 0.1282 seconds (files took 0.30204 seconds to load) 1 example, 0 failures …
  53. $ chef exec kitchen test Let’s test our cookbook Recipe:

    hf-varnish::default * apt_package[varnish] action install - install version 3.0.5-2 of package varnish Package "varnish" should be installed Finished in 0.10598 seconds (files took 0.28897 seconds to load) 1 example, 0 failures …
  54. if node['platform'] == 'centos' ver = node['platform_version'].to_i if ver ==

    6 rpm_url = 'https://repo.varnish-cache.org/redhat/varnish-3.0.el6.rpm' else fail "#{ver} is an unsupported version of centos" end path = File.join(Chef::Config[:file_cache_path], '/varnish-3.0.rpm') remote_file path do source rpm_url action :create_if_missing end rpm_package 'varnish-rpm' do source rpm_url action :install end end package 'varnish' do action :install end 5/hf-varnish/recipes/default.rb Create recipes/default.rb in your text editor
  55. require 'chefspec' platforms = { 'ubuntu' => %w(12.04 14.04), 'centos'

    => %w(6.6) } describe 'hf-varnish::default' do platforms.each do |platform, versions| versions.each do |version| context "on #{platform}-#{version}" do let(:chef_run) do ChefSpec::SoloRunner.new(platform: platform, version: version, file_cache_path: '/tmp') end 5/hf-varnish/spec/default_spec.rb Create spec/default_spec.rb in your text editor
  56. if platform == 'centos' it 'should download the repo rpm'

    do chef_run.converge(described_recipe) expect(chef_run).to create_remote_file_if_missing('/tmp/ varnish-3.0.rpm') end it 'should install the rpm' do chef_run.converge(described_recipe) expect(chef_run).to install_rpm_package('varnish-rpm') end end it 'should install the varnish package' do chef_run.converge(described_recipe) expect(chef_run).to install_package('varnish') end end end end end 5/hf-varnish/spec/default_spec.rb Create spec/default_spec.rb in your text editor
  57. 5/hf-varnish/.kitchen.yml Update .kitchen.yml in your text editor --- driver: name:

    vagrant provisioner: name: chef_solo platforms: - name: ubuntu-12.04 - name: ubuntu-14.04 - name: centos-6.6 suites: - name: default run_list: - recipe[hf-varnish::default]
  58. require 'serverspec' set :backend, :exec describe package('varnish-release'), :if => os[:family]

    == 'redhat' do it { should be_installed } end describe package('varnish') do it { should be_installed } end 5/hf-varnish/test/integration/default/serverspec/default_spec.rb Create test/integration/default/serverspec/default_spec.rb in your text editor
  59. $ chef exec foodcritic -f any . Let’s test our

    cookbook again! $ chef exec rubocop Inspecting 4 files .... 4 files inspected, no offenses detected $ chef exec rspec --color --format progress ..... Finished in 0.11968 seconds (files took 2.81 seconds to load) 5 examples, 0 failures
  60. $ chef exec kitchen test Let’s test our cookbook -

    update content in file /tmp/kitchen/cache/varnish-3.0.rpm from none to 32d6d5 - restore selinux security context - install version 3.0-1.el6 of package varnish-rpm - install version 3.0.6-1.el6 of package varnish Package "varnish-release" should be installed Package "varnish" should be installed …
  61. #!/bin/sh set -e chef exec foodcritic -f any . echo

    chef exec rubocop echo chef exec rspec --color --format progress echo chef exec kitchen test 5/hf-varnish/spec/default_spec.rb And now we introduce the test.sh file I like to use….
  62. Continuous Delivery of infrastructure code • Individual Git repositories for

    each cookbook • Use many small cookbooks that do one thing so are easy to test • Use wrapper cookbook to collate individual cookbooks per application • Force all cookbooks to be uploaded to Chef server by Jenkins; Jenkins is the gatekeeper This works for Hello Future but the best
 solution may be different for your business
  63. Continuous Delivery of infrastructure code Write code Push to Git

    Jenkins job fires via Git hook Failure Jenkins uploads to Chef Server Use on staging Success
  64. Continuous Delivery of infrastructure code Jenkins job to promote cookbook

    to production via Berkshelf Freeze cookbook version on Chef server Tie cookbook version in Chef environment “production” Production
  65. Chef training • Day 1 
 Getting Started with Chef

    and cookbook development • Day 2
 Chef customer resources, patterns and best practices and testing • Day 3
 Testing and setting up a Chef Continuous Delivery pipeline Hello Future will be holding two or three day Chef
 training course later in the year Talk to Andy if you’re interested
  66. Hello Future • We’re hiring for a Web Developer and

    a Junior DevOps Consultant/System Administrator Don’t forget…
  67. Hello Future • And of course if you need help

    with any of this we’re available a very reasonable consultancy rates! Don’t forget…