Airconf14 TDD Infra

slides - http://dvps.me/airconf14-tdd-infra
video - http://dvps.me/airconf14-tdd-infra-youtube

Programmers have been test driving software design for a while, but the infrastructure that runs that software is still often manually configured and maintained.

Now that every second company is using Puppet or Chef, the infrastructure is finally automated by using software, and the natural next step is test driving this software as well.

Learn the reasons why tests for infrastructure code and tests in general are a good idea, and the various ways of writing tests for your infrastructure automation.

Transcript

1. Test Driven Infrastructure why and how
2. link to slides dvps.me/airconf14-tdd-infra link to video dvps.me/airconf14-tdd-infra-youtube Programmers have been test driving software design for a while, but the infrastructure that runs that software is still often manually configured and maintained. Now that every second company is using Puppet or Chef, the infrastructure is finally automated by using software, and the natural next step is test driving this software as well. Learn the reasons why tests for infrastructure code and tests in general are a good idea, and the various ways of writing tests for your infrastructure automation.
3. what is this about? ◦ terminology ◦ infrastructure as code ◦ testing infrastructure
4. Software & IT devs & ops
5. physical servers • 4w to order • 1w to install operating system • 2d to re-install manually / 2h automated virtual servers • 2d to re-install manually / 2h automated docker • 1s to re-install
6. it-ops & developer pain • repeat boring work ◦ debug ◦ document • deal with people • exit comfort zone
7. KEEP CALM AND AUTOMATE EVERYTHING
8. dev/qa/business ask: How do we know if a system is ready? ops-IT usual answer: “Check the logs” silo of IT
9. silo of Dev developers do their own IT adopt chef, puppet, ansible, … new tools/skills => broken code => broken infrastructure => broken operation => $$$
12. Infrastructure as code
13. What is Infrastructure? ◦ Configuration ◦ Services ◦ OS tuning ◦ Patches ◦ Cloud Server Provision
14. This is infrastructure …
15. Infrastructure as Code “In today’s computer industry, we still typically install and maintain computers the way the automotive industry built cars in the early 1900s. An individual craftsman manually manipulates a machine into being, and manually maintains it afterwards. The automotive industry discovered first mass production, then mass customisation using standard tooling. The systems administration industry has a long way to go, but is getting there.” — infrastructures.org / 1997
16. This can be coded …
17. This too can be coded ... awsofa.info
19. Infrastructure Automation (code) infoq.com/presentations/Infrastructure-as-Code Gareth Rushgrove, June 2012 #!/bin/sh NODES="webserver database" for n in $NODES do ssh $n complex-script.sh done
20. zeroturnaround.com/labs/rebel-labs-release-it-ops-devops-productivity-report-2013
21. ◦ open source ◦ configures servers ◦ cookbooks & recipes ◦ ruby dsl Chef (example)
22. redis cookbook cookbooks/redis/recipes/default.rb package "redis-server" do action :upgrade end service "redis-server" do action :nothing supports status: true, restart: true end template "/etc/redis/redis.conf" do source "redis.conf.erb" owner "root" mode "0644" variables({ bind: node[:redis][:bind] }) notifies :restart, "service[redis-server]" end 1. cookbook/redis/attributes/default.rb 2. solo.json
23. $ sudo chef-solo -c solo.rb -j solo.json Starting Chef Client, version 11.4.0 Compiling Cookbooks... Converging 3 resources Recipe: redis::default * package[redis-server] action upgrade (up to date) * service[redis-server] action nothing (up to date) * template[/etc/redis/redis.conf] action create - update template[/etc/redis/redis.conf] from 81b4f1 to 8a6cec --- /etc/redis/redis.conf 2011-07-27 17:26:50.000000000 +0000 +++ /tmp/chef-rendered-template20130406-2537-f8vlv6 2013-04-06 ... @@ -27,7 +27,7 @@ # If you want you can bind a single interface, if the bind option is not # specified all the interfaces will listen for incoming connections. # -bind 127.0.0.1 +bind 0.0.0.0 # Specify the path for the unix socket that will be used to listen for # incoming connections. There is no default, so Redis will not listen * service[redis-server] action restart - restart service service[redis-server] Chef Client finished, 2 resources updated install package run service configuration + restart
24. testing infrastructure code
25. Infrastructure Test #!/usr/bin/env bats @test "git binary is found in PATH" { run which git [ "$status" -eq 0 ] }
26. BASH Automated Testing System github.com/sstephenson/bats/ August 13, 2014 - 0.4.0 “Lots” of improvement December 30, 2011 - 0.1.0 Initial public release $ bats addition.bats ✓ addition using bc ✓ addition using dc 2 tests, 0 failures #!/usr/bin/env bats @test "addition using bc" { result="$(echo 2+2 | bc)" [ "$result" -eq 4 ] } @test "addition using dc" { result="$(echo 2 2+p | dc)" [ "$result" -eq 4 ] }
27. bugs a good reason to test • package didn’t actually install • service didn’t actually start • firewall doesn’t actually restrict a port • security patch didn’t apply • security constraint • removing “add something” ◦ doesn’t mean “something removed”
28. lint best-practice coding standards ◦ code linting ◦ rubocop - ruby lint ◦ flake8 - python lint ◦ foodcritic - chef ◦ puppet-lint ◦ ansible-lint custom extensions written by your team or community github.com/customink-webops/foodcritic-rules
29. puppet-lint in action $ puppet-lint /etc/puppet/modules foo/manifests/bar.pp - ERROR: trailing whitespace found on line 1 apache/manifests/server.pp - WARNING: variable not enclosed in {} on line 56 ... $ puppet-lint --fix /etc/puppet/modules foo/manifests/bar.pp - FIXED: trailing whitespace found on line 1 apache/manifests/server.pp - FIXED: variable not enclosed in {} on line 56 ...
30. unit testing controlled environment single-unit test ◦ minitest (for chef) ◦ rspec ◦ serverspec ◦ chefspec ◦ rspec-puppet
31. # Test Cases class TestNginx < MiniTest::Chef::TestCase def test_config_file_exist assert File.exist?('/etc/nginx.conf') end def test_succeed assert run_status.success? end end minitest chef handler # Spec Cases describe_recipe 'nginx:configuration' do it 'configures nginx version to 1.4.4' do node[:nginx][:version].should == '1.4.4' end end describe_recipe 'nginx:package' do it 'installs version 1.4.4' do result = assert_sh("nginx -V") assert_includes result, "nginx/1.4.4" end end
32. rspec in action # chefspec it 'installs apache2' do expect(chef_run).to install_package 'apache' end # rspec-puppet it { should contain_package('httpd').with_ensure('installed') }
33. regression testing test a fixed bug so it never returns policy not a tool
34. ◦ kitchen.ci ◦ multi-os ◦ multi-distro ◦ multi-params multivariate testing of “the world” integration testing
35. $ kitchen test -----> Starting Kitchen (v1.1.1) -----> Cleaning up any prior instances of <default-centos-64> -----> Destroying <default-centos-64>... Finished destroying <default-centos-64> (0m0.00s). -----> Testing <default-centos-64> -----> Creating <default-centos-64>... Bringing machine 'default' up with 'virtualbox' provider... [default] Importing base box 'opscode-centos-6.4'... ... Finished converging <default-centos-64> (0m50.11s). -----> Setting up <default-centos-64>... Finished setting up <default-centos-64> (0m0.00s). -----> Verifying <default-centos-64>... Finished verifying <default-centos-64> (0m0.00s). -----> Destroying <default-centos-64>... [default] Forcing shutdown of VM... [default] Destroying VM and associated drives... Vagrant instance <default-centos-64> destroyed. Finished destroying <default-centos-64> (0m7.59s). Finished testing <default-centos-64> (1m43.98s). -----> Kitchen is finished. (1m44.27s) kitchen.ci
36. Chef Testing Book ◦ cucumber chef cucumber-chef.org ◦ minitest handler github.com/btm/minitest-handler-cookbook ◦ test kitchen kitchen.ci 2011
37. Feature: Technical team members can log into team server So that we are more efficient with our time As a technical team We can connect to a shared server to collaborate on client work Scenario: Users can connect to server via ssh key Given a newly bootstrapped server When the technical users recipe is applied Then a user should be able to ssh to the server Scenario: Default shell is bash Given a newly bootstrapped server When the technical users recipe is applied And a user connects to the server via ssh Then their login shell should be "bash" 2 scenarios (2 undefined) 12 steps (7 undefined) 0m0.013s Given /^a newly bootstrapped server$/ do ... When /^the technical users recipe is applied$/ do ... Then /^a user should be able to ssh to the server$/ do When /^a user connects to the server via ssh$/ do ... Then /^their login shell should be "([^"]*)"$/ do |arg1| Given /^with a user's default shell changed to "([^"]*)"$/ do |arg1| ... When /^the user connects to the server via ssh$/ do ... Cucumber blogs.oracle.com/martin/entry/behavior_driven_infrastructure Nov 05, 2009 Martin Englund Scenario: Users can connect to server via ssh key Given a newly bootstrapped server When the technical users recipe is applied Then a user should be able to ssh to the server
38. # Strainerfile knife test: knife cookbook test $COOKBOOK foodcritic: foodcritic -I foodcritic/* cookbooks/$COOKBOOK chefspec: spec kitchen: kitchen converge default-ubuntu-1204 # all together now $ strain my_cookbook strainer - doing it all together
39. Example Cookbook full of tests github.com/opscode-cookbooks/java • kitchen.ci • bats • chefspec
40. continuous integration
41. Continuous Integration / CD ◦ 1997 Extreme Programming extremeprogramming.org/rules/integrateoften.html ◦ 2000 Martin Fowler martinfowler.com/articles/continuousIntegration.html ◦ 2007 Paul Duvall (CI book) ◦ 2010 Jez Humble (CD book)
42. Continuous Integration Tools Jenkins CI / Bamboo / TeamCity CI-aaS - Travis CI / Circle CI / … magnum-ci.com / semaphoreapp.com / codeship.io / drone.io / solanolabs.com / shiningpanda-ci.com hosted-ci.com / fazend.com / appharbor.com / cloudbees.com /clinkerhq.com
43. Stoplight
44. airpair.me/evgenyz
45. link to slides dvps.me/airconf14-test-driven-infrastructure
46. Thank you! We invite you to join Operations Israel Facebook group on on.fb.me/Ops-IL www.devops.co.il