Don't Shrug! Integration Testing for Infrastructure with Ruby Frameworks

code for talk is located at https://github.com/bryanwb/tk-demo

this presentation focuses heavily on test-kitchen

Transcript

1. Don't Shrug! Using Ruby Testing Frameworks To Bring Sanity to Your Infrastructure FOSDEM 2013, Bryan W. Berry @bryanwb Code for this demo: https://github.com/bryanwb/tk-demo
2. Bryan W. Berry • Sysadmin UN Food and Agr. Org in Rome, Italy • Co-host of FoodFightShow podcast along w/ Nathen Harvey • @bryanwb
4. Or not . . .
5. Already got it all figured out?
6. Infrastructure as Infrastructure as Code? Code? No Problem!
7. Maybe this isn't so complicated But . . .
8. Big Data Deathstar Coming for U!
9. Apache Mesos It's about to get a lot more complicated
10. The Data Center needs an Operating System Cloudera Impala
11. Two Types of Testing White Box Black Box code in isolation Integration Testing
12. You should use Ruby for Integration Testing • Lightweight (mostly) • Great libraries (mostly) • Awesome Test DSLs like Rspec • Easy to mock services (Sinatra) • Vagrant, Vagrant, Vagrant Note: not the current Vagrant logo
13. Prod,QA,Dev,Vagrant Vagrant is your Development environment QA,Dev in use by developers are almost production • VirtualBox • Openstack* • KVM* * support coming Note: not the current Vagrant logo
14. Basic Vagrantfile Vagrant::Config.run do |config| config.vm.box = "lucid32" config.vm.provision :chef_solo do |chef| # We're going to download our cookbooks from the web chef.recipe_url = "http://files.vagrantup.com/cookbooks.tar.gz" # Tell chef what recipe to run chef.add_recipe("vagrant_main") end end
15. Multiple Distros distros = { :lucid32 => { :url => 'https://example.com/ubuntu-10.04-i386.box' }, :centos6_3_32 => { :url => 'https://example.com/centos-6.3-i386.box' } } Vagrant::Config.run do |config| distros.each_pair do |name,options| config.vm.define name do |dist_config| dist_config.vm.box = name.to_s dist_config.vm.box_url = options[:url] # rest of vagrant config # . . . end end
16. Workflow and Vagrant • Vagrantfile defines vms but not actions • Need a separate tool for this, Rake or shell script • Start Apache, haproxy, and back-end web service, then test communication between all 3
17. Example Setup Apache Haproxy Web Service Hello World! GET / localhost
18. Multi-VM Vagrantfile Vagrant::Config.run do |config| config.vm.define “apache” do # . . . end config.vm.define “haproxy” do # . . . end config.vm.define “backend-app” do # . . . end end
19. Workflow with Rake task :up do puts "About to run vagrant-up..." $env = Vagrant::Environment.new $env.vms.each do |vm| vm.cli(“up”) end end task :test => :up do sleep 100 # wait a long time result = $env.vms[“apache”].cli('curl localhost | grep “Hello World”') exit result # result is 0 or 1 to indicate success or failure end
20. Most important Vagrant Command $ vagrant destroy -f
21. Frameworks Basic • Cucumber • Rspec • Minitest Advanced • Test-Kitchen • Chef-workflow
22. Rspec it "should say It works!" do response = Faraday.get “http://localhost” response.content.should =~ /It works/ end Focus on Behavior
23. Rspec Already used by whitebox-testing frameworks like • Chefspec • Rspec-Puppet Tests expectations for how a program should behave
24. minitest-chef-handler • Runs minitest suites after Chef recipes • Has full access to the node object, chef-server • Can also access underlying system minitest supports more traditional TestCase syntax and Rspec style expectations
25. Apache # apache2/files/default/minitest/default_test.rb describe_recipe “apache2::default” do it “responds with 'It Works!'” do response = Faraday.get “localhost” assert_match response.content, /It works/ end end
26. More Apache Tests It “installs httpd package” do package('httpd').must_be_installed end It “starts httpd service” do service('httpd').must_be_running end
27. Chef Integration It “installs httpd package” do package(node['apache2']['package']).must_be_installed end It “starts httpd service” do service(node['apache2']['package']).must_be_running end
28. Custom Matchers It “installs httpd package” do package('httpd').must_be_installed end It “starts httpd service” do service('httpd').must_be_running end extensions to minitest
29. Vagrant + Chef + Minitest # apache2/Vagrantfile # …. config.vm.box = “Centos6-i386” chef.run_list = [ “minitest-handler”, “apache2” ]
30. Result • Screenshot of minitest results
31. CI Integration • Minitest-chef-handler returns non-zero if any test fails
32. Wouldn't this be awesome?
33. Test-Kitchen • Framework to test a single cookbook across different operating systems • Can run tests on VirtualBox or Openstack • Supports minitest or cucumber • Configuration DSL
34. Usage Test-kitchen defines base machines for you
36. $ kitchen test Runs each configuration in each of the defined platforms (4) Number of Test Runs = 4 x No. of configurations Apache cookbook 96 test runs = 4 x 24 configurations
37. Usage To test all VMs against all configurations $ kitchen test # tests all configurations on # ubuntu-12.04 # ubuntu-10.10 # centos-5.8 # centos-6.3
38. Test-kitchen structure
39. Usage To test a single VM against a single “configuration” $ kitchen test –platform ubuntu-12.04 –configuration default
40. Kitchenfile cookbook “apache2” do configuration "mod_php5" configuration "mod_ssl" exclude { :platform => 'centos', :configuration => 'mod_auth_cas' } run_list_extras ['apache2_test::setup'] end
41. Scaffolding # apache2_test/attributes/setup.rb default['apache']['auth_username'] = 'bork' default['apache']['auth_password'] = 'secret' default['apache']['cache_expiry_seconds'] = 60 default['apache']['app_dir'] = '/home/apache2/env' # apache2_test/setup.rb chef_gem “faraday” # our tests will use this node.set[:apache][:default_site_enabled] = true
42. Our Test # apache2/files/default/minitest/default_test.rb require 'faraday' describe_recipe “apache2::default” do it “responds with 'Hello World!'” do response = Faraday.get “localhost” assert_match response.content, /Hello World/ end end
43. Disadvantages • Designed for single-node configurations • Doesn't expose underlying Vagrant config • Test runs are sequential, no support for concurrent test runs
44. Jamie-CI Test-Kitchen 1.0 • Written by Fletcher Nichol • Only does two things 1. Defines machines 2. Exposes Driver interface for different virtualization backends • Drivers* exist for: • Linux containers • Vagrant • EC2 *Some still need to be renamed
45. Test-Kitchen 1.0 • Define VMs using a single yaml file • Supports Librarian-Chef and Berkshelf • Tests VMs concurrently! • No Workflow! Not opinionated! • No docs either!
46. No Dependency on Chef
50. .kitchen.yml • Why YAML? • Expressive • Easy for other programming languages to parse • Platforms – base VMs with a minimum config • Suite – actual configuration you want tested • Number of test runs = Platforms x Suites Note: There is debate whether there will also be code-based configuration for Test-Kitchen 1.0. Stay tuned . . .
51. .kitchen.local.ec2.yml
53. Kitchen 1.0 commands • Destroy • Create • Converge • Test! $ bundle exec kitchen test OR $ bundle exec kitchen test mod-proxy-centos-63 $ bundle exec kitchen test #{suite}-#{platform}
54. BATS!!!!
55. Bash Automated Testing System
56. Bash Automated Testing System
57. Issues for “Smoke” tests • Don't need to test multiple OS versions • How to DRY up common driver configurations? • Where to put IP Configuration?
60. Directory Layout One log file per VM “Smoke” Test
61. tk-demo Demo project for test-kitchen, to demonstrate how to use it for mulit-vm integration tests https://github.com/bryanwb/tk-demo
62. test-kitchen + Rake
64. Wiring things together is hard Apache Haproxy App Server Hello World! GET / localhost
65. Chef-Workflow Apache Haproxy App Server Chef-Server Chef-client Chef-client Chef-client A set of helpers for managing workflow
66. Chef-Workflow • Written by Erik Hollensbe • 3 libraries • Chef-workflow • chef-workflow-tasklib • chef-workflow-testlib • Deserves more attention than it is given here
67. Chef-workflow Task Example
68. Chef-workflow Test Example
69. We need a layered approach to systems management
70. Image credit: http://i.technet.microsoft.com/dynimg/IC197700.gif
72. BOSS* Bryan's Orchestration is Super-Simple DSL *Just a mock-up
73. even simpler
74. Awesome Ruby Libraries for use in integration tests • Faraday • Sinatra • Rspec-dns • Ruby-dns
76. Integration Testing Heroes
77. In Summary • You can do basic integration testing right now with vagrant + rake • Concurrent VM task execution important • Need DSL for integration tests • Test-Kitchen is still alpha but very exciting • There is gold in Chef-Workflow • We need Public CI for chef cookbooks, puppet modules, and cfengine bundles
78. References • minitest-chef-handler • chef-workflow • RSpec • Test-Kitchen • Test-Kitchen 1.0 (formerly known as Jamie-CI) • Vagrant • berkshelf • Apache Mesos • Cloudera Impala Copyright Bryan W. Berry, Creative Commons 3.0, Unported
79. References • kitchen-vagrant • Celluloid • BATS • tk-demo, the code used in this presentation Copyright Bryan W. Berry, Creative Commons 3.0, Unported
80. Special Thanks To • Fletcher Nichol • Jamie Winsor • Nathen Harvey • Joshua Timberman • Erik Hollensbe for their assistance w/ this presentation