Slide 1
Slide 1 text
Don't Shrug!
Using Ruby Testing Frameworks
To Bring Sanity
to Your Infrastructure
FOSDEM 2013, Bryan W. Berry @bryanwb
Code for this demo: https://github.com/bryanwb/tk-demo
Slide 2
Slide 2 text
Bryan W. Berry
●
Sysadmin UN Food and Agr. Org in Rome, Italy
●
Co-host of FoodFightShow podcast along w/
Nathen Harvey
●
@bryanwb
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Or not . . .
Slide 5
Slide 5 text
Already got it
all figured out?
Slide 6
Slide 6 text
Infrastructure as
Infrastructure as
Code?
Code?
No Problem!
Slide 7
Slide 7 text
Maybe this
isn't so
complicated
But . . .
Slide 8
Slide 8 text
Big Data Deathstar
Coming for U!
Slide 9
Slide 9 text
Apache Mesos
It's about to get a lot more
complicated
Slide 10
Slide 10 text
The Data Center needs an
Operating System
Cloudera Impala
Slide 11
Slide 11 text
Two Types of Testing
White Box Black Box
code in isolation Integration Testing
Slide 12
Slide 12 text
You should use Ruby for
Integration Testing
●
Lightweight (mostly)
●
Great libraries (mostly)
●
Awesome Test DSLs like Rspec
●
Easy to mock services (Sinatra)
●
Vagrant, Vagrant, Vagrant
Note: not the current Vagrant logo
Slide 13
Slide 13 text
Prod,QA,Dev,Vagrant
Vagrant is your Development environment
QA,Dev in use by developers are almost production
●
VirtualBox
●
Openstack*
●
KVM*
* support coming
Note: not the current Vagrant logo
Slide 14
Slide 14 text
Basic Vagrantfile
Vagrant::Config.run do |config|
config.vm.box = "lucid32"
config.vm.provision :chef_solo do |chef|
# We're going to download our cookbooks from the web
chef.recipe_url = "http://files.vagrantup.com/cookbooks.tar.gz"
# Tell chef what recipe to run
chef.add_recipe("vagrant_main")
end
end
Slide 15
Slide 15 text
Multiple Distros
distros = {
:lucid32 => { :url => 'https://example.com/ubuntu-10.04-i386.box' },
:centos6_3_32 => { :url => 'https://example.com/centos-6.3-i386.box' }
}
Vagrant::Config.run do |config|
distros.each_pair do |name,options|
config.vm.define name do |dist_config|
dist_config.vm.box = name.to_s
dist_config.vm.box_url = options[:url]
# rest of vagrant config
# . . .
end
end
Slide 16
Slide 16 text
Workflow and Vagrant
●
Vagrantfile defines vms but not actions
●
Need a separate tool for this, Rake or shell
script
●
Start Apache, haproxy, and back-end web
service, then test communication between all 3
Slide 17
Slide 17 text
Example Setup
Apache Haproxy
Web
Service
Hello World!
GET / localhost
Slide 18
Slide 18 text
Multi-VM Vagrantfile
Vagrant::Config.run do |config|
config.vm.define “apache” do
# . . .
end
config.vm.define “haproxy” do
# . . .
end
config.vm.define “backend-app” do
# . . .
end
end
Slide 19
Slide 19 text
Workflow with Rake
task :up do
puts "About to run vagrant-up..."
$env = Vagrant::Environment.new
$env.vms.each do |vm|
vm.cli(“up”)
end
end
task :test => :up do
sleep 100 # wait a long time
result = $env.vms[“apache”].cli('curl localhost | grep “Hello World”')
exit result # result is 0 or 1 to indicate success or failure
end
Slide 20
Slide 20 text
Most important Vagrant Command
$ vagrant destroy -f
Slide 21
Slide 21 text
Frameworks
Basic
●
Cucumber
●
Rspec
●
Minitest
Advanced
●
Test-Kitchen
●
Chef-workflow
Slide 22
Slide 22 text
Rspec
it "should say It works!" do
response = Faraday.get “http://localhost”
response.content.should =~ /It works/
end
Focus on Behavior
Slide 23
Slide 23 text
Rspec
Already used by whitebox-testing
frameworks like
●
Chefspec
●
Rspec-Puppet
Tests expectations for how a program
should behave
Slide 24
Slide 24 text
minitest-chef-handler
●
Runs minitest suites after Chef recipes
●
Has full access to the node object, chef-server
●
Can also access underlying system
minitest supports more traditional TestCase
syntax and Rspec style expectations
Slide 25
Slide 25 text
Apache
# apache2/files/default/minitest/default_test.rb
describe_recipe “apache2::default” do
it “responds with 'It Works!'” do
response = Faraday.get “localhost”
assert_match response.content, /It works/
end
end
Slide 26
Slide 26 text
More Apache Tests
It “installs httpd package” do
package('httpd').must_be_installed
end
It “starts httpd service” do
service('httpd').must_be_running
end
Slide 27
Slide 27 text
Chef Integration
It “installs httpd package” do
package(node['apache2']['package']).must_be_installed
end
It “starts httpd service” do
service(node['apache2']['package']).must_be_running
end
Slide 28
Slide 28 text
Custom Matchers
It “installs httpd package” do
package('httpd').must_be_installed
end
It “starts httpd service” do
service('httpd').must_be_running
end
extensions to minitest
Slide 29
Slide 29 text
Vagrant + Chef + Minitest
# apache2/Vagrantfile
# ….
config.vm.box = “Centos6-i386”
chef.run_list = [
“minitest-handler”,
“apache2”
]
Slide 30
Slide 30 text
Result
●
Screenshot of minitest results
Slide 31
Slide 31 text
CI Integration
●
Minitest-chef-handler returns non-zero if any
test fails
Slide 32
Slide 32 text
Wouldn't this be awesome?
Slide 33
Slide 33 text
Test-Kitchen
●
Framework to test a single cookbook across
different operating systems
●
Can run tests on VirtualBox or Openstack
●
Supports minitest or cucumber
●
Configuration DSL
Slide 34
Slide 34 text
Usage
Test-kitchen defines base machines for you
Slide 35
Slide 35 text
No content
Slide 36
Slide 36 text
$ kitchen test
Runs each configuration in each of the defined
platforms (4)
Number of Test Runs = 4 x No. of configurations
Apache cookbook
96 test runs = 4 x 24 configurations
Slide 37
Slide 37 text
Usage
To test all VMs against all configurations
$ kitchen test
# tests all configurations on
# ubuntu-12.04
# ubuntu-10.10
# centos-5.8
# centos-6.3
Slide 38
Slide 38 text
Test-kitchen structure
Slide 39
Slide 39 text
Usage
To test a single VM against a single “configuration”
$ kitchen test –platform ubuntu-12.04 –configuration
default
Slide 40
Slide 40 text
Kitchenfile
cookbook “apache2” do
configuration "mod_php5"
configuration "mod_ssl"
exclude { :platform => 'centos',
:configuration => 'mod_auth_cas' }
run_list_extras ['apache2_test::setup']
end
Slide 41
Slide 41 text
Scaffolding
# apache2_test/attributes/setup.rb
default['apache']['auth_username'] = 'bork'
default['apache']['auth_password'] = 'secret'
default['apache']['cache_expiry_seconds'] = 60
default['apache']['app_dir'] = '/home/apache2/env'
# apache2_test/setup.rb
chef_gem “faraday” # our tests will use this
node.set[:apache][:default_site_enabled] = true
Slide 42
Slide 42 text
Our Test
# apache2/files/default/minitest/default_test.rb
require 'faraday'
describe_recipe “apache2::default” do
it “responds with 'Hello World!'” do
response = Faraday.get “localhost”
assert_match response.content, /Hello World/
end
end
Slide 43
Slide 43 text
Disadvantages
●
Designed for single-node configurations
●
Doesn't expose underlying Vagrant config
●
Test runs are sequential, no support for
concurrent test runs
Slide 44
Slide 44 text
Jamie-CI
Test-Kitchen 1.0
●
Written by Fletcher Nichol
●
Only does two things
1. Defines machines
2. Exposes Driver interface for different virtualization
backends
●
Drivers* exist for:
●
Linux containers
●
Vagrant
●
EC2
*Some still need to be renamed
Slide 45
Slide 45 text
Test-Kitchen 1.0
●
Define VMs using a single yaml file
●
Supports Librarian-Chef and Berkshelf
●
Tests VMs concurrently!
●
No Workflow! Not opinionated!
●
No docs either!
Slide 46
Slide 46 text
No Dependency on Chef
Slide 47
Slide 47 text
No content
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
.kitchen.yml
●
Why YAML?
●
Expressive
●
Easy for other programming languages to parse
●
Platforms – base VMs with a minimum config
●
Suite – actual configuration you want tested
●
Number of test runs = Platforms x Suites
Note: There is debate whether there will also be code-based configuration
for Test-Kitchen 1.0. Stay tuned . . .
Slide 51
Slide 51 text
.kitchen.local.ec2.yml
Slide 52
Slide 52 text
No content
Slide 53
Slide 53 text
Kitchen 1.0 commands
●
Destroy
●
Create
●
Converge
●
Test!
$ bundle exec kitchen test
OR
$ bundle exec kitchen test mod-proxy-centos-63
$ bundle exec kitchen test #{suite}-#{platform}
Slide 54
Slide 54 text
BATS!!!!
Slide 55
Slide 55 text
Bash Automated Testing System
Slide 56
Slide 56 text
Bash Automated Testing System
Slide 57
Slide 57 text
Issues for “Smoke” tests
●
Don't need to test multiple OS versions
●
How to DRY up common driver configurations?
●
Where to put IP Configuration?
Slide 58
Slide 58 text
No content
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
Directory
Layout
One log file
per VM
“Smoke” Test
Slide 61
Slide 61 text
tk-demo
Demo project for test-kitchen, to demonstrate
how to use it for mulit-vm integration tests
https://github.com/bryanwb/tk-demo
Slide 62
Slide 62 text
test-kitchen + Rake
Slide 63
Slide 63 text
No content
Slide 64
Slide 64 text
Wiring things together is hard
Apache Haproxy
App
Server
Hello World!
GET / localhost
Slide 65
Slide 65 text
Chef-Workflow
Apache Haproxy
App
Server
Chef-Server
Chef-client Chef-client Chef-client
A set of helpers for managing workflow
Slide 66
Slide 66 text
Chef-Workflow
●
Written by Erik Hollensbe
●
3 libraries
●
Chef-workflow
●
chef-workflow-tasklib
●
chef-workflow-testlib
●
Deserves more attention than it is given here
Slide 67
Slide 67 text
Chef-workflow
Task Example
Slide 68
Slide 68 text
Chef-workflow
Test Example
Slide 69
Slide 69 text
We need a layered
approach to
systems
management
Slide 70
Slide 70 text
Image credit: http://i.technet.microsoft.com/dynimg/IC197700.gif
Slide 71
Slide 71 text
No content
Slide 72
Slide 72 text
BOSS* Bryan's Orchestration is
Super-Simple DSL
*Just a mock-up
Slide 73
Slide 73 text
even simpler
Slide 74
Slide 74 text
Awesome Ruby Libraries
for use in integration tests
●
Faraday
●
Sinatra
●
Rspec-dns
●
Ruby-dns
Slide 75
Slide 75 text
No content
Slide 76
Slide 76 text
Integration Testing
Heroes
Slide 77
Slide 77 text
In Summary
●
You can do basic integration testing right now
with vagrant + rake
●
Concurrent VM task execution important
●
Need DSL for integration tests
●
Test-Kitchen is still alpha but very exciting
●
There is gold in Chef-Workflow
●
We need Public CI for chef cookbooks, puppet
modules, and cfengine bundles
Slide 78
Slide 78 text
References
●
minitest-chef-handler
●
chef-workflow
●
RSpec
●
Test-Kitchen
●
Test-Kitchen 1.0 (formerly known as Jamie-CI)
●
Vagrant
●
berkshelf
●
Apache Mesos
●
Cloudera Impala
Copyright Bryan W. Berry, Creative Commons 3.0, Unported
Slide 79
Slide 79 text
References
●
kitchen-vagrant
●
Celluloid
●
BATS
●
tk-demo, the code used in this presentation
Copyright Bryan W. Berry, Creative Commons 3.0, Unported
Slide 80
Slide 80 text
Special Thanks To
●
Fletcher Nichol
●
Jamie Winsor
●
Nathen Harvey
●
Joshua Timberman
●
Erik Hollensbe
for their assistance w/ this presentation