Don't Shrug! Integration Testing for Infrastructure with Ruby Frameworks

code for talk is located at https://github.com/bryanwb/tk-demo

this presentation focuses heavily on test-kitchen

Transcript

1. Don't Shrug!
   Using Ruby Testing Frameworks
   To Bring Sanity
   to Your Infrastructure
   FOSDEM 2013, Bryan W. Berry @bryanwb
   Code for this demo: https://github.com/bryanwb/tk-demo
   

   View Slide

2. Bryan W. Berry
   ●
   Sysadmin UN Food and Agr. Org in Rome, Italy
   ●
   Co-host of FoodFightShow podcast along w/
   Nathen Harvey
   ●
   @bryanwb
   

   View Slide

3. View Slide

4. Or not . . .
   

   View Slide

5. Already got it
   all figured out?
   

   View Slide

6. Infrastructure as
   Infrastructure as
   Code?
   Code?
   No Problem!
   

   View Slide

7. Maybe this
   isn't so
   complicated
   But . . .
   

   View Slide

8. Big Data Deathstar
   Coming for U!
   

   View Slide

9. Apache Mesos
   It's about to get a lot more
   complicated
   

   View Slide

10. The Data Center needs an
    Operating System
    Cloudera Impala
    

    View Slide

11. Two Types of Testing
    White Box Black Box
    code in isolation Integration Testing
    

    View Slide

12. You should use Ruby for
    Integration Testing
    ●
    Lightweight (mostly)
    ●
    Great libraries (mostly)
    ●
    Awesome Test DSLs like Rspec
    ●
    Easy to mock services (Sinatra)
    ●
    Vagrant, Vagrant, Vagrant
    Note: not the current Vagrant logo
    

    View Slide

13. Prod,QA,Dev,Vagrant
    Vagrant is your Development environment
    QA,Dev in use by developers are almost production
    ●
    VirtualBox
    ●
    Openstack*
    ●
    KVM*
    * support coming
    Note: not the current Vagrant logo
    

    View Slide

14. Basic Vagrantfile
    Vagrant::Config.run do |config|
    config.vm.box = "lucid32"
    config.vm.provision :chef_solo do |chef|
    # We're going to download our cookbooks from the web
    chef.recipe_url = "http://files.vagrantup.com/cookbooks.tar.gz"
    # Tell chef what recipe to run
    chef.add_recipe("vagrant_main")
    end
    end
    

    View Slide

15. Multiple Distros
    distros = {
    :lucid32 => { :url => 'https://example.com/ubuntu-10.04-i386.box' },
    :centos6_3_32 => { :url => 'https://example.com/centos-6.3-i386.box' }
    }
    Vagrant::Config.run do |config|
    distros.each_pair do |name,options|
    config.vm.define name do |dist_config|
    dist_config.vm.box = name.to_s
    dist_config.vm.box_url = options[:url]
    # rest of vagrant config
    # . . .
    end
    end
    

    View Slide

16. Workflow and Vagrant
    ●
    Vagrantfile defines vms but not actions
    ●
    Need a separate tool for this, Rake or shell
    script
    ●
    Start Apache, haproxy, and back-end web
    service, then test communication between all 3
    

    View Slide

17. Example Setup
    Apache Haproxy
    Web
    Service
    Hello World!
    GET / localhost
    

    View Slide

18. Multi-VM Vagrantfile
    Vagrant::Config.run do |config|
    config.vm.define “apache” do
    # . . .
    end
    config.vm.define “haproxy” do
    # . . .
    end
    config.vm.define “backend-app” do
    # . . .
    end
    end
    

    View Slide

19. Workflow with Rake
    task :up do
    puts "About to run vagrant-up..."
    $env = Vagrant::Environment.new
    $env.vms.each do |vm|
    vm.cli(“up”)
    end
    end
    task :test => :up do
    sleep 100 # wait a long time
    result = $env.vms[“apache”].cli('curl localhost | grep “Hello World”')
    exit result # result is 0 or 1 to indicate success or failure
    end
    

    View Slide

20. Most important Vagrant Command
    $ vagrant destroy -f
    

    View Slide

21. Frameworks
    Basic
    ●
    Cucumber
    ●
    Rspec
    ●
    Minitest
    Advanced
    ●
    Test-Kitchen
    ●
    Chef-workflow
    

    View Slide

22. Rspec
    it "should say It works!" do
    response = Faraday.get “http://localhost”
    response.content.should =~ /It works/
    end
    Focus on Behavior
    

    View Slide

23. Rspec
    Already used by whitebox-testing
    frameworks like
    ●
    Chefspec
    ●
    Rspec-Puppet
    Tests expectations for how a program
    should behave
    

    View Slide

24. minitest-chef-handler
    ●
    Runs minitest suites after Chef recipes
    ●
    Has full access to the node object, chef-server
    ●
    Can also access underlying system
    minitest supports more traditional TestCase
    syntax and Rspec style expectations
    

    View Slide

25. Apache
    # apache2/files/default/minitest/default_test.rb
    describe_recipe “apache2::default” do
    it “responds with 'It Works!'” do
    response = Faraday.get “localhost”
    assert_match response.content, /It works/
    end
    end
    

    View Slide

26. More Apache Tests
    It “installs httpd package” do
    package('httpd').must_be_installed
    end
    It “starts httpd service” do
    service('httpd').must_be_running
    end
    

    View Slide

27. Chef Integration
    It “installs httpd package” do
    package(node['apache2']['package']).must_be_installed
    end
    It “starts httpd service” do
    service(node['apache2']['package']).must_be_running
    end
    

    View Slide

28. Custom Matchers
    It “installs httpd package” do
    package('httpd').must_be_installed
    end
    It “starts httpd service” do
    service('httpd').must_be_running
    end
    extensions to minitest
    

    View Slide

29. Vagrant + Chef + Minitest
    # apache2/Vagrantfile
    # ….
    config.vm.box = “Centos6-i386”
    chef.run_list = [
    “minitest-handler”,
    “apache2”
    ]
    

    View Slide

30. Result
    ●
    Screenshot of minitest results
    

    View Slide

31. CI Integration
    ●
    Minitest-chef-handler returns non-zero if any
    test fails
    

    View Slide

32. Wouldn't this be awesome?
    

    View Slide

33. Test-Kitchen
    ●
    Framework to test a single cookbook across
    different operating systems
    ●
    Can run tests on VirtualBox or Openstack
    ●
    Supports minitest or cucumber
    ●
    Configuration DSL
    

    View Slide

34. Usage
    Test-kitchen defines base machines for you
    

    View Slide

35. View Slide

36. $ kitchen test
    Runs each configuration in each of the defined
    platforms (4)
    Number of Test Runs = 4 x No. of configurations
    Apache cookbook
    96 test runs = 4 x 24 configurations
    

    View Slide

37. Usage
    To test all VMs against all configurations
    $ kitchen test
    # tests all configurations on
    # ubuntu-12.04
    # ubuntu-10.10
    # centos-5.8
    # centos-6.3
    

    View Slide

38. Test-kitchen structure
    

    View Slide

39. Usage
    To test a single VM against a single “configuration”
    $ kitchen test –platform ubuntu-12.04 –configuration
    default
    

    View Slide

40. Kitchenfile
    cookbook “apache2” do
    configuration "mod_php5"
    configuration "mod_ssl"
    exclude { :platform => 'centos',
    :configuration => 'mod_auth_cas' }
    run_list_extras ['apache2_test::setup']
    end
    

    View Slide

41. Scaffolding
    # apache2_test/attributes/setup.rb
    default['apache']['auth_username'] = 'bork'
    default['apache']['auth_password'] = 'secret'
    default['apache']['cache_expiry_seconds'] = 60
    default['apache']['app_dir'] = '/home/apache2/env'
    # apache2_test/setup.rb
    chef_gem “faraday” # our tests will use this
    node.set[:apache][:default_site_enabled] = true
    

    View Slide

42. Our Test
    # apache2/files/default/minitest/default_test.rb
    require 'faraday'
    describe_recipe “apache2::default” do
    it “responds with 'Hello World!'” do
    response = Faraday.get “localhost”
    assert_match response.content, /Hello World/
    end
    end
    

    View Slide

43. Disadvantages
    ●
    Designed for single-node configurations
    ●
    Doesn't expose underlying Vagrant config
    ●
    Test runs are sequential, no support for
    concurrent test runs
    

    View Slide

44. Jamie-CI
    Test-Kitchen 1.0
    ●
    Written by Fletcher Nichol
    ●
    Only does two things
    1. Defines machines
    2. Exposes Driver interface for different virtualization
    backends
    ●
    Drivers* exist for:
    ●
    Linux containers
    ●
    Vagrant
    ●
    EC2
    *Some still need to be renamed
    

    View Slide

45. Test-Kitchen 1.0
    ●
    Define VMs using a single yaml file
    ●
    Supports Librarian-Chef and Berkshelf
    ●
    Tests VMs concurrently!
    ●
    No Workflow! Not opinionated!
    ●
    No docs either!
    

    View Slide

46. No Dependency on Chef
    

    View Slide

47. View Slide

48. View Slide

49. View Slide

50. .kitchen.yml
    ●
    Why YAML?
    ●
    Expressive
    ●
    Easy for other programming languages to parse
    ●
    Platforms – base VMs with a minimum config
    ●
    Suite – actual configuration you want tested
    ●
    Number of test runs = Platforms x Suites
    Note: There is debate whether there will also be code-based configuration
    for Test-Kitchen 1.0. Stay tuned . . .
    

    View Slide

51. .kitchen.local.ec2.yml
    

    View Slide

52. View Slide

53. Kitchen 1.0 commands
    ●
    Destroy
    ●
    Create
    ●
    Converge
    ●
    Test!
    $ bundle exec kitchen test
    OR
    $ bundle exec kitchen test mod-proxy-centos-63
    $ bundle exec kitchen test #{suite}-#{platform}
    

    View Slide

54. BATS!!!!
    

    View Slide

55. Bash Automated Testing System
    

    View Slide

56. Bash Automated Testing System
    

    View Slide

57. Issues for “Smoke” tests
    ●
    Don't need to test multiple OS versions
    ●
    How to DRY up common driver configurations?
    ●
    Where to put IP Configuration?
    

    View Slide

58. View Slide

59. View Slide

60. Directory
    Layout
    One log file
    per VM
    “Smoke” Test
    

    View Slide

61. tk-demo
    Demo project for test-kitchen, to demonstrate
    how to use it for mulit-vm integration tests
    https://github.com/bryanwb/tk-demo
    

    View Slide

62. test-kitchen + Rake
    

    View Slide

63. View Slide

64. Wiring things together is hard
    Apache Haproxy
    App
    Server
    Hello World!
    GET / localhost
    

    View Slide

65. Chef-Workflow
    Apache Haproxy
    App
    Server
    Chef-Server
    Chef-client Chef-client Chef-client
    A set of helpers for managing workflow
    

    View Slide

66. Chef-Workflow
    ●
    Written by Erik Hollensbe
    ●
    3 libraries
    ●
    Chef-workflow
    ●
    chef-workflow-tasklib
    ●
    chef-workflow-testlib
    ●
    Deserves more attention than it is given here
    

    View Slide

67. Chef-workflow
    Task Example
    

    View Slide

68. Chef-workflow
    Test Example
    

    View Slide

69. We need a layered
    approach to
    systems
    management
    

    View Slide

70. Image credit: http://i.technet.microsoft.com/dynimg/IC197700.gif
    

    View Slide

71. View Slide

72. BOSS* Bryan's Orchestration is
    Super-Simple DSL
    *Just a mock-up
    

    View Slide

73. even simpler
    

    View Slide

74. Awesome Ruby Libraries
    for use in integration tests
    ●
    Faraday
    ●
    Sinatra
    ●
    Rspec-dns
    ●
    Ruby-dns
    

    View Slide

75. View Slide

76. Integration Testing
    Heroes
    

    View Slide

77. In Summary
    ●
    You can do basic integration testing right now
    with vagrant + rake
    ●
    Concurrent VM task execution important
    ●
    Need DSL for integration tests
    ●
    Test-Kitchen is still alpha but very exciting
    ●
    There is gold in Chef-Workflow
    ●
    We need Public CI for chef cookbooks, puppet
    modules, and cfengine bundles
    

    View Slide

78. References
    ●
    minitest-chef-handler
    ●
    chef-workflow
    ●
    RSpec
    ●
    Test-Kitchen
    ●
    Test-Kitchen 1.0 (formerly known as Jamie-CI)
    ●
    Vagrant
    ●
    berkshelf
    ●
    Apache Mesos
    ●
    Cloudera Impala
    Copyright Bryan W. Berry, Creative Commons 3.0, Unported
    

    View Slide

79. References
    ●
    kitchen-vagrant
    ●
    Celluloid
    ●
    BATS
    ●
    tk-demo, the code used in this presentation
    Copyright Bryan W. Berry, Creative Commons 3.0, Unported
    

    View Slide

80. Special Thanks To
    ●
    Fletcher Nichol
    ●
    Jamie Winsor
    ●
    Nathen Harvey
    ●
    Joshua Timberman
    ●
    Erik Hollensbe
    for their assistance w/ this presentation
    

    View Slide