Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ChefConf 2013 - Test Kitchen: Multi-Platform Integration Testing for the Masses

ChefConf 2013 - Test Kitchen: Multi-Platform Integration Testing for the Masses

Test Kitchen was born out of an Opscode Engineering need to drive cross-platform integration tests for our Ruby and Erlang projects. Jamie CI was created by Fletcher Nichol to solve a similar problem he had with cross-platform cookbook testing. This talk tells the story of how these two projects joined forces to become the awesomeness that is Test Kitchen 1.0!

We will also cover Test Kitchen's design goals for 1.0, the new modular architecture, and how it can be extended to solve new problems. We'll also discuss how the new version can be used for cookbook integration testing. Testing shouldn't be a chore, and the tools you use should get out of your way to let you run tests as quickly as possible.

9891e8299426fb9b6e361b84b3155a2d?s=128

Fletcher Nichol

April 28, 2013
Tweet

Transcript

  1. Test Kitchen Multi-Platform Integration Testing for the Masses Seth Chisamore,

    Opscode Joshua Timberman, Opscode Fletcher Nichol, Blue Box Group
  2. A Tool is Born Part 1

  3. LDAP authentication is added to Private Chef's WebUI

  4. “How in tarnation are we going to test this??”

  5. NO PROBLEM!! We have a Cucumber integration test suite.

  6. None
  7. But we need to run it against ALL THE PLATFORMS

    Private Chef ships on!
  8. (›ớ‹,)

  9. Test setup was complicated

  10. Now rinse and repeat on 3 more platforms!

  11. None
  12. None
  13. None
  14. None
  15. None
  16. Test Kitchen Goes Public Part 2

  17. "Hey we should use Test Kitchen for cookbook testing..."

  18. Cookbook Testing • Actually run Chef on a node •

    Across every platform they support • By hand!
  19. None
  20. Requirements were gathered • Cross Platform • Regression testing •

    Linting, Unit/Spec testing • Functional/integration testing • Jenkins/CI
  21. Development continued • Andrew Crump (foodcritic author) expanded Test Kitchen

    for cookbook projects • He set up Jenkins builds for cookbooks • Announced the release, early and often • Cookbooks were updated to support Test Kitchen
  22. Test Kitchen is awesome • "kitchen test" in a repository:

    • Creates Kitchenfile • Runs foodcritic, knife cookbook test • Resolves cookbook dependencies • Runs vagrant w/ default recipe • Runs minitest-chef tests if present • Automatically.
  23. None
  24. There were Limitations • Adding new providers (e.g. OpenStack) was

    non-trivial • Override/local customization (e.g., Kitchenfile, Vagrantfile) was also non-trivial
  25. Jamie CI: The Accidental Project Part 3

  26. RVM Cookbook Complex

  27. RVM Cookbook Multiple installation setups

  28. RVM Cookbook Lengthy converges (+40 minutes/platform)

  29. RVM Cookbook Many resources/ providers (LWRPs)

  30. Friday Night Hacks™

  31. Friday Night Hacks™ December 1, 2012

  32. Friday Night Hacks™ RVM pull request testing

  33. Friday Night Hacks™ Frustration…

  34. Friday Night Hacks™ …

  35. “I know, I’ll use Vagrant!”

  36. Friday Night Hacks™ ✔ Insane Vagrantfile

  37. Friday Night Hacks™ ✔ Extract to Ruby Hash

  38. Friday Night Hacks™ ✔ Extract to YAML

  39. “Houston, we have a library!”

  40. Validation

  41. Validation December 2012

  42. Validation Frequent Google Hangout meetings

  43. Validation Working quietly in the open

  44. Kumbaya Part 4

  45. The Opscode Email

  46. The Opscode Email January 10, 2013

  47. The Opscode Email Introduce Jamie

  48. The Opscode Email Explain the “Why?”

  49. The Opscode Email Wait for feedback

  50. ...

  51. ...and...

  52. Success!

  53. YAY!

  54. Jamie codebase + Test Kitchen name

  55. Kitchen Concepts Part 5

  56. Concepts

  57. Concepts .kitchen.yml

  58. .kitchen.yml Description of testing problem

  59. .kitchen.yml Declarative & static

  60. Concepts Platform

  61. Platform Operating system or target system

  62. Platform OS version

  63. Platform OS architecture

  64. Platform Networking, virtualized, Chef version, etc.

  65. Platform It has a NAME

  66. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list:

    - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
  67. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list:

    - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
  68. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list:

    - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
  69. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list:

    - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
  70. Concepts Suite

  71. Suite The Chef configuration

  72. Suite A run list

  73. Suite Node attributes

  74. Suite It has a NAME

  75. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes:

    rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
  76. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes:

    rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
  77. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes:

    rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
  78. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes:

    rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
  79. Concepts Instance

  80. Instance Suite + Platform = Instance

  81. Instance Name is auto- generated

  82. Instance “The thing you run to test”

  83. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63

    <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
  84. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63

    <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
  85. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63

    <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
  86. Concepts Instance Actions

  87. Instance Actions “Lifecycle activities”

  88. Instance Actions 1. Create

  89. Instance Actions 2. Converge

  90. Instance Actions 3. Setup

  91. Instance Actions 4. Verify

  92. Instance Actions 5. Destroy

  93. Instance Actions (11). Test

  94. Concepts Driver

  95. Driver Implementation of instance actions

  96. Driver Ships as a RubyGem

  97. Driver One driver per instance

  98. Driver Defines its own configuration

  99. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: -

    name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
  100. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: -

    name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
  101. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: -

    name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
  102. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: -

    name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
  103. Driver Implementations kitchen-bluebox kitchen-ec2 kitchen-lxc kitchen-openstack kitchen-rackspace kitchen-vagrant

  104. Driver `kitchen driver discover`

  105. Lightning Guide Part 6

  106. Getting started

  107. … in 3 Steps

  108. Go! gem install test-kitchen --pre

  109. Go! kitchen init

  110. Go! kitchen test

  111. Go! Boom.

  112. The Future Part 7

  113. Spit & Polish • Documentation • Test Coverage • Better

    Error Output
  114. Core Improvements • Provisioner abstraction (e.g., Puppet support) • Windows

    support (host/guest)
  115. 3rd Party Integration

  116. Out a Ways • Multi-VM/Cluster support

  117. Thanks fnichol fnichol fnichol@nichol.ca schisamo schisamo schisamo@opscode.com jtimberman jtimberman joshua@opscode.com

  118. Photo Credits: http://www.flickr.com/photos/vagabondblogger/4163372966/ http://www.flickr.com/photos/26524277@N04/6870401434/ http://www.flickr.com/photos/walkn/3859852351/ http://www.flickr.com/photos/hebe/2171258965/ http://www.flickr.com/photos/methyl_lives/6009938677/ http://www.flickr.com/photos/derekbever/4413413377/ http://www.flickr.com/photos/emmajane/2580835224/ http://www.flickr.com/photos/whartz/1913603367/

    http://www.flickr.com/photos/bobaubuchon/5664242537/ http://www.flickr.com/photos/kubina/2090583987/ http://www.flickr.com/photos/leejordan/2483708561/ http://www.flickr.com/photos/peyri/304354485/ http://www.flickr.com/photos/joeshlabotnik/3707230247/ http://www.flickr.com/photos/skrb/179608762/ http://www.flickr.com/photos/nick_c/112111560/ http://www.flickr.com/photos/beltzner/2954046668/ http://www.flickr.com/photos/mattwyn/2448866914/ http://www.flickr.com/photos/beltzner/2954046668 http://www.flickr.com/photos/qmnonic/4290558018/