ChefConf 2013 - Test Kitchen: Multi-Platform Integration Testing for the Masses

Test Kitchen was born out of an Opscode Engineering need to drive cross-platform integration tests for our Ruby and Erlang projects. Jamie CI was created by Fletcher Nichol to solve a similar problem he had with cross-platform cookbook testing. This talk tells the story of how these two projects joined forces to become the awesomeness that is Test Kitchen 1.0!

We will also cover Test Kitchen's design goals for 1.0, the new modular architecture, and how it can be extended to solve new problems. We'll also discuss how the new version can be used for cookbook integration testing. Testing shouldn't be a chore, and the tools you use should get out of your way to let you run tests as quickly as possible.

Transcript

1. Test Kitchen Multi-Platform Integration Testing for the Masses Seth Chisamore, Opscode Joshua Timberman, Opscode Fletcher Nichol, Blue Box Group
2. A Tool is Born Part 1
3. LDAP authentication is added to Private Chef's WebUI
4. “How in tarnation are we going to test this??”
5. NO PROBLEM!! We have a Cucumber integration test suite.
7. But we need to run it against ALL THE PLATFORMS Private Chef ships on!
8. (›ớ‹,)
9. Test setup was complicated
10. Now rinse and repeat on 3 more platforms!
16. Test Kitchen Goes Public Part 2
17. "Hey we should use Test Kitchen for cookbook testing..."
18. Cookbook Testing • Actually run Chef on a node • Across every platform they support • By hand!
20. Requirements were gathered • Cross Platform • Regression testing • Linting, Unit/Spec testing • Functional/integration testing • Jenkins/CI
21. Development continued • Andrew Crump (foodcritic author) expanded Test Kitchen for cookbook projects • He set up Jenkins builds for cookbooks • Announced the release, early and often • Cookbooks were updated to support Test Kitchen
22. Test Kitchen is awesome • "kitchen test" in a repository: • Creates Kitchenfile • Runs foodcritic, knife cookbook test • Resolves cookbook dependencies • Runs vagrant w/ default recipe • Runs minitest-chef tests if present • Automatically.
24. There were Limitations • Adding new providers (e.g. OpenStack) was non-trivial • Override/local customization (e.g., Kitchenfile, Vagrantfile) was also non-trivial
25. Jamie CI: The Accidental Project Part 3
26. RVM Cookbook Complex
27. RVM Cookbook Multiple installation setups
28. RVM Cookbook Lengthy converges (+40 minutes/platform)
29. RVM Cookbook Many resources/ providers (LWRPs)
30. Friday Night Hacks™
31. Friday Night Hacks™ December 1, 2012
32. Friday Night Hacks™ RVM pull request testing
33. Friday Night Hacks™ Frustration…
34. Friday Night Hacks™ …
35. “I know, I’ll use Vagrant!”
36. Friday Night Hacks™ ✔ Insane Vagrantfile
37. Friday Night Hacks™ ✔ Extract to Ruby Hash
38. Friday Night Hacks™ ✔ Extract to YAML
39. “Houston, we have a library!”
40. Validation
41. Validation December 2012
42. Validation Frequent Google Hangout meetings
43. Validation Working quietly in the open
44. Kumbaya Part 4
45. The Opscode Email
46. The Opscode Email January 10, 2013
47. The Opscode Email Introduce Jamie
48. The Opscode Email Explain the “Why?”
49. The Opscode Email Wait for feedback
50. ...
51. ...and...
52. Success!
53. YAY!
54. Jamie codebase + Test Kitchen name
55. Kitchen Concepts Part 5
56. Concepts
57. Concepts .kitchen.yml
58. .kitchen.yml Description of testing problem
59. .kitchen.yml Declarative & static
60. Concepts Platform
61. Platform Operating system or target system
62. Platform OS version
63. Platform OS architecture
64. Platform Networking, virtualized, Chef version, etc.
65. Platform It has a NAME
66. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list: - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
67. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list: - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
68. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list: - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
69. platforms: - name: ubuntu-12.04 driver_config: box: opscode-ubuntu-12.04 box_url: https://...box run_list: - recipe[apt] - name: centos-6.3 driver_config: box: opscode-centos-6.3 box_url: https://...box run_list: - recipe[yum::epel] .kitchen.yml: platform
70. Concepts Suite
71. Suite The Chef configuration
72. Suite A run list
73. Suite Node attributes
74. Suite It has a NAME
75. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes: rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
76. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes: rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
77. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes: rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
78. suites: - name: rubies run_list: - recipe[java] - recipe[rvm::system] attributes: rvm: default_ruby: system rubies: - 1.9.3 - version: 1.9.3-p327 patch: railsexpress - version: 1.8.7 rubygems_version: 1.6.0 .kitchen.yml: suite
79. Concepts Instance
80. Instance Suite + Platform = Instance
81. Instance Name is auto- generated
82. Instance “The thing you run to test”
83. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63 <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
84. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63 <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
85. > kitchen list Instance Last Action rubies-ubuntu-1204 <Not Created> rubies-centos-63 <Not Created> system-ubuntu-1204 <Not Created> system-centos-63 <Not Created> Instances
86. Concepts Instance Actions
87. Instance Actions “Lifecycle activities”
88. Instance Actions 1. Create
89. Instance Actions 2. Converge
90. Instance Actions 3. Setup
91. Instance Actions 4. Verify
92. Instance Actions 5. Destroy
93. Instance Actions (11). Test
94. Concepts Driver
95. Driver Implementation of instance actions
96. Driver Ships as a RubyGem
97. Driver One driver per instance
98. Driver Defines its own configuration
99. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: - name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
100. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: - name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
101. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: - name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
102. --- driver_plugin: ec2 driver_config: region: us-east-1 flavor_id: m1.small platforms: - name: smartos driver_plugin: vagrant driver_config: box: smartos - name: freebsd driver_config: image_id: ami-5339bb3a .kitchen.yml: driver_config
103. Driver Implementations kitchen-bluebox kitchen-ec2 kitchen-lxc kitchen-openstack kitchen-rackspace kitchen-vagrant
104. Driver `kitchen driver discover`
105. Lightning Guide Part 6
106. Getting started
107. … in 3 Steps
108. Go! gem install test-kitchen --pre
109. Go! kitchen init
110. Go! kitchen test
111. Go! Boom.
112. The Future Part 7
113. Spit & Polish • Documentation • Test Coverage • Better Error Output
114. Core Improvements • Provisioner abstraction (e.g., Puppet support) • Windows support (host/guest)
115. 3rd Party Integration
116. Out a Ways • Multi-VM/Cluster support
117. Thanks fnichol fnichol fnichol@nichol.ca schisamo schisamo schisamo@opscode.com jtimberman jtimberman joshua@opscode.com
118. Photo Credits: http://www.flickr.com/photos/vagabondblogger/4163372966/ http://www.flickr.com/photos/26524277@N04/6870401434/ http://www.flickr.com/photos/walkn/3859852351/ http://www.flickr.com/photos/hebe/2171258965/ http://www.flickr.com/photos/methyl_lives/6009938677/ http://www.flickr.com/photos/derekbever/4413413377/ http://www.flickr.com/photos/emmajane/2580835224/ http://www.flickr.com/photos/whartz/1913603367/ http://www.flickr.com/photos/bobaubuchon/5664242537/ http://www.flickr.com/photos/kubina/2090583987/ http://www.flickr.com/photos/leejordan/2483708561/ http://www.flickr.com/photos/peyri/304354485/ http://www.flickr.com/photos/joeshlabotnik/3707230247/ http://www.flickr.com/photos/skrb/179608762/ http://www.flickr.com/photos/nick_c/112111560/ http://www.flickr.com/photos/beltzner/2954046668/ http://www.flickr.com/photos/mattwyn/2448866914/ http://www.flickr.com/photos/beltzner/2954046668 http://www.flickr.com/photos/qmnonic/4290558018/