Puppet Module Reusability

Presentation from PuppetConf 2013 in San Francisco.

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

Transcript

1. Puppet Module Reusability Learning from shipping to the Forge Gareth Rushgrove
2. Who (Who is this person?)
3. @garethr
4. UK Government Digital Service
8. Last code I wrote
11. 7. Gareth Rushgrove
12. Last puppet module I wrote
13. The Problem (Sharing modules is hard)
14. Not invented here syndrome 1
15. A search for puppet-apache
16. 145 Gareth Rushgrove
17. 145 Gareth Rushgrove
18. Puppetlabs apache
19. 67 contributors Gareth Rushgrove
20. 281 forks Gareth Rushgrove
21. 281 Gareth Rushgrove
22. Vendor everything pattern 2
23. Not publishing to the Forge 3
24. Puppet Forge
25. Undocumented modules 4
26. Official docs guidelines
27. (Too) opinionated configuration 5
28. Sometimes I just want to write nginx configuration Gareth Rushgrove
29. Duplicate resources 6
30. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove
31. Stdlib (Start here)
32. Standard Library from Puppetlabs
33. Fail fast Gareth Rushgrove
34. Validation Gareth Rushgrove
35. Useful error messages Gareth Rushgrove
36. fail("${::operatingsystem} not supported") Gareth Rushgrove
37. validate_string Gareth Rushgrove
38. validate_string($version) Gareth Rushgrove
39. <Puppet::Error: true is not a string. Gareth Rushgrove
40. validate_slength Gareth Rushgrove
41. validate_re Gareth Rushgrove
42. validate_hash Gareth Rushgrove
43. Gareth Rushgrove validate_cmd
44. validate_bool Gareth Rushgrove
45. Avoid duplicate resources Gareth Rushgrove
46. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove
47. include 'gcc' Gareth Rushgrove
48. package{[ 'python-pip', 'python-ldap', ]: ensure => installed } Gareth Rushgrove
49. ensure_packages Gareth Rushgrove
50. ensure_packages([ 'python-pip', 'python-ldap', ]) Gareth Rushgrove
51. ensure_resource Gareth Rushgrove
52. Nicer interfaces Gareth Rushgrove
53. str2bool Gareth Rushgrove
54. any2array Gareth Rushgrove
55. And much more Gareth Rushgrove
56. Dependency management is everywhere (Else)
57. NPM, Bundler, Pip, Mix, Leiningen Gareth Rushgrove
58. Librarian Puppet
59. r10k
60. echo 'modules' >> .gitignore Gareth Rushgrove
61. dependency 'puppetlabs/stdlib' dependency 'garethr/erlang' dependency 'maestrodev/wget' Gareth Rushgrove
62. forge "http://forge.puppetlabs.com" mod 'puppetlabs/ruby' mod 'puppetlabs/ntp' mod 'puppetlabs/git' mod 'puppetlabs/vcsrepo' mod 'puppetlabs/apt' mod 'puppetlabs/gcc' Gareth Rushgrove
63. librarian-puppet install Gareth Rushgrove
64. A Nice Module Pattern (Structuring modules for reuse)
65. R.I.Pienaar
66. install, config, service, params Gareth Rushgrove
67. manifests !"" config.pp !"" init.pp !"" install.pp !"" params.pp #"" service.pp Gareth Rushgrove
68. class sample ( ) inherits sample::params { class { 'sample::install': } -> class { 'sample::config': } ~> class { 'sample::service': } -> Class['sample'] } Gareth Rushgrove
69. anchor { 'sample::begin': } -> class { 'sample::install': } -> class { 'sample::config': } class { 'sample::service': } -> anchor { 'sample::end': } Gareth Rushgrove
70. Anchor['sample::begin'] ~> Class['sample::service'] Class['sample::install'] ~> Class['sample::service'] Class['sample::config'] ~> Class['sample::service'] Gareth Rushgrove
71. Puppet module tool Gareth Rushgrove
72. puppet module generate name Gareth Rushgrove
73. Default module skeleton Gareth Rushgrove
74. In puppet source code
75. !"" manifests !"" spec !"" tests !"" Modulefile !"" .gitignore #"" README Gareth Rushgrove
76. Creating your own module skeleton Gareth Rushgrove
77. ~/.puppet/var/puppet-module/skeleton Gareth Rushgrove
78. puppet module skeleton
79. !"" manifests !"" spec !"" tests !"" templates !"" files !"" lib !"" Gemfile Gareth Rushgrove
80. !"" Rakefile !"" .nodeset.yml !"" .fixtures.yml !"" .travis.yml !"" Modulefile !"" .gitignore #"" README.md Gareth Rushgrove
81. Creates install, config, service, params classes Gareth Rushgrove
82. Dependency management with Bundler Gareth Rushgrove
83. Better unit testing setup using rspec- puppet-helper Gareth Rushgrove
84. Adds syntax and lint checking Gareth Rushgrove
85. Adds Travis CI configuration Gareth Rushgrove
86. Adds integration tests with rspec-system Gareth Rushgrove
87. Adds module management with maestrodev blacksmith Gareth Rushgrove
88. Focus on the interface Gareth Rushgrove
89. Minimize number of entry points Gareth Rushgrove
90. Allow overriding provided templates Gareth Rushgrove
91. Multi-OS Support (Where to abstract the differences)
92. Vary default parameters Gareth Rushgrove
93. Keep logic in one place Gareth Rushgrove
94. case $::osfamily { 'Debian': { } 'RedHat', 'Amazon': { } default: { fail("${::operatingsystem} not su } Gareth Rushgrove
95. ARM-9 Data in Modules
96. garethr-erlang
97. 0.0.x supported Ubuntu only Gareth Rushgrove
98. 0.1.x supports Ubuntu, Debian, Redhat, Suse Gareth Rushgrove
99. Gareth Rushgrove
100. Pull requests to the rescue
102. Insist on tests with open source contributions Gareth Rushgrove
103. Module Testing (Code should have tests)
104. Why test a declarative language? Gareth Rushgrove
105. Modules increasingly contain logic Gareth Rushgrove
106. Modules increasingly take arguments Gareth Rushgrove
107. Modules increasingly have interfaces with other modules Gareth Rushgrove
108. Modules increasingly used in many Ruby and Puppet version combinations Gareth Rushgrove
109. Good news. The tools got better Gareth Rushgrove
110. puppet-lint Gareth Rushgrove
111. Puppet style guide
112. Available as a gem
113. puppet-lint --with-filename /etc/puppet/modules foo/manifests/bar.pp: trailing whitespace found on line 1 apache/manifests/server.pp: variable not enclosed in {} on line 56 Gareth Rushgrove
114. puppet-syntax Gareth Rushgrove
115. Validate Puppet and ERB syntax
116. require 'puppet-syntax/tasks/puppet-syntax' Gareth Rushgrove
117. bundle exec rake syntax ---> syntax:manifests ---> syntax:templates Gareth Rushgrove
118. rspec-puppet Gareth Rushgrove
119. Unit testing for Puppet
120. context "epel enabled" do let(:params) {{ :epel_enable => true }} it { should contain_class('epel') } end Gareth Rushgrove
121. context "epel disabled" do let(:params) {{ :epel_enable => false }} it { should_not contain_class('epel') } end Gareth Rushgrove
122. Fixtures, matchers
123. Test the interface not the implementation Gareth Rushgrove
124. All of the above Gareth Rushgrove
125. task :test => [ :syntax, :lint, :spec, ] Gareth Rushgrove
126. bundle exec rake test Gareth Rushgrove
127. Gareth Rushgrove
128. Nice continuous integration
130. Test pull request branches too
131. --- language: ruby before_install: rm Gemfile.lock || true rvm: - 1.8.7 - 1.9.3 script: bundle exec rake test env: matrix: - PUPPET_VERSION="~> 2.7.0" - PUPPET_VERSION="~> 3.1.0" - PUPPET_VERSION="~> 3.2.0" Gareth Rushgrove
132. A matrix of possibilities Gareth Rushgrove
133. rspec-system Gareth Rushgrove
134. Integration test against real systems
135. Puppet helpers for rspec-system
136. Vagrant driver
137. VSphere provider
138. Test that Puppet runs without errors Gareth Rushgrove
139. it 'should run without errors' do pp = "class { 'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 end end Gareth Rushgrove
140. Test runs are idempotent Gareth Rushgrove
141. it 'should run without errors' do pp = "class { 'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 r.refresh r.exit_code.should be_zero end end Gareth Rushgrove
142. Test that the module installs relevant binaries Gareth Rushgrove
143. it 'should install the erl binary' do shell 'which erl' do |r| r.stdout.should =~ /\/usr\/bin\/e r.stderr.should be_empty r.exit_code.should be_zero end end Gareth Rushgrove
144. Test against different operating systems Gareth Rushgrove
145. default_set: 'centos-64-x64' sets: 'centos-64-x64': nodes: "main.foo.vm": prefab: 'centos-64-x64' 'fedora-18-x64': nodes: "main.foo.vm": prefab: 'fedora-18-x64' 'debian-607-x64': nodes: "main.foo.vm": Gareth Rushgrove
146. Room for improvements (Making reuse easier in Puppet)
147. Run your own Forge 1
148. A more secure Forge 2
149. Bless a dependency management tool 3
150. Optional dependencies 4
151. dependency 'puppetlabs/stdlib' optional_dependency 'puppetlabs/apache' Gareth Rushgrove
152. Private classes 5
153. private class elixir::install { Gareth Rushgrove
154. Parameter naming conventions 6
155. example42 Proposal
156. Interfaces in Puppet 7
157. interface packageandservice { $version $enable $package_name $template } Gareth Rushgrove
158. class diamond ( $version, $enable, $package_name, $template, ) { implements packageandservice Gareth Rushgrove
159. Questions? (And thanks for listening)