Puppet Module Reusability

Puppet Module Reusability

Presentation from PuppetConf 2013 in San Francisco.

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

August 23, 2013
Tweet

Transcript

  1. Puppet Module Reusability Learning from shipping to the Forge Gareth

    Rushgrove
  2. Who (Who is this person?)

  3. @garethr

  4. UK Government Digital Service

  5. None
  6. None
  7. None
  8. Last code I wrote

  9. None
  10. None
  11. 7. Gareth Rushgrove

  12. Last puppet module I wrote

  13. The Problem (Sharing modules is hard)

  14. Not invented here syndrome 1

  15. A search for puppet-apache

  16. 145 Gareth Rushgrove

  17. 145 Gareth Rushgrove

  18. Puppetlabs apache

  19. 67 contributors Gareth Rushgrove

  20. 281 forks Gareth Rushgrove

  21. 281 Gareth Rushgrove

  22. Vendor everything pattern 2

  23. Not publishing to the Forge 3

  24. Puppet Forge

  25. Undocumented modules 4

  26. Official docs guidelines

  27. (Too) opinionated configuration 5

  28. Sometimes I just want to write nginx configuration Gareth Rushgrove

  29. Duplicate resources 6

  30. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove

  31. Stdlib (Start here)

  32. Standard Library from Puppetlabs

  33. Fail fast Gareth Rushgrove

  34. Validation Gareth Rushgrove

  35. Useful error messages Gareth Rushgrove

  36. fail("${::operatingsystem} not supported") Gareth Rushgrove

  37. validate_string Gareth Rushgrove

  38. validate_string($version) Gareth Rushgrove

  39. <Puppet::Error: true is not a string. Gareth Rushgrove

  40. validate_slength Gareth Rushgrove

  41. validate_re Gareth Rushgrove

  42. validate_hash Gareth Rushgrove

  43. Gareth Rushgrove validate_cmd

  44. validate_bool Gareth Rushgrove

  45. Avoid duplicate resources Gareth Rushgrove

  46. Package { 'build-essential’: ensure => installed, } Gareth Rushgrove

  47. include 'gcc' Gareth Rushgrove

  48. package{[ 'python-pip', 'python-ldap', ]: ensure => installed } Gareth Rushgrove

  49. ensure_packages Gareth Rushgrove

  50. ensure_packages([ 'python-pip', 'python-ldap', ]) Gareth Rushgrove

  51. ensure_resource Gareth Rushgrove

  52. Nicer interfaces Gareth Rushgrove

  53. str2bool Gareth Rushgrove

  54. any2array Gareth Rushgrove

  55. And much more Gareth Rushgrove

  56. Dependency management is everywhere (Else)

  57. NPM, Bundler, Pip, Mix, Leiningen Gareth Rushgrove

  58. Librarian Puppet

  59. r10k

  60. echo 'modules' >> .gitignore Gareth Rushgrove

  61. dependency 'puppetlabs/stdlib' dependency 'garethr/erlang' dependency 'maestrodev/wget' Gareth Rushgrove

  62. forge "http://forge.puppetlabs.com" mod 'puppetlabs/ruby' mod 'puppetlabs/ntp' mod 'puppetlabs/git' mod 'puppetlabs/vcsrepo'

    mod 'puppetlabs/apt' mod 'puppetlabs/gcc' Gareth Rushgrove
  63. librarian-puppet install Gareth Rushgrove

  64. A Nice Module Pattern (Structuring modules for reuse)

  65. R.I.Pienaar

  66. install, config, service, params Gareth Rushgrove

  67. manifests !"" config.pp !"" init.pp !"" install.pp !"" params.pp #""

    service.pp Gareth Rushgrove
  68. class sample ( ) inherits sample::params { class { 'sample::install':

    } -> class { 'sample::config': } ~> class { 'sample::service': } -> Class['sample'] } Gareth Rushgrove
  69. anchor { 'sample::begin': } -> class { 'sample::install': } ->

    class { 'sample::config': } class { 'sample::service': } -> anchor { 'sample::end': } Gareth Rushgrove
  70. Anchor['sample::begin'] ~> Class['sample::service'] Class['sample::install'] ~> Class['sample::service'] Class['sample::config'] ~> Class['sample::service'] Gareth

    Rushgrove
  71. Puppet module tool Gareth Rushgrove

  72. puppet module generate name Gareth Rushgrove

  73. Default module skeleton Gareth Rushgrove

  74. In puppet source code

  75. !"" manifests !"" spec !"" tests !"" Modulefile !"" .gitignore

    #"" README Gareth Rushgrove
  76. Creating your own module skeleton Gareth Rushgrove

  77. ~/.puppet/var/puppet-module/skeleton Gareth Rushgrove

  78. puppet module skeleton

  79. !"" manifests !"" spec !"" tests !"" templates !"" files

    !"" lib !"" Gemfile Gareth Rushgrove
  80. !"" Rakefile !"" .nodeset.yml !"" .fixtures.yml !"" .travis.yml !"" Modulefile

    !"" .gitignore #"" README.md Gareth Rushgrove
  81. Creates install, config, service, params classes Gareth Rushgrove

  82. Dependency management with Bundler Gareth Rushgrove

  83. Better unit testing setup using rspec- puppet-helper Gareth Rushgrove

  84. Adds syntax and lint checking Gareth Rushgrove

  85. Adds Travis CI configuration Gareth Rushgrove

  86. Adds integration tests with rspec-system Gareth Rushgrove

  87. Adds module management with maestrodev blacksmith Gareth Rushgrove

  88. Focus on the interface Gareth Rushgrove

  89. Minimize number of entry points Gareth Rushgrove

  90. Allow overriding provided templates Gareth Rushgrove

  91. Multi-OS Support (Where to abstract the differences)

  92. Vary default parameters Gareth Rushgrove

  93. Keep logic in one place Gareth Rushgrove

  94. case $::osfamily { 'Debian': { } 'RedHat', 'Amazon': { }

    default: { fail("${::operatingsystem} not su } Gareth Rushgrove
  95. ARM-9 Data in Modules

  96. garethr-erlang

  97. 0.0.x supported Ubuntu only Gareth Rushgrove

  98. 0.1.x supports Ubuntu, Debian, Redhat, Suse Gareth Rushgrove

  99. Gareth Rushgrove

  100. Pull requests to the rescue

  101. None
  102. Insist on tests with open source contributions Gareth Rushgrove

  103. Module Testing (Code should have tests)

  104. Why test a declarative language? Gareth Rushgrove

  105. Modules increasingly contain logic Gareth Rushgrove

  106. Modules increasingly take arguments Gareth Rushgrove

  107. Modules increasingly have interfaces with other modules Gareth Rushgrove

  108. Modules increasingly used in many Ruby and Puppet version combinations

    Gareth Rushgrove
  109. Good news. The tools got better Gareth Rushgrove

  110. puppet-lint Gareth Rushgrove

  111. Puppet style guide

  112. Available as a gem

  113. puppet-lint --with-filename /etc/puppet/modules foo/manifests/bar.pp: trailing whitespace found on line 1

    apache/manifests/server.pp: variable not enclosed in {} on line 56 Gareth Rushgrove
  114. puppet-syntax Gareth Rushgrove

  115. Validate Puppet and ERB syntax

  116. require 'puppet-syntax/tasks/puppet-syntax' Gareth Rushgrove

  117. bundle exec rake syntax ---> syntax:manifests ---> syntax:templates Gareth Rushgrove

  118. rspec-puppet Gareth Rushgrove

  119. Unit testing for Puppet

  120. context "epel enabled" do let(:params) {{ :epel_enable => true }}

    it { should contain_class('epel') } end Gareth Rushgrove
  121. context "epel disabled" do let(:params) {{ :epel_enable => false }}

    it { should_not contain_class('epel') } end Gareth Rushgrove
  122. Fixtures, matchers

  123. Test the interface not the implementation Gareth Rushgrove

  124. All of the above Gareth Rushgrove

  125. task :test => [ :syntax, :lint, :spec, ] Gareth Rushgrove

  126. bundle exec rake test Gareth Rushgrove

  127. Gareth Rushgrove

  128. Nice continuous integration

  129. None
  130. Test pull request branches too

  131. --- language: ruby before_install: rm Gemfile.lock || true rvm: -

    1.8.7 - 1.9.3 script: bundle exec rake test env: matrix: - PUPPET_VERSION="~> 2.7.0" - PUPPET_VERSION="~> 3.1.0" - PUPPET_VERSION="~> 3.2.0" Gareth Rushgrove
  132. A matrix of possibilities Gareth Rushgrove

  133. rspec-system Gareth Rushgrove

  134. Integration test against real systems

  135. Puppet helpers for rspec-system

  136. Vagrant driver

  137. VSphere provider

  138. Test that Puppet runs without errors Gareth Rushgrove

  139. it 'should run without errors' do pp = "class {

    'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 end end Gareth Rushgrove
  140. Test runs are idempotent Gareth Rushgrove

  141. it 'should run without errors' do pp = "class {

    'sample': }" puppet_apply(pp) do |r| r.exit_code.should == 2 r.refresh r.exit_code.should be_zero end end Gareth Rushgrove
  142. Test that the module installs relevant binaries Gareth Rushgrove

  143. it 'should install the erl binary' do shell 'which erl'

    do |r| r.stdout.should =~ /\/usr\/bin\/e r.stderr.should be_empty r.exit_code.should be_zero end end Gareth Rushgrove
  144. Test against different operating systems Gareth Rushgrove

  145. default_set: 'centos-64-x64' sets: 'centos-64-x64': nodes: "main.foo.vm": prefab: 'centos-64-x64' 'fedora-18-x64': nodes:

    "main.foo.vm": prefab: 'fedora-18-x64' 'debian-607-x64': nodes: "main.foo.vm": Gareth Rushgrove
  146. Room for improvements (Making reuse easier in Puppet)

  147. Run your own Forge 1

  148. A more secure Forge 2

  149. Bless a dependency management tool 3

  150. Optional dependencies 4

  151. dependency 'puppetlabs/stdlib' optional_dependency 'puppetlabs/apache' Gareth Rushgrove

  152. Private classes 5

  153. private class elixir::install { Gareth Rushgrove

  154. Parameter naming conventions 6

  155. example42 Proposal

  156. Interfaces in Puppet 7

  157. interface packageandservice { $version $enable $package_name $template } Gareth Rushgrove

  158. class diamond ( $version, $enable, $package_name, $template, ) { implements

    packageandservice Gareth Rushgrove
  159. Questions? (And thanks for listening)