Puppet Module Reusability

Presentation from PuppetConf 2013 in San Francisco.

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

Transcript

1. Puppet Module
   Reusability
   Learning from shipping to the Forge
   Gareth Rushgrove
   

   View Slide

2. Who
   (Who is this person?)
   

   View Slide

3. @garethr
   

   View Slide

4. UK Government
   Digital Service
   

   View Slide

5. View Slide

6. View Slide

7. View Slide

8. Last code
   I wrote
   

   View Slide

9. View Slide

10. View Slide

11. 7. Gareth Rushgrove
    

    View Slide

12. Last puppet
    module I wrote
    

    View Slide

13. The Problem
    (Sharing modules is hard)
    

    View Slide

14. Not invented here
    syndrome
    1
    

    View Slide

15. A search for
    puppet-apache
    

    View Slide

16. 145
    Gareth Rushgrove
    

    View Slide

17. 145
    Gareth Rushgrove
    

    View Slide

18. Puppetlabs
    apache
    

    View Slide

19. 67
    contributors
    Gareth Rushgrove
    

    View Slide

20. 281
    forks
    Gareth Rushgrove
    

    View Slide

21. 281
    Gareth Rushgrove
    

    View Slide

22. Vendor everything
    pattern
    2
    

    View Slide

23. Not publishing to the
    Forge
    3
    

    View Slide

24. Puppet
    Forge
    

    View Slide

25. Undocumented
    modules
    4
    

    View Slide

26. Official docs
    guidelines
    

    View Slide

27. (Too) opinionated
    configuration
    5
    

    View Slide

28. Sometimes I just
    want to write nginx
    configuration
    Gareth Rushgrove
    

    View Slide

29. Duplicate resources
    6
    

    View Slide

30. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove
    

    View Slide

31. Stdlib
    (Start here)
    

    View Slide

32. Standard
    Library from
    Puppetlabs
    

    View Slide

33. Fail fast
    Gareth Rushgrove
    

    View Slide

34. Validation
    Gareth Rushgrove
    

    View Slide

35. Useful error
    messages
    Gareth Rushgrove
    

    View Slide

36. fail("${::operatingsystem}
    not supported")
    Gareth Rushgrove
    

    View Slide

37. validate_string
    Gareth Rushgrove
    

    View Slide

38. validate_string($version)
    Gareth Rushgrove
    

    View Slide

39. not a string.
    Gareth Rushgrove
    

    View Slide

40. validate_slength
    Gareth Rushgrove
    

    View Slide

41. validate_re
    Gareth Rushgrove
    

    View Slide

42. validate_hash
    Gareth Rushgrove
    

    View Slide

43. Gareth Rushgrove
    validate_cmd
    

    View Slide

44. validate_bool
    Gareth Rushgrove
    

    View Slide

45. Avoid duplicate
    resources
    Gareth Rushgrove
    

    View Slide

46. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove
    

    View Slide

47. include 'gcc'
    Gareth Rushgrove
    

    View Slide

48. package{[
    'python-pip',
    'python-ldap',
    ]:
    ensure => installed
    }
    Gareth Rushgrove
    

    View Slide

49. ensure_packages
    Gareth Rushgrove
    

    View Slide

50. ensure_packages([
    'python-pip',
    'python-ldap',
    ])
    Gareth Rushgrove
    

    View Slide

51. ensure_resource
    Gareth Rushgrove
    

    View Slide

52. Nicer interfaces
    Gareth Rushgrove
    

    View Slide

53. str2bool
    Gareth Rushgrove
    

    View Slide

54. any2array
    Gareth Rushgrove
    

    View Slide

55. And much more
    Gareth Rushgrove
    

    View Slide

56. Dependency
    management is
    everywhere
    (Else)
    

    View Slide

57. NPM, Bundler, Pip,
    Mix, Leiningen
    Gareth Rushgrove
    

    View Slide

58. Librarian
    Puppet
    

    View Slide

59. r10k
    

    View Slide

60. echo 'modules' >> .gitignore
    Gareth Rushgrove
    

    View Slide

61. dependency 'puppetlabs/stdlib'
    dependency 'garethr/erlang'
    dependency 'maestrodev/wget'
    Gareth Rushgrove
    

    View Slide

62. forge "http://forge.puppetlabs.com"
    mod 'puppetlabs/ruby'
    mod 'puppetlabs/ntp'
    mod 'puppetlabs/git'
    mod 'puppetlabs/vcsrepo'
    mod 'puppetlabs/apt'
    mod 'puppetlabs/gcc'
    Gareth Rushgrove
    

    View Slide

63. librarian-puppet install
    Gareth Rushgrove
    

    View Slide

64. A Nice
    Module Pattern
    (Structuring modules for reuse)
    

    View Slide

65. R.I.Pienaar
    

    View Slide

66. install, config,
    service, params
    Gareth Rushgrove
    

    View Slide

67. manifests
    !"" config.pp
    !"" init.pp
    !"" install.pp
    !"" params.pp
    #"" service.pp
    Gareth Rushgrove
    

    View Slide

68. class sample (
    ) inherits sample::params {
    class { 'sample::install': } ->
    class { 'sample::config': } ~>
    class { 'sample::service': } ->
    Class['sample']
    }
    Gareth Rushgrove
    

    View Slide

69. anchor { 'sample::begin': } ->
    class { 'sample::install': } ->
    class { 'sample::config': }
    class { 'sample::service': } ->
    anchor { 'sample::end': }
    Gareth Rushgrove
    

    View Slide

70. Anchor['sample::begin'] ~>
    Class['sample::service']
    Class['sample::install'] ~>
    Class['sample::service']
    Class['sample::config'] ~>
    Class['sample::service']
    Gareth Rushgrove
    

    View Slide

71. Puppet module tool
    Gareth Rushgrove
    

    View Slide

72. puppet module generate name
    Gareth Rushgrove
    

    View Slide

73. Default module
    skeleton
    Gareth Rushgrove
    

    View Slide

74. In puppet
    source code
    

    View Slide

75. !"" manifests
    !"" spec
    !"" tests
    !"" Modulefile
    !"" .gitignore
    #"" README
    Gareth Rushgrove
    

    View Slide

76. Creating your own
    module skeleton
    Gareth Rushgrove
    

    View Slide

77. ~/.puppet/var/puppet-module/skeleton
    Gareth Rushgrove
    

    View Slide

78. puppet module
    skeleton
    

    View Slide

79. !"" manifests
    !"" spec
    !"" tests
    !"" templates
    !"" files
    !"" lib
    !"" Gemfile
    Gareth Rushgrove
    

    View Slide

80. !"" Rakefile
    !"" .nodeset.yml
    !"" .fixtures.yml
    !"" .travis.yml
    !"" Modulefile
    !"" .gitignore
    #"" README.md
    Gareth Rushgrove
    

    View Slide

81. Creates install,
    config, service,
    params classes
    Gareth Rushgrove
    

    View Slide

82. Dependency
    management with
    Bundler
    Gareth Rushgrove
    

    View Slide

83. Better unit testing
    setup using rspec-
    puppet-helper
    Gareth Rushgrove
    

    View Slide

84. Adds syntax and lint
    checking
    Gareth Rushgrove
    

    View Slide

85. Adds Travis CI
    configuration
    Gareth Rushgrove
    

    View Slide

86. Adds integration tests
    with rspec-system
    Gareth Rushgrove
    

    View Slide

87. Adds module
    management with
    maestrodev
    blacksmith
    Gareth Rushgrove
    

    View Slide

88. Focus on the
    interface
    Gareth Rushgrove
    

    View Slide

89. Minimize number of
    entry points
    Gareth Rushgrove
    

    View Slide

90. Allow overriding
    provided templates
    Gareth Rushgrove
    

    View Slide

91. Multi-OS Support
    (Where to abstract the differences)
    

    View Slide

92. Vary default
    parameters
    Gareth Rushgrove
    

    View Slide

93. Keep logic in one
    place
    Gareth Rushgrove
    

    View Slide

94. case $::osfamily {
    'Debian': {
    }
    'RedHat', 'Amazon': {
    }
    default: {
    fail("${::operatingsystem} not su
    }
    Gareth Rushgrove
    

    View Slide

95. ARM-9 Data
    in Modules
    

    View Slide

96. garethr-erlang
    

    View Slide

97. 0.0.x supported
    Ubuntu only
    Gareth Rushgrove
    

    View Slide

98. 0.1.x supports
    Ubuntu, Debian,
    Redhat, Suse
    Gareth Rushgrove
    

    View Slide

99. Gareth Rushgrove
    

    View Slide

100. Pull requests to
     the rescue
     

     View Slide

101. View Slide

102. Insist on tests with
     open source
     contributions
     Gareth Rushgrove
     

     View Slide

103. Module Testing
     (Code should have tests)
     

     View Slide

104. Why test a
     declarative
     language?
     Gareth Rushgrove
     

     View Slide

105. Modules increasingly
     contain logic
     Gareth Rushgrove
     

     View Slide

106. Modules increasingly
     take arguments
     Gareth Rushgrove
     

     View Slide

107. Modules increasingly
     have interfaces with
     other modules
     Gareth Rushgrove
     

     View Slide

108. Modules increasingly
     used in many Ruby
     and Puppet version
     combinations
     Gareth Rushgrove
     

     View Slide

109. Good news. The tools
     got better
     Gareth Rushgrove
     

     View Slide

110. puppet-lint
     Gareth Rushgrove
     

     View Slide

111. Puppet
     style guide
     

     View Slide

112. Available
     as a gem
     

     View Slide

113. puppet-lint --with-filename /etc/puppet/modules
     foo/manifests/bar.pp: trailing whitespace found
     on line 1
     apache/manifests/server.pp: variable not enclosed
     in {} on line 56
     Gareth Rushgrove
     

     View Slide

114. puppet-syntax
     Gareth Rushgrove
     

     View Slide

115. Validate Puppet
     and ERB syntax
     

     View Slide

116. require 'puppet-syntax/tasks/puppet-syntax'
     Gareth Rushgrove
     

     View Slide

117. bundle exec rake syntax
     ---> syntax:manifests
     ---> syntax:templates
     Gareth Rushgrove
     

     View Slide

118. rspec-puppet
     Gareth Rushgrove
     

     View Slide

119. Unit testing
     for Puppet
     

     View Slide

120. context "epel enabled" do
     let(:params) {{ :epel_enable => true }}
     it { should contain_class('epel') }
     end
     Gareth Rushgrove
     

     View Slide

121. context "epel disabled" do
     let(:params) {{ :epel_enable => false }}
     it { should_not contain_class('epel') }
     end
     Gareth Rushgrove
     

     View Slide

122. Fixtures,
     matchers
     

     View Slide

123. Test the interface not
     the implementation
     Gareth Rushgrove
     

     View Slide

124. All of the above
     Gareth Rushgrove
     

     View Slide

125. task :test => [
     :syntax,
     :lint,
     :spec,
     ]
     Gareth Rushgrove
     

     View Slide

126. bundle exec rake test
     Gareth Rushgrove
     

     View Slide

127. Gareth Rushgrove
     

     View Slide

128. Nice continuous
     integration
     

     View Slide

129. View Slide

130. Test pull request
     branches too
     

     View Slide

131. ---
     language: ruby
     before_install: rm Gemfile.lock || true
     rvm:
     - 1.8.7
     - 1.9.3
     script: bundle exec rake test
     env:
     matrix:
     - PUPPET_VERSION="~> 2.7.0"
     - PUPPET_VERSION="~> 3.1.0"
     - PUPPET_VERSION="~> 3.2.0"
     Gareth Rushgrove
     

     View Slide

132. A matrix of
     possibilities
     Gareth Rushgrove
     

     View Slide

133. rspec-system
     Gareth Rushgrove
     

     View Slide

134. Integration
     test against
     real systems
     

     View Slide

135. Puppet
     helpers for
     rspec-system
     

     View Slide

136. Vagrant
     driver
     

     View Slide

137. VSphere
     provider
     

     View Slide

138. Test that Puppet runs
     without errors
     Gareth Rushgrove
     

     View Slide

139. it 'should run without errors' do
     pp = "class { 'sample': }"
     puppet_apply(pp) do |r|
     r.exit_code.should == 2
     end
     end
     Gareth Rushgrove
     

     View Slide

140. Test runs are
     idempotent
     Gareth Rushgrove
     

     View Slide

141. it 'should run without errors' do
     pp = "class { 'sample': }"
     puppet_apply(pp) do |r|
     r.exit_code.should == 2
     r.refresh
     r.exit_code.should be_zero
     end
     end
     Gareth Rushgrove
     

     View Slide

142. Test that the module
     installs relevant
     binaries
     Gareth Rushgrove
     

     View Slide

143. it 'should install the erl binary' do
     shell 'which erl' do |r|
     r.stdout.should =~ /\/usr\/bin\/e
     r.stderr.should be_empty
     r.exit_code.should be_zero
     end
     end
     Gareth Rushgrove
     

     View Slide

144. Test against different
     operating systems
     Gareth Rushgrove
     

     View Slide

145. default_set: 'centos-64-x64'
     sets:
     'centos-64-x64':
     nodes:
     "main.foo.vm":
     prefab: 'centos-64-x64'
     'fedora-18-x64':
     nodes:
     "main.foo.vm":
     prefab: 'fedora-18-x64'
     'debian-607-x64':
     nodes:
     "main.foo.vm":
     Gareth Rushgrove
     

     View Slide

146. Room for
     improvements
     (Making reuse easier in Puppet)
     

     View Slide

147. Run your own Forge
     1
     

     View Slide

148. A more secure Forge
     2
     

     View Slide

149. Bless a dependency
     management tool
     3
     

     View Slide

150. Optional
     dependencies
     4
     

     View Slide

151. dependency 'puppetlabs/stdlib'
     optional_dependency 'puppetlabs/apache'
     Gareth Rushgrove
     

     View Slide

152. Private classes
     5
     

     View Slide

153. private class elixir::install {
     Gareth Rushgrove
     

     View Slide

154. Parameter naming
     conventions
     6
     

     View Slide

155. example42
     Proposal
     

     View Slide

156. Interfaces in Puppet
     7
     

     View Slide

157. interface packageandservice {
     $version
     $enable
     $package_name
     $template
     }
     Gareth Rushgrove
     

     View Slide

158. class diamond (
     $version,
     $enable,
     $package_name,
     $template,
     ) {
     implements packageandservice
     Gareth Rushgrove
     

     View Slide

159. Questions?
     (And thanks for listening)
     

     View Slide