Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet Module Reusability

Puppet Module Reusability

Presentation from PuppetConf 2013 in San Francisco.

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

Gareth Rushgrove

August 23, 2013
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Puppet Module
    Reusability
    Learning from shipping to the Forge
    Gareth Rushgrove

    View Slide

  2. Who
    (Who is this person?)

    View Slide

  3. @garethr

    View Slide

  4. UK Government
    Digital Service

    View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. Last code
    I wrote

    View Slide

  9. View Slide

  10. View Slide

  11. 7. Gareth Rushgrove

    View Slide

  12. Last puppet
    module I wrote

    View Slide

  13. The Problem
    (Sharing modules is hard)

    View Slide

  14. Not invented here
    syndrome
    1

    View Slide

  15. A search for
    puppet-apache

    View Slide

  16. 145
    Gareth Rushgrove

    View Slide

  17. 145
    Gareth Rushgrove

    View Slide

  18. Puppetlabs
    apache

    View Slide

  19. 67
    contributors
    Gareth Rushgrove

    View Slide

  20. 281
    forks
    Gareth Rushgrove

    View Slide

  21. 281
    Gareth Rushgrove

    View Slide

  22. Vendor everything
    pattern
    2

    View Slide

  23. Not publishing to the
    Forge
    3

    View Slide

  24. Puppet
    Forge

    View Slide

  25. Undocumented
    modules
    4

    View Slide

  26. Official docs
    guidelines

    View Slide

  27. (Too) opinionated
    configuration
    5

    View Slide

  28. Sometimes I just
    want to write nginx
    configuration
    Gareth Rushgrove

    View Slide

  29. Duplicate resources
    6

    View Slide

  30. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove

    View Slide

  31. Stdlib
    (Start here)

    View Slide

  32. Standard
    Library from
    Puppetlabs

    View Slide

  33. Fail fast
    Gareth Rushgrove

    View Slide

  34. Validation
    Gareth Rushgrove

    View Slide

  35. Useful error
    messages
    Gareth Rushgrove

    View Slide

  36. fail("${::operatingsystem}
    not supported")
    Gareth Rushgrove

    View Slide

  37. validate_string
    Gareth Rushgrove

    View Slide

  38. validate_string($version)
    Gareth Rushgrove

    View Slide

  39. not a string.
    Gareth Rushgrove

    View Slide

  40. validate_slength
    Gareth Rushgrove

    View Slide

  41. validate_re
    Gareth Rushgrove

    View Slide

  42. validate_hash
    Gareth Rushgrove

    View Slide

  43. Gareth Rushgrove
    validate_cmd

    View Slide

  44. validate_bool
    Gareth Rushgrove

    View Slide

  45. Avoid duplicate
    resources
    Gareth Rushgrove

    View Slide

  46. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove

    View Slide

  47. include 'gcc'
    Gareth Rushgrove

    View Slide

  48. package{[
    'python-pip',
    'python-ldap',
    ]:
    ensure => installed
    }
    Gareth Rushgrove

    View Slide

  49. ensure_packages
    Gareth Rushgrove

    View Slide

  50. ensure_packages([
    'python-pip',
    'python-ldap',
    ])
    Gareth Rushgrove

    View Slide

  51. ensure_resource
    Gareth Rushgrove

    View Slide

  52. Nicer interfaces
    Gareth Rushgrove

    View Slide

  53. str2bool
    Gareth Rushgrove

    View Slide

  54. any2array
    Gareth Rushgrove

    View Slide

  55. And much more
    Gareth Rushgrove

    View Slide

  56. Dependency
    management is
    everywhere
    (Else)

    View Slide

  57. NPM, Bundler, Pip,
    Mix, Leiningen
    Gareth Rushgrove

    View Slide

  58. Librarian
    Puppet

    View Slide

  59. r10k

    View Slide

  60. echo 'modules' >> .gitignore
    Gareth Rushgrove

    View Slide

  61. dependency 'puppetlabs/stdlib'
    dependency 'garethr/erlang'
    dependency 'maestrodev/wget'
    Gareth Rushgrove

    View Slide

  62. forge "http://forge.puppetlabs.com"
    mod 'puppetlabs/ruby'
    mod 'puppetlabs/ntp'
    mod 'puppetlabs/git'
    mod 'puppetlabs/vcsrepo'
    mod 'puppetlabs/apt'
    mod 'puppetlabs/gcc'
    Gareth Rushgrove

    View Slide

  63. librarian-puppet install
    Gareth Rushgrove

    View Slide

  64. A Nice
    Module Pattern
    (Structuring modules for reuse)

    View Slide

  65. R.I.Pienaar

    View Slide

  66. install, config,
    service, params
    Gareth Rushgrove

    View Slide

  67. manifests
    !"" config.pp
    !"" init.pp
    !"" install.pp
    !"" params.pp
    #"" service.pp
    Gareth Rushgrove

    View Slide

  68. class sample (
    ) inherits sample::params {
    class { 'sample::install': } ->
    class { 'sample::config': } ~>
    class { 'sample::service': } ->
    Class['sample']
    }
    Gareth Rushgrove

    View Slide

  69. anchor { 'sample::begin': } ->
    class { 'sample::install': } ->
    class { 'sample::config': }
    class { 'sample::service': } ->
    anchor { 'sample::end': }
    Gareth Rushgrove

    View Slide

  70. Anchor['sample::begin'] ~>
    Class['sample::service']
    Class['sample::install'] ~>
    Class['sample::service']
    Class['sample::config'] ~>
    Class['sample::service']
    Gareth Rushgrove

    View Slide

  71. Puppet module tool
    Gareth Rushgrove

    View Slide

  72. puppet module generate name
    Gareth Rushgrove

    View Slide

  73. Default module
    skeleton
    Gareth Rushgrove

    View Slide

  74. In puppet
    source code

    View Slide

  75. !"" manifests
    !"" spec
    !"" tests
    !"" Modulefile
    !"" .gitignore
    #"" README
    Gareth Rushgrove

    View Slide

  76. Creating your own
    module skeleton
    Gareth Rushgrove

    View Slide

  77. ~/.puppet/var/puppet-module/skeleton
    Gareth Rushgrove

    View Slide

  78. puppet module
    skeleton

    View Slide

  79. !"" manifests
    !"" spec
    !"" tests
    !"" templates
    !"" files
    !"" lib
    !"" Gemfile
    Gareth Rushgrove

    View Slide

  80. !"" Rakefile
    !"" .nodeset.yml
    !"" .fixtures.yml
    !"" .travis.yml
    !"" Modulefile
    !"" .gitignore
    #"" README.md
    Gareth Rushgrove

    View Slide

  81. Creates install,
    config, service,
    params classes
    Gareth Rushgrove

    View Slide

  82. Dependency
    management with
    Bundler
    Gareth Rushgrove

    View Slide

  83. Better unit testing
    setup using rspec-
    puppet-helper
    Gareth Rushgrove

    View Slide

  84. Adds syntax and lint
    checking
    Gareth Rushgrove

    View Slide

  85. Adds Travis CI
    configuration
    Gareth Rushgrove

    View Slide

  86. Adds integration tests
    with rspec-system
    Gareth Rushgrove

    View Slide

  87. Adds module
    management with
    maestrodev
    blacksmith
    Gareth Rushgrove

    View Slide

  88. Focus on the
    interface
    Gareth Rushgrove

    View Slide

  89. Minimize number of
    entry points
    Gareth Rushgrove

    View Slide

  90. Allow overriding
    provided templates
    Gareth Rushgrove

    View Slide

  91. Multi-OS Support
    (Where to abstract the differences)

    View Slide

  92. Vary default
    parameters
    Gareth Rushgrove

    View Slide

  93. Keep logic in one
    place
    Gareth Rushgrove

    View Slide

  94. case $::osfamily {
    'Debian': {
    }
    'RedHat', 'Amazon': {
    }
    default: {
    fail("${::operatingsystem} not su
    }
    Gareth Rushgrove

    View Slide

  95. ARM-9 Data
    in Modules

    View Slide

  96. garethr-erlang

    View Slide

  97. 0.0.x supported
    Ubuntu only
    Gareth Rushgrove

    View Slide

  98. 0.1.x supports
    Ubuntu, Debian,
    Redhat, Suse
    Gareth Rushgrove

    View Slide

  99. Gareth Rushgrove

    View Slide

  100. Pull requests to
    the rescue

    View Slide

  101. View Slide

  102. Insist on tests with
    open source
    contributions
    Gareth Rushgrove

    View Slide

  103. Module Testing
    (Code should have tests)

    View Slide

  104. Why test a
    declarative
    language?
    Gareth Rushgrove

    View Slide

  105. Modules increasingly
    contain logic
    Gareth Rushgrove

    View Slide

  106. Modules increasingly
    take arguments
    Gareth Rushgrove

    View Slide

  107. Modules increasingly
    have interfaces with
    other modules
    Gareth Rushgrove

    View Slide

  108. Modules increasingly
    used in many Ruby
    and Puppet version
    combinations
    Gareth Rushgrove

    View Slide

  109. Good news. The tools
    got better
    Gareth Rushgrove

    View Slide

  110. puppet-lint
    Gareth Rushgrove

    View Slide

  111. Puppet
    style guide

    View Slide

  112. Available
    as a gem

    View Slide

  113. puppet-lint --with-filename /etc/puppet/modules
    foo/manifests/bar.pp: trailing whitespace found
    on line 1
    apache/manifests/server.pp: variable not enclosed
    in {} on line 56
    Gareth Rushgrove

    View Slide

  114. puppet-syntax
    Gareth Rushgrove

    View Slide

  115. Validate Puppet
    and ERB syntax

    View Slide

  116. require 'puppet-syntax/tasks/puppet-syntax'
    Gareth Rushgrove

    View Slide

  117. bundle exec rake syntax
    ---> syntax:manifests
    ---> syntax:templates
    Gareth Rushgrove

    View Slide

  118. rspec-puppet
    Gareth Rushgrove

    View Slide

  119. Unit testing
    for Puppet

    View Slide

  120. context "epel enabled" do
    let(:params) {{ :epel_enable => true }}
    it { should contain_class('epel') }
    end
    Gareth Rushgrove

    View Slide

  121. context "epel disabled" do
    let(:params) {{ :epel_enable => false }}
    it { should_not contain_class('epel') }
    end
    Gareth Rushgrove

    View Slide

  122. Fixtures,
    matchers

    View Slide

  123. Test the interface not
    the implementation
    Gareth Rushgrove

    View Slide

  124. All of the above
    Gareth Rushgrove

    View Slide

  125. task :test => [
    :syntax,
    :lint,
    :spec,
    ]
    Gareth Rushgrove

    View Slide

  126. bundle exec rake test
    Gareth Rushgrove

    View Slide

  127. Gareth Rushgrove

    View Slide

  128. Nice continuous
    integration

    View Slide

  129. View Slide

  130. Test pull request
    branches too

    View Slide

  131. ---
    language: ruby
    before_install: rm Gemfile.lock || true
    rvm:
    - 1.8.7
    - 1.9.3
    script: bundle exec rake test
    env:
    matrix:
    - PUPPET_VERSION="~> 2.7.0"
    - PUPPET_VERSION="~> 3.1.0"
    - PUPPET_VERSION="~> 3.2.0"
    Gareth Rushgrove

    View Slide

  132. A matrix of
    possibilities
    Gareth Rushgrove

    View Slide

  133. rspec-system
    Gareth Rushgrove

    View Slide

  134. Integration
    test against
    real systems

    View Slide

  135. Puppet
    helpers for
    rspec-system

    View Slide

  136. Vagrant
    driver

    View Slide

  137. VSphere
    provider

    View Slide

  138. Test that Puppet runs
    without errors
    Gareth Rushgrove

    View Slide

  139. it 'should run without errors' do
    pp = "class { 'sample': }"
    puppet_apply(pp) do |r|
    r.exit_code.should == 2
    end
    end
    Gareth Rushgrove

    View Slide

  140. Test runs are
    idempotent
    Gareth Rushgrove

    View Slide

  141. it 'should run without errors' do
    pp = "class { 'sample': }"
    puppet_apply(pp) do |r|
    r.exit_code.should == 2
    r.refresh
    r.exit_code.should be_zero
    end
    end
    Gareth Rushgrove

    View Slide

  142. Test that the module
    installs relevant
    binaries
    Gareth Rushgrove

    View Slide

  143. it 'should install the erl binary' do
    shell 'which erl' do |r|
    r.stdout.should =~ /\/usr\/bin\/e
    r.stderr.should be_empty
    r.exit_code.should be_zero
    end
    end
    Gareth Rushgrove

    View Slide

  144. Test against different
    operating systems
    Gareth Rushgrove

    View Slide

  145. default_set: 'centos-64-x64'
    sets:
    'centos-64-x64':
    nodes:
    "main.foo.vm":
    prefab: 'centos-64-x64'
    'fedora-18-x64':
    nodes:
    "main.foo.vm":
    prefab: 'fedora-18-x64'
    'debian-607-x64':
    nodes:
    "main.foo.vm":
    Gareth Rushgrove

    View Slide

  146. Room for
    improvements
    (Making reuse easier in Puppet)

    View Slide

  147. Run your own Forge
    1

    View Slide

  148. A more secure Forge
    2

    View Slide

  149. Bless a dependency
    management tool
    3

    View Slide

  150. Optional
    dependencies
    4

    View Slide

  151. dependency 'puppetlabs/stdlib'
    optional_dependency 'puppetlabs/apache'
    Gareth Rushgrove

    View Slide

  152. Private classes
    5

    View Slide

  153. private class elixir::install {
    Gareth Rushgrove

    View Slide

  154. Parameter naming
    conventions
    6

    View Slide

  155. example42
    Proposal

    View Slide

  156. Interfaces in Puppet
    7

    View Slide

  157. interface packageandservice {
    $version
    $enable
    $package_name
    $template
    }
    Gareth Rushgrove

    View Slide

  158. class diamond (
    $version,
    $enable,
    $package_name,
    $template,
    ) {
    implements packageandservice
    Gareth Rushgrove

    View Slide

  159. Questions?
    (And thanks for listening)

    View Slide