Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet Module Reusability

Puppet Module Reusability

Presentation from PuppetConf 2013 in San Francisco.

A simple search for "puppet-apache" on GitHub returns 70 separate repositories. An awful lot of people are busy reinventing the same configuration wheel. Configuration management tools promise write once, run anywhere code; but writing code that can be used by anyone looks like a lot of work. This presentation aims to show anyone familiar with Puppet how to write reusable modules and importantly how to make them compatible with already shared modules released on the Forge or elsewhere. We'll look at when and why testing a declarative language is actually useful, examples of good and bad modules and how to re-factor puppet code for re-usability. We'll also talk about potential improvements to Puppet that would make reuse easier.

Gareth Rushgrove

August 23, 2013
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Puppet Module
    Reusability
    Learning from shipping to the Forge
    Gareth Rushgrove

    View full-size slide

  2. Who
    (Who is this person?)

    View full-size slide

  3. UK Government
    Digital Service

    View full-size slide

  4. Last code
    I wrote

    View full-size slide

  5. 7. Gareth Rushgrove

    View full-size slide

  6. Last puppet
    module I wrote

    View full-size slide

  7. The Problem
    (Sharing modules is hard)

    View full-size slide

  8. Not invented here
    syndrome
    1

    View full-size slide

  9. A search for
    puppet-apache

    View full-size slide

  10. 145
    Gareth Rushgrove

    View full-size slide

  11. 145
    Gareth Rushgrove

    View full-size slide

  12. Puppetlabs
    apache

    View full-size slide

  13. 67
    contributors
    Gareth Rushgrove

    View full-size slide

  14. 281
    forks
    Gareth Rushgrove

    View full-size slide

  15. 281
    Gareth Rushgrove

    View full-size slide

  16. Vendor everything
    pattern
    2

    View full-size slide

  17. Not publishing to the
    Forge
    3

    View full-size slide

  18. Undocumented
    modules
    4

    View full-size slide

  19. Official docs
    guidelines

    View full-size slide

  20. (Too) opinionated
    configuration
    5

    View full-size slide

  21. Sometimes I just
    want to write nginx
    configuration
    Gareth Rushgrove

    View full-size slide

  22. Duplicate resources
    6

    View full-size slide

  23. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove

    View full-size slide

  24. Stdlib
    (Start here)

    View full-size slide

  25. Standard
    Library from
    Puppetlabs

    View full-size slide

  26. Fail fast
    Gareth Rushgrove

    View full-size slide

  27. Validation
    Gareth Rushgrove

    View full-size slide

  28. Useful error
    messages
    Gareth Rushgrove

    View full-size slide

  29. fail("${::operatingsystem}
    not supported")
    Gareth Rushgrove

    View full-size slide

  30. validate_string
    Gareth Rushgrove

    View full-size slide

  31. validate_string($version)
    Gareth Rushgrove

    View full-size slide

  32. not a string.
    Gareth Rushgrove

    View full-size slide

  33. validate_slength
    Gareth Rushgrove

    View full-size slide

  34. validate_re
    Gareth Rushgrove

    View full-size slide

  35. validate_hash
    Gareth Rushgrove

    View full-size slide

  36. Gareth Rushgrove
    validate_cmd

    View full-size slide

  37. validate_bool
    Gareth Rushgrove

    View full-size slide

  38. Avoid duplicate
    resources
    Gareth Rushgrove

    View full-size slide

  39. Package { 'build-essential’:
    ensure => installed,
    }
    Gareth Rushgrove

    View full-size slide

  40. include 'gcc'
    Gareth Rushgrove

    View full-size slide

  41. package{[
    'python-pip',
    'python-ldap',
    ]:
    ensure => installed
    }
    Gareth Rushgrove

    View full-size slide

  42. ensure_packages
    Gareth Rushgrove

    View full-size slide

  43. ensure_packages([
    'python-pip',
    'python-ldap',
    ])
    Gareth Rushgrove

    View full-size slide

  44. ensure_resource
    Gareth Rushgrove

    View full-size slide

  45. Nicer interfaces
    Gareth Rushgrove

    View full-size slide

  46. str2bool
    Gareth Rushgrove

    View full-size slide

  47. any2array
    Gareth Rushgrove

    View full-size slide

  48. And much more
    Gareth Rushgrove

    View full-size slide

  49. Dependency
    management is
    everywhere
    (Else)

    View full-size slide

  50. NPM, Bundler, Pip,
    Mix, Leiningen
    Gareth Rushgrove

    View full-size slide

  51. Librarian
    Puppet

    View full-size slide

  52. echo 'modules' >> .gitignore
    Gareth Rushgrove

    View full-size slide

  53. dependency 'puppetlabs/stdlib'
    dependency 'garethr/erlang'
    dependency 'maestrodev/wget'
    Gareth Rushgrove

    View full-size slide

  54. forge "http://forge.puppetlabs.com"
    mod 'puppetlabs/ruby'
    mod 'puppetlabs/ntp'
    mod 'puppetlabs/git'
    mod 'puppetlabs/vcsrepo'
    mod 'puppetlabs/apt'
    mod 'puppetlabs/gcc'
    Gareth Rushgrove

    View full-size slide

  55. librarian-puppet install
    Gareth Rushgrove

    View full-size slide

  56. A Nice
    Module Pattern
    (Structuring modules for reuse)

    View full-size slide

  57. install, config,
    service, params
    Gareth Rushgrove

    View full-size slide

  58. manifests
    !"" config.pp
    !"" init.pp
    !"" install.pp
    !"" params.pp
    #"" service.pp
    Gareth Rushgrove

    View full-size slide

  59. class sample (
    ) inherits sample::params {
    class { 'sample::install': } ->
    class { 'sample::config': } ~>
    class { 'sample::service': } ->
    Class['sample']
    }
    Gareth Rushgrove

    View full-size slide

  60. anchor { 'sample::begin': } ->
    class { 'sample::install': } ->
    class { 'sample::config': }
    class { 'sample::service': } ->
    anchor { 'sample::end': }
    Gareth Rushgrove

    View full-size slide

  61. Anchor['sample::begin'] ~>
    Class['sample::service']
    Class['sample::install'] ~>
    Class['sample::service']
    Class['sample::config'] ~>
    Class['sample::service']
    Gareth Rushgrove

    View full-size slide

  62. Puppet module tool
    Gareth Rushgrove

    View full-size slide

  63. puppet module generate name
    Gareth Rushgrove

    View full-size slide

  64. Default module
    skeleton
    Gareth Rushgrove

    View full-size slide

  65. In puppet
    source code

    View full-size slide

  66. !"" manifests
    !"" spec
    !"" tests
    !"" Modulefile
    !"" .gitignore
    #"" README
    Gareth Rushgrove

    View full-size slide

  67. Creating your own
    module skeleton
    Gareth Rushgrove

    View full-size slide

  68. ~/.puppet/var/puppet-module/skeleton
    Gareth Rushgrove

    View full-size slide

  69. puppet module
    skeleton

    View full-size slide

  70. !"" manifests
    !"" spec
    !"" tests
    !"" templates
    !"" files
    !"" lib
    !"" Gemfile
    Gareth Rushgrove

    View full-size slide

  71. !"" Rakefile
    !"" .nodeset.yml
    !"" .fixtures.yml
    !"" .travis.yml
    !"" Modulefile
    !"" .gitignore
    #"" README.md
    Gareth Rushgrove

    View full-size slide

  72. Creates install,
    config, service,
    params classes
    Gareth Rushgrove

    View full-size slide

  73. Dependency
    management with
    Bundler
    Gareth Rushgrove

    View full-size slide

  74. Better unit testing
    setup using rspec-
    puppet-helper
    Gareth Rushgrove

    View full-size slide

  75. Adds syntax and lint
    checking
    Gareth Rushgrove

    View full-size slide

  76. Adds Travis CI
    configuration
    Gareth Rushgrove

    View full-size slide

  77. Adds integration tests
    with rspec-system
    Gareth Rushgrove

    View full-size slide

  78. Adds module
    management with
    maestrodev
    blacksmith
    Gareth Rushgrove

    View full-size slide

  79. Focus on the
    interface
    Gareth Rushgrove

    View full-size slide

  80. Minimize number of
    entry points
    Gareth Rushgrove

    View full-size slide

  81. Allow overriding
    provided templates
    Gareth Rushgrove

    View full-size slide

  82. Multi-OS Support
    (Where to abstract the differences)

    View full-size slide

  83. Vary default
    parameters
    Gareth Rushgrove

    View full-size slide

  84. Keep logic in one
    place
    Gareth Rushgrove

    View full-size slide

  85. case $::osfamily {
    'Debian': {
    }
    'RedHat', 'Amazon': {
    }
    default: {
    fail("${::operatingsystem} not su
    }
    Gareth Rushgrove

    View full-size slide

  86. ARM-9 Data
    in Modules

    View full-size slide

  87. garethr-erlang

    View full-size slide

  88. 0.0.x supported
    Ubuntu only
    Gareth Rushgrove

    View full-size slide

  89. 0.1.x supports
    Ubuntu, Debian,
    Redhat, Suse
    Gareth Rushgrove

    View full-size slide

  90. Gareth Rushgrove

    View full-size slide

  91. Pull requests to
    the rescue

    View full-size slide

  92. Insist on tests with
    open source
    contributions
    Gareth Rushgrove

    View full-size slide

  93. Module Testing
    (Code should have tests)

    View full-size slide

  94. Why test a
    declarative
    language?
    Gareth Rushgrove

    View full-size slide

  95. Modules increasingly
    contain logic
    Gareth Rushgrove

    View full-size slide

  96. Modules increasingly
    take arguments
    Gareth Rushgrove

    View full-size slide

  97. Modules increasingly
    have interfaces with
    other modules
    Gareth Rushgrove

    View full-size slide

  98. Modules increasingly
    used in many Ruby
    and Puppet version
    combinations
    Gareth Rushgrove

    View full-size slide

  99. Good news. The tools
    got better
    Gareth Rushgrove

    View full-size slide

  100. puppet-lint
    Gareth Rushgrove

    View full-size slide

  101. Puppet
    style guide

    View full-size slide

  102. Available
    as a gem

    View full-size slide

  103. puppet-lint --with-filename /etc/puppet/modules
    foo/manifests/bar.pp: trailing whitespace found
    on line 1
    apache/manifests/server.pp: variable not enclosed
    in {} on line 56
    Gareth Rushgrove

    View full-size slide

  104. puppet-syntax
    Gareth Rushgrove

    View full-size slide

  105. Validate Puppet
    and ERB syntax

    View full-size slide

  106. require 'puppet-syntax/tasks/puppet-syntax'
    Gareth Rushgrove

    View full-size slide

  107. bundle exec rake syntax
    ---> syntax:manifests
    ---> syntax:templates
    Gareth Rushgrove

    View full-size slide

  108. rspec-puppet
    Gareth Rushgrove

    View full-size slide

  109. Unit testing
    for Puppet

    View full-size slide

  110. context "epel enabled" do
    let(:params) {{ :epel_enable => true }}
    it { should contain_class('epel') }
    end
    Gareth Rushgrove

    View full-size slide

  111. context "epel disabled" do
    let(:params) {{ :epel_enable => false }}
    it { should_not contain_class('epel') }
    end
    Gareth Rushgrove

    View full-size slide

  112. Fixtures,
    matchers

    View full-size slide

  113. Test the interface not
    the implementation
    Gareth Rushgrove

    View full-size slide

  114. All of the above
    Gareth Rushgrove

    View full-size slide

  115. task :test => [
    :syntax,
    :lint,
    :spec,
    ]
    Gareth Rushgrove

    View full-size slide

  116. bundle exec rake test
    Gareth Rushgrove

    View full-size slide

  117. Gareth Rushgrove

    View full-size slide

  118. Nice continuous
    integration

    View full-size slide

  119. Test pull request
    branches too

    View full-size slide

  120. ---
    language: ruby
    before_install: rm Gemfile.lock || true
    rvm:
    - 1.8.7
    - 1.9.3
    script: bundle exec rake test
    env:
    matrix:
    - PUPPET_VERSION="~> 2.7.0"
    - PUPPET_VERSION="~> 3.1.0"
    - PUPPET_VERSION="~> 3.2.0"
    Gareth Rushgrove

    View full-size slide

  121. A matrix of
    possibilities
    Gareth Rushgrove

    View full-size slide

  122. rspec-system
    Gareth Rushgrove

    View full-size slide

  123. Integration
    test against
    real systems

    View full-size slide

  124. Puppet
    helpers for
    rspec-system

    View full-size slide

  125. Vagrant
    driver

    View full-size slide

  126. VSphere
    provider

    View full-size slide

  127. Test that Puppet runs
    without errors
    Gareth Rushgrove

    View full-size slide

  128. it 'should run without errors' do
    pp = "class { 'sample': }"
    puppet_apply(pp) do |r|
    r.exit_code.should == 2
    end
    end
    Gareth Rushgrove

    View full-size slide

  129. Test runs are
    idempotent
    Gareth Rushgrove

    View full-size slide

  130. it 'should run without errors' do
    pp = "class { 'sample': }"
    puppet_apply(pp) do |r|
    r.exit_code.should == 2
    r.refresh
    r.exit_code.should be_zero
    end
    end
    Gareth Rushgrove

    View full-size slide

  131. Test that the module
    installs relevant
    binaries
    Gareth Rushgrove

    View full-size slide

  132. it 'should install the erl binary' do
    shell 'which erl' do |r|
    r.stdout.should =~ /\/usr\/bin\/e
    r.stderr.should be_empty
    r.exit_code.should be_zero
    end
    end
    Gareth Rushgrove

    View full-size slide

  133. Test against different
    operating systems
    Gareth Rushgrove

    View full-size slide

  134. default_set: 'centos-64-x64'
    sets:
    'centos-64-x64':
    nodes:
    "main.foo.vm":
    prefab: 'centos-64-x64'
    'fedora-18-x64':
    nodes:
    "main.foo.vm":
    prefab: 'fedora-18-x64'
    'debian-607-x64':
    nodes:
    "main.foo.vm":
    Gareth Rushgrove

    View full-size slide

  135. Room for
    improvements
    (Making reuse easier in Puppet)

    View full-size slide

  136. Run your own Forge
    1

    View full-size slide

  137. A more secure Forge
    2

    View full-size slide

  138. Bless a dependency
    management tool
    3

    View full-size slide

  139. Optional
    dependencies
    4

    View full-size slide

  140. dependency 'puppetlabs/stdlib'
    optional_dependency 'puppetlabs/apache'
    Gareth Rushgrove

    View full-size slide

  141. Private classes
    5

    View full-size slide

  142. private class elixir::install {
    Gareth Rushgrove

    View full-size slide

  143. Parameter naming
    conventions
    6

    View full-size slide

  144. example42
    Proposal

    View full-size slide

  145. Interfaces in Puppet
    7

    View full-size slide

  146. interface packageandservice {
    $version
    $enable
    $package_name
    $template
    }
    Gareth Rushgrove

    View full-size slide

  147. class diamond (
    $version,
    $enable,
    $package_name,
    $template,
    ) {
    implements packageandservice
    Gareth Rushgrove

    View full-size slide

  148. Questions?
    (And thanks for listening)

    View full-size slide