Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet Modules: Apps for Ops

Puppet Modules: Apps for Ops

B609a962940cfae86652a4f1bad73275?s=128

Justin Bronn

April 11, 2014
Tweet

Transcript

  1. Puppet Modules Apps for Ops! ! Justin Bronn, Esq. @jbronn

  2. Slides / Notes https://github.com/jbronn

  3. Configuration Management?

  4. Complexity

  5. <HTML>! <IMG> /cgi-bin/perl.cgi

  6. None
  7. None
  8. mod_wsgi

  9. Don’t Repeat Yourself • Even a “simple” web application has

    a large number of components • You don’t want to configure them manually
  10. Why Puppet?

  11. None
  12. crypt.py

  13. None
  14. Why Puppet? • Mature (in Licensing & Security) • “Explicit

    is better than Implicit” • Wide OS support: • Linux (of course), Windows, Mac • OpenBSD, Solaris, AIX,HP-UX
  15. Installing Puppet

  16. Ubuntu 12.04 $ sudo gem install \! --bindir /usr/local/bin! puppet

    $ sudo apt-get -y install \! libruby libshadow-ruby1.8 rubygems Packages also available: apt.puppetlabs.com
  17. OS X $ sudo gem install \! --bindir /usr/local/bin! puppet

    Package also available: https://downloads.puppetlabs.com/mac/
  18. Windows C:\> msiexec /qn /i puppet-3.4.3.msi ^! PUPPET_AGENT_STARTUP_MODE=Disabled! MSI from:

    https://downloads.puppetlabs.com/windows/
  19. Puppet Language

  20. Overview • Puppet, the language, is a Ruby DSL •

    Puppet code lives in files called “manifests”; files end with “.pp” suffix
  21. Overview • Catalog: compiled Puppet code, describes relationships between “resources”

    • Catalog is a an directed acyclic graph • Puppet applies catalog, bringing system into state declared in manifests
  22. Resources

  23. package {'openssh-server':! ensure => installed,! }


  24. package {'openssh-server':! ensure => installed,! }


  25. package {'openssh-server':! ensure => installed,! }


  26. package {'openssh-server':! ensure => installed,! }


  27. package {'openssh-server':! ensure => absent,! }


  28. package {'Django':! ensure => '1.6.2',! provider => 'pip',! }


  29. Basics

  30. Declarative

  31. $ssh_dir = '/etc/ssh'! file { $ssh_dir:! ensure => directory,! owner

    => 'root',! group => 'root',! mode => '0755',! }! $ssh_dir = '/foo/ssh'!
  32. $ssh_dir = '/etc/ssh'! file { $ssh_dir:! ensure => directory,! owner

    => 'root',! group => 'root',! mode => '0644',! }!
  33. $array = ['foo', 'bar']! ! $hash = {'key' => 'value'}!

    $value = $hash['key']! ! $py_versions = split('2.7,3.4', ',')! Data Structures / Function Call
  34. $noun = 'Lumberjack'! $sentence = "I'm a ${noun}"! String Interpolation

  35. if $python_version == '3.4' {! notice('You have asyncio!')! } !

    ! case $python_version {! /^(2.6|2.7)$/: {! }! '3.4': {! }! default: {! fail('Unsupported version!')! }! }! Flow Control
  36. Facts

  37. $ facter | less! ! $ facter fqdn ipaddress osfamily!

    fqdn => puppet.local! ipaddress => 10.0.2.15! osfamily => Debian!
  38. if $::osfamily == 'RedHat' {! $py_dev_pkg = 'python-devel'! } elsif

    $::osfamily == 'Debian'{! $py_dev_pkg = 'python-dev'! }! ! ! $:: gives you facts in Puppet
  39. if $::osfamily == 'RedHat' {! $py_dev_pkg = 'python-devel'! } elsif

    $::osfamily == 'Debian'{! $py_dev_pkg = 'python-dev'! }! ! ! $:: gives you facts in Puppet
  40. Resource Ordering

  41. Resource References Package['openssh-server'] Resource type is capitalized Resource name is

    in brackets
  42. $sshd_config = "${ssh_dir}/sshd_config"! ! file {$sshd_config:! ensure => file,! owner

    => 'root',! group => 'root',! mode => '0644',! content => template('sshd_config.erb'),! require => Package['openssh-server'],! }
 Require Metaparameter
  43. $sshd_config = "${ssh_dir}/sshd_config"! ! file {$sshd_config:! ensure => file,! owner

    => 'root',! group => 'root',! mode => '0644',! content => template('sshd_config.erb'),! require => Package['openssh-server'],! }
 Require Metaparameter
  44. Package[openssh-server] File[/etc/ssh/sshd_config]

  45. service {'ssh':! ensure => running,! enable => true,! subscribe =>

    File[$sshd_config],! }
 Subscribe Metaparameter
  46. service {'ssh':! ensure => running,! enable => true,! subscribe =>

    File[$sshd_config],! }
 Subscribe Metaparameter
  47. file { $sshd_config:! ...! notify => Service['ssh'],! ...! }
 service

    {'ssh':! ensure => running,! enable => true,! } Notify Metaparameter
  48. file { $sshd_config:! ...! notify => Service['ssh'],! ...! }
 service

    {'ssh':! ensure => running,! enable => true,! } Notify Metaparameter
  49. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows

  50. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows

  51. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows

  52. Puppet Modules

  53. Modules • Containers for Puppet code • Analogous to Python

    Packages • May be installed from a Puppet “Forge”
  54. Installing Modules

  55. puppet module install <vendor>-<name>

  56. $ puppet module install counsyl-python Notice: Preparing to install into

    /Users/justin/.puppet/modules ...! Notice: Downloading from https://forge.puppetlabs.com ...! Notice: Installing -- do not interrupt ...! /Users/justin/.puppet/modules! !"# counsyl-python (v0.9.3)! !"# counsyl-sys (v0.9.13)! !"" puppetlabs-stdlib (v4.1.0)! !
  57. https://forge.puppetlabs.com https://github.com/jbronn/django-forge

  58. Module Structure

  59. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/ $ puppet module generate

    \! counsyl-pyapp
  60. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/

  61. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/

  62. manifests/init.pp class pyapp {! ...! }
 “class” name must match

    name of module
  63. manifests/install.pp class pyapp::install {! ...! }
 Can have other namespaces,

    but file name must match name of file.
  64. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/

  65. file { ‘/path/to/settings.py':! ensure => file,! owner => 'root',! group

    => 'root',! mode => '0644',! content => template('pyapp/settings.py.erb'),! } Puppet Templates
  66. file { ‘/path/to/settings.py':! ensure => file,! owner => 'root',! group

    => 'root',! mode => '0644',! content => template('pyapp/settings.py.erb'),! } Puppet Templates pyapp/templates/settings.py.erb
  67. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/

  68. file { '/path/to/foo.js':! ensure => file,! owner => 'root',! group

    => 'root',! mode => '0644',! source => 'puppet:///modules/pyapp/foo.js'! } Puppet Files
  69. file { '/path/to/foo.js':! ensure => file,! owner => 'root',! group

    => 'root',! mode => '0644',! source => 'puppet:///modules/pyapp/foo.js'! } Puppet Files pyapp/files/foo.js
  70. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/

  71. Writing Modules for Python

  72. Roadmap • “Simple” Django application • Needs Celery (and Redis)

    • Nginx reverse-proxy to Gunicorn
  73. class pyapp {! include python! }
 Python / Pip

  74. class pyapp {! include python! include python::virtualenv! }
 Virtualenv counsyl-python

  75. class pyapp {! include python! include python::virtualenv! include redis! }


    Redis counsyl-redis
  76. class pyapp {! include python! include python::virtualenv! include redis! include

    nginx! }
 Nginx counsyl-nginx
  77. Mix & Match

  78. Python Types

  79. package {'pyapp':! ensure => '0.1.0',! source => 'git+https://github.com …’, provider

    => 'pip',! }
 pip provider
  80. package {'pyapp':! ensure => '0.1.0',! source => 'git+https://github.com …’, provider

    => 'pip',! }
 pip provider
  81. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options =>

    [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
  82. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options =>

    [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
  83. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options =>

    [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
  84. venv {'/srv/pyapp': } ! ! ! ! venv type

  85. venv {'/srv/pyapp': ! owner => $user,! group => $group,! system_site_packages

    => true,! }! venv type
  86. venv_package {'celery@/srv/pyapp': ! ensure => installed,! }! venv_package type

  87. $venv = '/srv/pyapp'! venv_package {"celery@${venv}": ! ensure => installed,! }!

    venv_package type
  88. $venv = '/srv/pyapp'! venv_package {"redis@${venv}": ! ensure => '2.9.1',! require

    => Class['redis'],! }! venv_package type
  89. Class Parameters

  90. class pyapp(! $venv = '/srv/pyapp',! $package = 'pyapp',! $source =

    'git+https://...',! $version = '0.1.0',! ){! ...! venv { $venv: }! venv_package { "${package}@${venv}":! ensure => $version,! source => $source,! }! ...! }
  91. class pyapp(! $venv = '/srv/pyapp'! $package = 'pyapp',! $source =

    'git+https://...' ! $version = '0.1.0',! ){! ...! venv { $venv: }! venv_package { "${package}@${venv}":! ensure => $version,! source => $source,! }! ...! }
  92. class {'pyapp':! version => '0.8',! } Using Class Parameters

  93. class {'pyapp':! version => '0.8',! } Using Class Parameters

  94. class {'pyapp':! version => '0.8',! } Using Class Parameters

  95. ERB

  96. ERB: Ruby Templates • This time, it’s Ruby • Great

    for when you need dynamic content in a configuration file (e.g., ServerName)
  97. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG

    = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
  98. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG

    = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
  99. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG

    = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
  100. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG

    = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
  101. Running Puppet

  102. Stand-Alone

  103. puppet apply manifest.pp

  104. puppet apply site.pp \! --modulepath /modules Customize Module Path

  105. puppet apply -e \! "include pyapp" “Evaluate”

  106. puppet agent

  107. The Puppetmaster • Centralized server for housing modules and configurations

    for nodes (servers) • Secured with HTTPS Client Certificates • Nodes must have certificate signed to receive configuration
  108. Node Master

  109. Node Master node$ sudo puppet agent —test! Info: Creating a

    new SSL key for node.local! Exiting; no certificate found
  110. Node Master master$ sudo puppet cert sign node.local! Info: Signed

    certificate request for node.local
  111. Node Master node$ sudo puppet agent —test! info: Caching certificate

    for node.local! info: Retrieving plugin! info: Caching catalog for node.local! info: Applying configuration version '1326210629'! notice: Finished catalog run in 0.11 seconds /etc/puppet/modules /etc/puppet/manifests
  112. None
  113. Module Development

  114. Use Vagrant!

  115. Vagrant • A must-have for DevOps development in general, not

    just Puppet • Also supports Ansible, Salt, Chef • You may think you can do better…
  116. Vagrantfile • Tells Vagrant what to do • An API

    for provisioning a VM (actually Ruby + magic) • Forward ports, change memory, multiple provisioners, multiple VMs, multiple NICs…
  117. Vagrantfile Vagrant.configure('2') do |config|! config.vm.box = "precise64"! config.vm.box_url = "http://..."!

    ...! end
  118. Puppet Provisioner Vagrant.configure('2') do |config|! ...! config.vm.provision "puppet" do |puppet|!

    puppet.facter = { "vagrant" => "1" }! puppet.module_path = "modules"! end! ...! end
  119. Puppet Provisioner Vagrant.configure('2') do |config|! ...! config.vm.provision "puppet" do |puppet|!

    puppet.facter = { "vagrant" => "1" }! puppet.module_path = "modules"! end! ...! end Looks for manifests/default.pp
  120. Another Ruby DSL? • Yeah… • Tremendous workflow benefits •

    Much easier to share your snowflake environments
  121. vagrant up • All end user has to do is

    type this command • No waiting for a server/sysadmin • Can get to work
  122. Base Boxes?

  123. Use Packer!

  124. Packer • Virtual Machine image creation tool • Written in

    Go, uses JSON templates • See my sample for creating a base box: • https://github.com/jbronn/packer-vagrant
  125. None
  126. Questions?