Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Workshop Puppet IPC12

1761ecd7fe763583553dde43e62c47bd?s=47 Joshua Thijssen
October 15, 2012
340

Workshop Puppet IPC12

1761ecd7fe763583553dde43e62c47bd?s=128

Joshua Thijssen

October 15, 2012
Tweet

Transcript

  1. Puppet for Dummies IPC - Mainz - Germany 14-17 october

    2012 workshop
  2. Joshua Thijssen / Netherlands Freelance consultant and trainer @ NoxLogic

    & TechAdemy Development in PHP, Python, C, Java Lead developer of Saffire Blog: http://adayinthelifeof.nl Email: jthijssen@noxlogic.nl Twitter: @jaytaph whoami 2
  3. .plan ➡ A bit of history / intro about why

    use Puppet and vagrant. ➡ Install / using vagrant ➡ Intro on writing puppet manifests ➡ Actually writing puppet manifests ➡ Misc (monitoring, mcollective, enc etc) 3
  4. Fingers crossed 4

  5. prerequisites 5 vagrantup.com ruby-lang.org virtualbox.org

  6. A bit of history 6

  7. Installing by hand 7

  8. More servers - preinstalls 8

  9. Virtualization - even more servers! 9

  10. “The cloud”(tm) 10

  11. 11 How are we going to manage our projects, servers,

    clouds, customers properly?
  12. PUPPET 12

  13. System admins vs developers 13

  14. 14 Puppet for sysadmins

  15. Puppet for sysadmins ➡ Control your infrastructure from a single

    point (of failure). ➡ Documented upgrades through version control. ➡ Easy upgrades. ➡ Acceptance infrastructure environments. 15
  16. 16 Puppet for developers

  17. Puppet for developers 17 LAMP

  18. Puppet for developers 18 PENELOPE

  19. Puppet for developers 19 PeNeLoPe = PHP NGINX LINUX POSTGRESQL

  20. LAMPGMVNMCSTRAH Linux Apache MySQL PHP Gearman MongoDB CouchDB Solr Tika

    Redis ActiveMQ Hadoop Varnish Nginx Memcache 20 Puppet for developers
  21. ➡ How do you make sure all developers are using

    the same versions of your components? ➡ The same configuration? ➡ Even the same components! ➡ New developers? New development install ➡ Keep development, acceptance, production in sync? 21 Puppet for developers
  22. 22 previous troubles * projects * developers = Puppet for

    developers
  23. Title Text 23

  24. Vagrant 24

  25. My first vagrant 25 $ git clone git://gist.github.com/3863869.git vagrant001 $

    cd vagrant001 $ vagrant up
  26. My first vagrant 26 [default] Importing base box 'lucid32'... [default]

    The guest additions on this VM do not match the install version of VirtualBox! This may cause things such as forwarded ports, shared folders, and more to not work properly. If any of those things fail on this machine, please update the guest additions and repackage the box. Guest Additions Version: 4.1.14 VirtualBox Version: 4.2.0 [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
  27. My first vagrant 27

  28. Vagrant is a tool for building and distributing virtualized development

    environments. 28
  29. AWESOMESAUCE 29

  30. Actually, it’s a simple tool that speaks to the virtualbox

    api (but it’s still awesome) 30
  31. ➡ Downloads (optionally) the requested base box ➡ Deploys and

    boots up a new VM. ➡ Runs optional provisioner (puppet / chef / shell) 31
  32. Multi VM’s Vagrant::Config.run do |config| config.vm.box = 'centos-62-64-puppet' config.vm.box_url =

    'http://../centos-6.2-64bit-puppet-vbox.4.1.12.box' config.vm.define :web do |web_config| web_config.vm.host_name = 'web.example.org' web_config.vm.forward_port 80 8080 ... end config.vm.define :database do |db_config| db_config.vm.host_name = 'db.example.org' db_config.vm.forward_port 3306 3306 ... end end Vagrantfile 32
  33. Joind.in example Vagrant::Config.run do |config| # We define one box

    (joindin), but config.vm.define :joindin do |ji_config| ji_config.vm.box = 'centos-62-64-puppet' ji_config.vm.box_url = 'http://.../centos-6.2-64bit-puppet-vbox.4.1.12.box' ji_config.vm.host_name = "joind.in" ji_config.vm.forward_port 80, 8080 # config.vm.share_folder "v-data", "/vagrant_data", "../data" ji_config.vm.boot_mode = :gui ji_config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.module_path = "puppet/modules" puppet.manifest_file = "joindin.pp" puppet.options = [ '--verbose', ] end end end 33 https://github.com/joindin/joind.in
  34. Quick tips ➡ Use 32bit boxes. Only 64bit when you

    need to, or when you are sure all developers can run them. ➡ Use NFS mounts on linux / osx. (Can’t on windows) config.vm.share_folder(“v-root”, “/vagrant”, “.”, :nfs => (RUBY_PLATFORM =~ /linux/ or RUBY_PLATFORM =~ /darwin/)) 34
  35. Base boxes ➡ Package from current images ➡ Download them

    (http://vagrantbox.es) ➡ Minimal install (netinstall) ➡ vagrant user + “public” private key ➡ ssh server 35
  36. Base boxes 36 $ vagrant box list lucid32 centos-63-32bit-puppet $

    vagrant box add lucid32 lucid32.box $ vagrant box add centos-63-32bit-puppet centos63.box $ vagrant package $ vagrant package --vagrantfile Vagrantfile.pkg --include README.txt
  37. Shared directories ➡ Work from local directory (IDE) ➡ Run

    remote (33.33.33.10) ➡ /vagrant default shared ➡ NFS, vboxfs ➡ watch out with file permissions! 37
  38. PUPPET 38

  39. ➡ Open source configuration management tool. ➡ Puppet Labs (Reductive

    Labs) ➡ Written in Ruby ➡ Open source: https://github.com/puppetlabs ➡ Commercial version available (puppet enterprise) 39
  40. ➡ Don’t tell HOW to do stuff. ➡ Tell WHAT

    to do. ¹ “yum install httpd” “apt-get install apache2” “install and run the apache webserver” 40
  41. 41 Schematic representation of a puppet infrastructure

  42. Puppet 42

  43. Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent

    https 43
  44. ➡ puppet master (puppetmasterd) ➡ puppet cert (puppetca) ➡ puppet

    agent (puppetd) ➡ facter 44
  45. Puppet master ➡ Central server ➡ File & Configuration server

    ➡ REST(ish) over HTTPS interface 45
  46. Puppet CA ➡ Certificate signing ➡ Creates, signs, checks x509

    certificates ➡ So you don’t have to worry about them 46
  47. List all nodes 47 root@puppetmaster:~# puppet cert --list --all +

    puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) puppetnode1.noxlogic.local (09:9D:1E:01:D0:A7:BA:FB:8C:F4:2D:96:78:34:54:44)
  48. Sign a node 48 root@puppetmaster:~# puppet cert --sign puppetnode1.noxlogic.local ....

    root@puppetmaster:~# puppet cert --list --all + puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) + puppetnode1.noxlogic.local (CC:50:49:98:1D:F9:06:36:0E:6E:31:F5:27:D8:50:D8)
  49. Puppet agent ➡ Runs on every node that will be

    managed by puppet as a daemon (or crontab, or mcollective). ➡ Calls the puppet master every 30 minutes for updates. ➡ Receives and executes a “catalog”. 49
  50. Facter ➡ Runs on nodes to gather system information. ➡

    Returns $variables to be used on the puppet master in the manifest files. 50
  51. Facter 51 [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn

    => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9
  52. Facts ➡ You can create your own facts: ➡ project

    names ➡ master / slave databases ➡ zend server ➡ directadmin / plesk 52
  53. 53 zendserver.rb: Facter.add(“Zendserver”) do confine :kernel => :linux setcode do

    if FileTest.exists?(“/usr/local/zend/bin”) “true” else “false” end end end Facts
  54. Multi VM vagrant 54 $ git clone git://gist.github.com/3887712.git vagrant002 $

    cd vagrant002 $ vagrant up
  55. 55 Our first manifests

  56. Title Text 56 package { “strace” : ensure => present,

    } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, content => “beer”, }
  57. package { “httpd” : ensure => present, } service {

    “httpd”: running => true, enable => true, } require => Package[“httpd”], 57
  58. Centos / Redhat service: httpd package: httpd config: /etc/httpd/conf/httpd.conf vhosts:

    /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 config: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available 58
  59. class webserver { package { “apache”: case $operatingsystem { centos,

    redhat { $packagename = “httpd” } debian, ubuntu { $packagename = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $packagename, ensure => installed, } service { “apache” : running => true, enable => true, require => Package[“apache”], } } 59
  60. [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local

    interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 60
  61. Puppet resources • augeas • computer • cron • exec

    • file • filebucket • group • host • interface • k5login • macauthorization • mailalias • maillist • mcx • mount • nagios_command • nagios_contact • nagios_contactgroup • nagios_host • nagios_hostdependency • nagios_hostescalation • nagios_hostextinfo • nagios_hostgroup • nagios_service • nagios_servicedependency • nagios_serviceescalation • nagios_serviceextinfo • nagios_servicegroup • nagios_timeperiod • notify • package • resources • router • schedule • scheduled_task • selboolean • selmodule • service • ssh_authorized_key • sshkey • stage • tidy • user • vlan • yumrepo • zfs • zone • zpool 61
  62. node “web01.example.org” { include webserver } node /^db\d+\.example\.org$/ { package

    { “mysql-server” : ensure => installed, } } 62 /etc/puppet/manifests/site.pp:
  63. node “web01.example.local” { $webserver_name = “web01.example.local” $webserver_alias = “www.example.local” $webserver_docroot

    = “/var/www/web01” include webserver } node “web02.example.local” { $webserver_name = “web02.example.local” $webserver_alias = “crm.example.local” $webserver_docroot = “/var/www/web02” include webserver } 63
  64. Multi VM vagrant with puppet installed 64 $ git clone

    git://gist.github.com/3887842.git vagrant003 $ cd vagrant003 $ vagrant up
  65. Crude apache install 65 https://gist.github.com/3887955

  66. 66

  67. Monitoring Distributed basically...

  68. Distributed monitoring ➡ Export resources ➡ @@nagios_host ➡ @@nagios_service ➡

    Collect on (monitoring) server: ➡ Nagios_host <<| |>> ➡ Nagios_service <<| |>> 68
  69. Distributed monitoring 69 http://www.slideshare.net/PuppetLabs/distributed-monitoring- at-hyves-puppet

  70. Frontends The foreman - Puppet dashboard

  71. 71

  72. 72

  73. 73

  74. 74

  75. ENC External Node Configuration

  76. [master] node_terminus = exec external_nodes = /path/to/your/app puppet.conf:

  77. --- classes: common: puppet: ntp: ntpserver: 0.pool.ntp.org aptsetup: additional_apt_repos: -

    deb localrepo.example.com/ubuntu lucid production - deb localrepo.example.com/ubuntu lucid vendor parameters: ntp_servers: - 0.pool.ntp.org - ntp.example.com mail_server: mail.example.com iburst: true environment: production output YAML:
  78. ➡ Input is nodename. ➡ Output is YAML structure. ➡

    *CAN* mix site.pp and ENC, but wouldn’t recommend it. (http://docs.puppetlabs.com/ guides/external_nodes.html#how-merging- works) ➡ Possible to store nodes inside databases, LDAP etc..
  79. MCollective 79

  80. Title Text ➡ Marionette Collective ➡ server orchestration or parallel

    job execution system 80
  81. Title Text 81 ACTIVEMQ Client MCollective Server Node Middleware Client

    MCollective Server MCollective Server Collective
  82. 82 $ mco ping node1.phpconference.org time=51.48 ms node2.phpconference.org time=91.23 ms

    puppetmaster.phpconference.org time=91.60 ms ---- ping statistics ---- 3 replies max: 91.60 min: 51.48 avg: 78.10
  83. Title Text 83 $ mco facts kernel Report for fact:

    kernel Linux found 3 times Finished processing 3 / 3 hosts in 47.99 ms $ mco facts hostname Report for fact: hostname node1 found 1 times node2 found 1 times puppetmaster found 1 times Finished processing 3 / 3 hosts in 50.65 ms
  84. ➡ mco rpc <agent> <commands> 84

  85. ➡ find all (zombie) processes in your collective. ➡ find

    servers with 80% of utilized memory and running MySQL. ➡ restart all apache webservers in the UK, with less than 4GB of memory, except the ones running on debian 6.0 85
  86. ➡ Run or deploy software ➡ Restart services ➡ Start

    puppet agent ➡ Upgrade your system ➡ Write your own agents! 86