Workshop Puppet IPC12

Transcript

1. Puppet for Dummies IPC - Mainz - Germany 14-17 october 2012 workshop
2. Joshua Thijssen / Netherlands Freelance consultant and trainer @ NoxLogic & TechAdemy Development in PHP, Python, C, Java Lead developer of Saffire Blog: http://adayinthelifeof.nl Email: jthijssen@noxlogic.nl Twitter: @jaytaph whoami 2
3. .plan ➡ A bit of history / intro about why use Puppet and vagrant. ➡ Install / using vagrant ➡ Intro on writing puppet manifests ➡ Actually writing puppet manifests ➡ Misc (monitoring, mcollective, enc etc) 3
4. Fingers crossed 4
5. prerequisites 5 vagrantup.com ruby-lang.org virtualbox.org
6. A bit of history 6
7. Installing by hand 7
8. More servers - preinstalls 8
9. Virtualization - even more servers! 9
10. “The cloud”(tm) 10
11. 11 How are we going to manage our projects, servers, clouds, customers properly?
12. PUPPET 12
13. System admins vs developers 13
14. 14 Puppet for sysadmins
15. Puppet for sysadmins ➡ Control your infrastructure from a single point (of failure). ➡ Documented upgrades through version control. ➡ Easy upgrades. ➡ Acceptance infrastructure environments. 15
16. 16 Puppet for developers
17. Puppet for developers 17 LAMP
18. Puppet for developers 18 PENELOPE
19. Puppet for developers 19 PeNeLoPe = PHP NGINX LINUX POSTGRESQL
20. LAMPGMVNMCSTRAH Linux Apache MySQL PHP Gearman MongoDB CouchDB Solr Tika Redis ActiveMQ Hadoop Varnish Nginx Memcache 20 Puppet for developers
21. ➡ How do you make sure all developers are using the same versions of your components? ➡ The same configuration? ➡ Even the same components! ➡ New developers? New development install ➡ Keep development, acceptance, production in sync? 21 Puppet for developers
22. 22 previous troubles * projects * developers = Puppet for developers
23. Title Text 23
24. Vagrant 24
25. My first vagrant 25 $ git clone git://gist.github.com/3863869.git vagrant001 $ cd vagrant001 $ vagrant up
26. My first vagrant 26 [default] Importing base box 'lucid32'... [default] The guest additions on this VM do not match the install version of VirtualBox! This may cause things such as forwarded ports, shared folders, and more to not work properly. If any of those things fail on this machine, please update the guest additions and repackage the box. Guest Additions Version: 4.1.14 VirtualBox Version: 4.2.0 [default] Matching MAC address for NAT networking... [default] Clearing any previously set forwarded ports... [default] Forwarding ports... [default] -- 22 => 2222 (adapter 1) [default] Creating shared folders metadata... [default] Clearing any previously set network interfaces... [default] Booting VM... [default] Waiting for VM to boot. This can take a few minutes. [default] VM booted and ready for use! [default] Mounting shared folders... [default] -- v-root: /vagrant
27. My first vagrant 27
28. Vagrant is a tool for building and distributing virtualized development environments. 28
29. AWESOMESAUCE 29
30. Actually, it’s a simple tool that speaks to the virtualbox api (but it’s still awesome) 30
31. ➡ Downloads (optionally) the requested base box ➡ Deploys and boots up a new VM. ➡ Runs optional provisioner (puppet / chef / shell) 31
32. Multi VM’s Vagrant::Config.run do |config| config.vm.box = 'centos-62-64-puppet' config.vm.box_url = 'http://../centos-6.2-64bit-puppet-vbox.4.1.12.box' config.vm.define :web do |web_config| web_config.vm.host_name = 'web.example.org' web_config.vm.forward_port 80 8080 ... end config.vm.define :database do |db_config| db_config.vm.host_name = 'db.example.org' db_config.vm.forward_port 3306 3306 ... end end Vagrantfile 32
33. Joind.in example Vagrant::Config.run do |config| # We define one box (joindin), but config.vm.define :joindin do |ji_config| ji_config.vm.box = 'centos-62-64-puppet' ji_config.vm.box_url = 'http://.../centos-6.2-64bit-puppet-vbox.4.1.12.box' ji_config.vm.host_name = "joind.in" ji_config.vm.forward_port 80, 8080 # config.vm.share_folder "v-data", "/vagrant_data", "../data" ji_config.vm.boot_mode = :gui ji_config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests" puppet.module_path = "puppet/modules" puppet.manifest_file = "joindin.pp" puppet.options = [ '--verbose', ] end end end 33 https://github.com/joindin/joind.in
34. Quick tips ➡ Use 32bit boxes. Only 64bit when you need to, or when you are sure all developers can run them. ➡ Use NFS mounts on linux / osx. (Can’t on windows) config.vm.share_folder(“v-root”, “/vagrant”, “.”, :nfs => (RUBY_PLATFORM =~ /linux/ or RUBY_PLATFORM =~ /darwin/)) 34
35. Base boxes ➡ Package from current images ➡ Download them (http://vagrantbox.es) ➡ Minimal install (netinstall) ➡ vagrant user + “public” private key ➡ ssh server 35
36. Base boxes 36 $ vagrant box list lucid32 centos-63-32bit-puppet $ vagrant box add lucid32 lucid32.box $ vagrant box add centos-63-32bit-puppet centos63.box $ vagrant package $ vagrant package --vagrantfile Vagrantfile.pkg --include README.txt
37. Shared directories ➡ Work from local directory (IDE) ➡ Run remote (33.33.33.10) ➡ /vagrant default shared ➡ NFS, vboxfs ➡ watch out with file permissions! 37
38. PUPPET 38
39. ➡ Open source configuration management tool. ➡ Puppet Labs (Reductive Labs) ➡ Written in Ruby ➡ Open source: https://github.com/puppetlabs ➡ Commercial version available (puppet enterprise) 39
40. ➡ Don’t tell HOW to do stuff. ➡ Tell WHAT to do. ¹ “yum install httpd” “apt-get install apache2” “install and run the apache webserver” 40
41. 41 Schematic representation of a puppet infrastructure
42. Puppet 42
43. Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent https 43
44. ➡ puppet master (puppetmasterd) ➡ puppet cert (puppetca) ➡ puppet agent (puppetd) ➡ facter 44
45. Puppet master ➡ Central server ➡ File & Configuration server ➡ REST(ish) over HTTPS interface 45
46. Puppet CA ➡ Certificate signing ➡ Creates, signs, checks x509 certificates ➡ So you don’t have to worry about them 46
47. List all nodes 47 root@puppetmaster:~# puppet cert --list --all + puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) puppetnode1.noxlogic.local (09:9D:1E:01:D0:A7:BA:FB:8C:F4:2D:96:78:34:54:44)
48. Sign a node 48 root@puppetmaster:~# puppet cert --sign puppetnode1.noxlogic.local .... root@puppetmaster:~# puppet cert --list --all + puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) + puppetnode1.noxlogic.local (CC:50:49:98:1D:F9:06:36:0E:6E:31:F5:27:D8:50:D8)
49. Puppet agent ➡ Runs on every node that will be managed by puppet as a daemon (or crontab, or mcollective). ➡ Calls the puppet master every 30 minutes for updates. ➡ Receives and executes a “catalog”. 49
50. Facter ➡ Runs on nodes to gather system information. ➡ Returns $variables to be used on the puppet master in the manifest files. 50
51. Facter 51 [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9
52. Facts ➡ You can create your own facts: ➡ project names ➡ master / slave databases ➡ zend server ➡ directadmin / plesk 52
53. 53 zendserver.rb: Facter.add(“Zendserver”) do confine :kernel => :linux setcode do if FileTest.exists?(“/usr/local/zend/bin”) “true” else “false” end end end Facts
54. Multi VM vagrant 54 $ git clone git://gist.github.com/3887712.git vagrant002 $ cd vagrant002 $ vagrant up
55. 55 Our first manifests
56. Title Text 56 package { “strace” : ensure => present, } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, content => “beer”, }
57. package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, } require => Package[“httpd”], 57
58. Centos / Redhat service: httpd package: httpd config: /etc/httpd/conf/httpd.conf vhosts: /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 config: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available 58
59. class webserver { package { “apache”: case $operatingsystem { centos, redhat { $packagename = “httpd” } debian, ubuntu { $packagename = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $packagename, ensure => installed, } service { “apache” : running => true, enable => true, require => Package[“apache”], } } 59
60. [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 60
61. Puppet resources • augeas • computer • cron • exec • file • filebucket • group • host • interface • k5login • macauthorization • mailalias • maillist • mcx • mount • nagios_command • nagios_contact • nagios_contactgroup • nagios_host • nagios_hostdependency • nagios_hostescalation • nagios_hostextinfo • nagios_hostgroup • nagios_service • nagios_servicedependency • nagios_serviceescalation • nagios_serviceextinfo • nagios_servicegroup • nagios_timeperiod • notify • package • resources • router • schedule • scheduled_task • selboolean • selmodule • service • ssh_authorized_key • sshkey • stage • tidy • user • vlan • yumrepo • zfs • zone • zpool 61
62. node “web01.example.org” { include webserver } node /^db\d+\.example\.org$/ { package { “mysql-server” : ensure => installed, } } 62 /etc/puppet/manifests/site.pp:
63. node “web01.example.local” { $webserver_name = “web01.example.local” $webserver_alias = “www.example.local” $webserver_docroot = “/var/www/web01” include webserver } node “web02.example.local” { $webserver_name = “web02.example.local” $webserver_alias = “crm.example.local” $webserver_docroot = “/var/www/web02” include webserver } 63
64. Multi VM vagrant with puppet installed 64 $ git clone git://gist.github.com/3887842.git vagrant003 $ cd vagrant003 $ vagrant up
65. Crude apache install 65 https://gist.github.com/3887955
67. Monitoring Distributed basically...
68. Distributed monitoring ➡ Export resources ➡ @@nagios_host ➡ @@nagios_service ➡ Collect on (monitoring) server: ➡ Nagios_host <<| |>> ➡ Nagios_service <<| |>> 68
69. Distributed monitoring 69 http://www.slideshare.net/PuppetLabs/distributed-monitoring- at-hyves-puppet
70. Frontends The foreman - Puppet dashboard
75. ENC External Node Configuration
76. [master] node_terminus = exec external_nodes = /path/to/your/app puppet.conf:
77. --- classes: common: puppet: ntp: ntpserver: 0.pool.ntp.org aptsetup: additional_apt_repos: - deb localrepo.example.com/ubuntu lucid production - deb localrepo.example.com/ubuntu lucid vendor parameters: ntp_servers: - 0.pool.ntp.org - ntp.example.com mail_server: mail.example.com iburst: true environment: production output YAML:
78. ➡ Input is nodename. ➡ Output is YAML structure. ➡ *CAN* mix site.pp and ENC, but wouldn’t recommend it. (http://docs.puppetlabs.com/ guides/external_nodes.html#how-merging- works) ➡ Possible to store nodes inside databases, LDAP etc..
79. MCollective 79
80. Title Text ➡ Marionette Collective ➡ server orchestration or parallel job execution system 80
81. Title Text 81 ACTIVEMQ Client MCollective Server Node Middleware Client MCollective Server MCollective Server Collective
82. 82 $ mco ping node1.phpconference.org time=51.48 ms node2.phpconference.org time=91.23 ms puppetmaster.phpconference.org time=91.60 ms ---- ping statistics ---- 3 replies max: 91.60 min: 51.48 avg: 78.10
83. Title Text 83 $ mco facts kernel Report for fact: kernel Linux found 3 times Finished processing 3 / 3 hosts in 47.99 ms $ mco facts hostname Report for fact: hostname node1 found 1 times node2 found 1 times puppetmaster found 1 times Finished processing 3 / 3 hosts in 50.65 ms
84. ➡ mco rpc <agent> <commands> 84
85. ➡ find all (zombie) processes in your collective. ➡ find servers with 80% of utilized memory and running MySQL. ➡ restart all apache webservers in the UK, with less than 4GB of memory, except the ones running on debian 6.0 85
86. ➡ Run or deploy software ➡ Restart services ➡ Start puppet agent ➡ Upgrade your system ➡ Write your own agents! 86