Puppet for dummies - PHPbenelux meetup - Gent

Transcript

1. Puppet for Dummies ZendCon - October 2011 Santa Clara - United States http://joind.in/3781 woensdag 25 april 12
2. Who am I? Joshua Thijssen Senior Software Engineer @ Enrise (Netherlands) Development in PHP, Python, Perl, C, Java, and system & DB admin. Blog: http://www.adayinthelifeof.nl Email: joshua@enrise.com Twitter: @jaytaph http://www.flickr.com/photos/akrabat/5422369749/in/photostream/ woensdag 25 april 12
3. The question of the day woensdag 25 april 12
4. The question of the day What is puppet and why should I care? woensdag 25 april 12
5. Why use puppet? “People are finally figuring out puppet and how it gets you to the pub by 4pm. Note that I’ve been at this pub since 2pm.” - Jorge Castro woensdag 25 april 12
6. Why use puppet? woensdag 25 april 12
7. What is puppet? Puppet is a (not necessarily the) solution for the following problem: How do we setup, manage, synchronize, and upgrade our internal and external infrastructure? woensdag 25 april 12
8. What is puppet? But isn’t that a sysadmin problem?? woensdag 25 april 12
9. What is puppet? Short answer: woensdag 25 april 12
10. What is puppet? Short answer: NO woensdag 25 april 12
11. How do we manage our infrastructure? woensdag 25 april 12
12. How do we manage our infrastructure? ‣ Solution 1: We don’t, woensdag 25 april 12
13. How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, woensdag 25 april 12
14. How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, ‣ Solution 3: We automate the process. woensdag 25 april 12
15. How do we manage our infrastructure? (1) woensdag 25 april 12
16. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. woensdag 25 april 12
17. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. woensdag 25 april 12
18. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ We only act on escalation woensdag 25 april 12
19. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ We only act on escalation ‣ reactive, not proactive woensdag 25 april 12
20. How do we manage our infrastructure? (2) woensdag 25 april 12
21. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. woensdag 25 april 12
22. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? woensdag 25 april 12
23. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Fight between stability and agility. woensdag 25 april 12
24. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Fight between stability and agility. ‣ Does your hosting company decide on whether you can use PHP5.3??? woensdag 25 april 12
25. How do we manage our infrastructure? (3) woensdag 25 april 12
26. How do we manage our infrastructure? (3) ‣ We are in charge. woensdag 25 april 12
27. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ Dedicated package repositories, tools, etc,.. woensdag 25 april 12
28. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ Dedicated package repositories, tools, etc,.. ‣ Use: cfEngine, chef, puppet. woensdag 25 april 12
29. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ Dedicated package repositories, tools, etc,.. ‣ Use: cfEngine, chef, puppet. ‣ It’s actually not that hard. woensdag 25 april 12
30. What is puppet? ‣ Open source configuration management tool. ‣ Written in Ruby ‣ Open source https://github.com/puppetlabs ‣ Commercial version available (puppet enterprise) woensdag 25 april 12
31. What is puppet? ‣ Don’t tell HOW to do stuff. ‣ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... woensdag 25 april 12
32. Architectural overview woensdag 25 april 12
33. Architectural overview Puppet woensdag 25 april 12
34. Architectural overview Puppet CA Puppet Master Puppet Agent https woensdag 25 april 12
35. Architectural overview Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent https woensdag 25 april 12
36. Puppet structure ‣ Puppet master (puppetmasterd) ‣ Puppet cert (puppetca) ‣ Puppet agent (puppetd) ‣ Facter woensdag 25 april 12
37. Puppet master (puppetmasterd) ‣ Central server ‣ File & configuration server ‣ REST over HTTPS interface woensdag 25 april 12
38. Puppet cert (puppet CA) ‣ Certificate signing server ‣ Creates, signs, checks x509 certificates ‣ So you don’t have to worry about it woensdag 25 april 12
39. Puppet cert (puppet CA) root@puppetmaster:~# puppet cert --list --all + puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) puppetnode1.noxlogic.local (09:9D:1E:01:D0:A7:BA:FB:8C:F4:2D:96:78:34:54:44) Check all systems that have connected to our CA server woensdag 25 april 12
40. Puppet cert (puppet CA) root@puppetmaster:~# puppet cert --sign puppetnode1.noxlogic.local .... root@puppetmaster:~# puppet cert --list --all + puppetmaster.noxlogic.local (74:A7:C8:27:72:0D:C1:DD:B8:71:0D:4F:37:69:3D:0C) + puppetnode1.noxlogic.local (CC:50:49:98:1D:F9:06:36:0E:6E:31:F5:27:D8:50:D8) Let’s sign our first node woensdag 25 april 12
41. Puppet agent (puppetd) ‣ Runs on every node that will be managed by puppet. ‣ Calls the puppet master every 30 minutes with system information. ‣ Receives and executes a catalog. woensdag 25 april 12
42. Facter ‣ Runs on nodes to gather system information. ‣ Returns $variables to be used in configuration. woensdag 25 april 12
43. Facter (1) [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 ‣ A simple list with info (also useable in your own tools) woensdag 25 april 12
44. Facter (2) ‣ Very simple to add new facts (in ruby, that is) ‣ You can add your own facts: ‣ project name ‣ master / slave database server ‣ zend server ‣ directadmin / plesk woensdag 25 april 12
45. Facter (3) ‣ Crude, but effective enough for us zendstudio.rb: Facter.add(“Zendserver”) do confine :kernel => :linux setcode do if FileTest.exists?(“/usr/local/zend/bin”) “true” else “false” end end end woensdag 25 april 12
46. How does it work Master Client Check cert Return facts Returns catalog woensdag 25 april 12
47. Puppet manifests ‣ Manifests are puppet definitions ‣ <filename>.pp ‣ Puppet DSL ‣ De-cla-ra-tive language ‣ Version your manifests! (git/svn) woensdag 25 april 12
48. Puppet manifests package { “mc” : ensure => present, } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, source => “puppet:///secret.txt”, } woensdag 25 april 12
49. Puppet manifests ‣ Spot the problem.... package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, require => Package[“httpd”], } woensdag 25 april 12
50. Puppet manifests ‣ Different distributions, different names Centos / Redhat service: httpd package: httpd config: /etc/httpd/conf/httpd.conf vhosts: /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 config: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available woensdag 25 april 12
51. Puppet manifests ‣ $operatingsystem is a FACT service { “apache”: case $operatingsystem { centos, redhat { $apache = “httpd” } debian, ubuntu { $apache = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $apache, running => true, } woensdag 25 april 12
52. Puppet manifests node default { $def_packages = [ “mc”, “strace”, “sysstat” ] package { $def_packages : ensure => latest, } } /etc/puppet/manifests/site.pp: ‣ “Main” manifest woensdag 25 april 12
53. Puppet manifests ‣ Defining nodes node ‘web.noxlogic.local’ { package { “httpd” : ensure => latest, } } node ‘db.noxlogic.local’ { package { “mysql-server” : ensure => installed, } } woensdag 25 april 12
54. Puppet manifests node basenode { user { “jaytaph” : ensure => present, gid => 1000, uid => 1000, home => “/home/jaytaph”, shell => “/bin/sh”, password => “supersecrethashedpassword”, } } node *.noxlogic.local inherits basenode { ... } woensdag 25 april 12
55. Puppet manifests woensdag 25 april 12
56. Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“httpd”], } package { “apache” : ensure => installed, } } woensdag 25 april 12
57. Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“httpd”], } package { “apache” : ensure => installed, } } file { “vhost_$hostname” : path => “/etc/httpd/conf/10-vhost.conf”, content => template(“vhost.template.erb”), notify => Service[“httpd”], } woensdag 25 april 12
58. Puppet manifests ‣ ERB Templates can use custom variables and facts <virtualHost <%= ipaddress %>:80> ServerName <%= webserver_name %> ServerAlias <%= webserver_alias %> DocumentRoot <%= webserver_docroot %> </virtualHost> vhost.template.erb woensdag 25 april 12
59. Puppet manifests node “web01.noxlogic.local” inherits base { $webserver_name = “web01.noxlogic.local” $webserver_alias = “www.noxlogic.local” $webserver_docroot = “/var/www/web01” import webserver } node “web02.noxlogic.local” inherits base { $webserver_name = “web02.noxlogic.local” $webserver_alias = “crm.noxlogic.local” $webserver_docroot = “/var/www/web02” import webserver } woensdag 25 april 12
60. What can puppet manage ‣ http://docs.puppetlabs.com/references/stable/type.html ‣ Almost everything. ‣ standard 48 different resource types ‣ Ranging from “file” to “cron” to “ssh_key” to “user” to “selinux”. ‣ Can control your Cisco routers and windows machines too (sortakinda) woensdag 25 april 12
61. Puppet modules ‣ A puppet module is a collection of resources, classes, templates. ‣ Used for easy distribution and code-reuse. ‣ Self-contained, run out-of-the-box woensdag 25 april 12
62. Puppet modules ‣ puppetforge / github ‣ Create your own (and share!). ‣ Use the ones from puppet enterprise edition. ‣ Use the standard layout / best practices woensdag 25 april 12
63. Puppet modules MODULE_PATH/ !""downcased_module_name/ #""files/ #""manifests/ $ #""init.pp $ !""foo.pp #""lib/ $ #""puppet/ $ $ #""parser/ $ $ $ !""functions/ $ $ #""provider/ $ $ !""type/ $ !""facter/ #""templates/ #""tests $ #""init.pp $ !""foo.pp !""README woensdag 25 april 12
64. Puppet modules class ntp::install { package{"ntpd": ensure => latest } } class ntp::config { File{ require => Class["ntp::install"], notify => Class["ntp::service"], owner => "root", group => "root", mode => 644 } file{"/etc/ntp.conf": source => "puppet:///ntp/ntp.conf"; "/etc/ntp/step-tickers": source => "puppet:///ntp/step-tickers"; } } class ntp::service { service{"ntp": ensure => running, enable => true, require => Class["ntp::config"], } } class ntp { include ntp::install, ntp::config, ntp::service } woensdag 25 april 12
65. Test your modules ‣ (Unit)test your modules ‣ Test them with: puppet apply --noop ‣ More advanced testing: cucumber / cucumber-puppet (BDD) woensdag 25 april 12
66. External Node Configuration (1) ‣ Split modules and nodes ‣ Nodes should be classes - params only (best case scenario?) ‣ Nodes can be configured through YAML woensdag 25 april 12
67. External Node Configuration (2) --- classes: - base parameters: puppetserver: puppet.enrise.local node1.enrise.local.yaml woensdag 25 april 12
68. External Node Configuration (2) --- classes: - base parameters: puppetserver: puppet.enrise.local node1.enrise.local.yaml node node1.enrise.local { $puppetserver = ‘puppet.enrise.local’ include base } woensdag 25 april 12
69. External Node Configuration (3) ‣ Ruby, PHP, Python, Perl, Pony, shellscript. ‣ REST, SOAP, XMLRPC. ‣ Use a database backend. ‣ Or use LDAP instead of YAML. Puppet doesn’t care how you create YAML files. woensdag 25 april 12
70. Confusing puppet things woensdag 25 april 12
71. Confusing puppet things ‣ Puppet went from v0.25 to v2.6. ‣ REST interface since 2.6. XMLRPC before that. ‣ One binary to rule them all (puppet). ‣ Puppet v2.7 switched from GPLv2 to apache2.0 license. woensdag 25 april 12
72. Confusing puppet things ‣ --test does not mean dry-run! (--noop does). ‣ It’s not object oriented. (puppet class != php class) ‣ It’s a declarative language. woensdag 25 april 12
73. Puppet dashboard http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif woensdag 25 april 12
74. Live demo | MCollective? woensdag 25 april 12
75. MCollective ‣ Puppet agent “calls” the master every 30 minutes. ‣ But what about realtime command & control? ‣ “Puppet kick”... (meh) ‣ MCollective (Marionette Collective) woensdag 25 april 12
76. MCollective ‣ How do we handle large number of nodes? ‣ Which systems running a database and have 16GB or less? ‣ Which systems are using <50% of available memory? ‣ Restart all apache services in timezone GMT+5. woensdag 25 april 12
77. MCollective ACTIVEMQ Client MCollective Server Node Middleware Client MCollective Server MCollective Server ‣ Middleware takes care of distribution, ‣ queued, broadcast etc.. Collective woensdag 25 april 12
78. MCollective http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html ‣ The collective woensdag 25 april 12
79. MCollective ‣ Filter out nodes based on facts $ mc-facts operatingsystem Report for fact: operatingsystem CentOS found 3 times Debian found 14 times Solaris found 4 times $ mc-facts -W operatingsystem=Centos operatingsystemrelease Report for fact: operatingsystemrelease 6.0 found 1 times 5.6 found 2 times woensdag 25 april 12
80. MCollective - cool stuff ‣ Display all running processes ‣ Run or deploy software ‣ Restart services ‣ Start puppet agent ‣ Upgrade your systems woensdag 25 april 12
81. Recap -ETOOMUCHINFO Let’s recap woensdag 25 april 12
82. Recap (1) ‣ Configuration management tool. ‣ Focusses on “what” instead of “how”. ‣ Scales from 1 to 100K+ systems. ‣ Uses descriptive manifests. ‣ Can use external node configurations. woensdag 25 april 12
83. Recap (2) ‣ Useful for sysadmins and developers. ‣ Keeps your infrastructure in sync. ‣ Keeps your infrastructure versioned. ‣ MCollective controls your hosts based on facts, not names. woensdag 25 april 12
84. Any questions? http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg woensdag 25 april 12
85. ‣ THANK YOU FOR YOUR ATTENTION woensdag 25 april 12