Puppet for dummies - ZendCon 2011

Transcript

1. Puppet for Dummies ZendCon - October 2011 Santa Clara - United States http://joind.in/3781 vrijdag 27 april 12
2. Who am I? Joshua Thijssen Senior Software Engineer @ Enrise (Netherlands) Development in PHP, Python, Perl, C, Java, and System & DB admin. Blog: http://adayinthelifeof.nl Email: joshua@enrise.com Twitter: @jaytaph http://www.flickr.com/photos/akrabat/5422369749/in/photostream/ vrijdag 27 april 12
3. Joind.in ‣ http://joind.in/3781 vrijdag 27 april 12
4. The question of the day vrijdag 27 april 12
5. The question of the day What is puppet and why should I care? vrijdag 27 april 12
6. Why should I care? “People are finally figuring out puppet and how it gets you to the pub by 4pm. Note that I’ve been at this pub since 2pm.” - Jorge Castro vrijdag 27 april 12
7. Why should I care (really)? vrijdag 27 april 12
8. What is puppet? Puppet is a (not necessarily the) solution for the following problem: How do we setup, manage, synchronize, and upgrade our internal and external infrastructure? vrijdag 27 april 12
9. But isn’t that a sysadmin problem? Sysadmin! Y U no fix problem! vrijdag 27 april 12
10. But isn’t that a sysadmin problem? Sysadmin! Y U no fix problem! NO vrijdag 27 april 12
11. What is puppet? LAMP-stack vrijdag 27 april 12
12. What is puppet? LAMP-stack Linux Apache MySQL PHP vrijdag 27 april 12
13. What is puppet? LAMPGMVNMCSTRAH-stack vrijdag 27 april 12
14. What is puppet? LAMPGMVNMCSTRAH-stack Linux Apache MySQL PHP Gearman MongoDB CouchDB Solr Tika Redis ActiveMQ Hadoop Varnish Ngnix Memcache vrijdag 27 april 12
15. How do we manage our infrastructure? vrijdag 27 april 12
16. How do we manage our infrastructure? ‣ Solution 1: We don’t, vrijdag 27 april 12
17. How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, vrijdag 27 april 12
18. How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, ‣ Solution 3: We automate the process. vrijdag 27 april 12
19. How do we manage our infrastructure? (1) ‣ Solution 1: we don’t vrijdag 27 april 12
20. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Solution 1: we don’t vrijdag 27 april 12
21. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ Solution 1: we don’t vrijdag 27 april 12
22. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ We only act on escalation ‣ Solution 1: we don’t vrijdag 27 april 12
23. How do we manage our infrastructure? (1) ‣ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ We only act on escalation ‣ reactive, not proactive ‣ Solution 1: we don’t vrijdag 27 april 12
24. How do we manage our infrastructure? (2) ‣ Solution 2: we outsource vrijdag 27 april 12
25. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ Solution 2: we outsource vrijdag 27 april 12
26. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Solution 2: we outsource vrijdag 27 april 12
27. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Fight between stability and agility. ‣ Solution 2: we outsource vrijdag 27 april 12
28. How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Fight between stability and agility. ‣ Does your hosting company decide on whether you can use PHP5.3??? ‣ Solution 2: we outsource vrijdag 27 april 12
29. How do we manage our infrastructure? (3) ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12
30. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12
31. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12
32. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Use: cfEngine, chef, puppet. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12
33. How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Use: cfEngine, chef, puppet. ‣ When done right, maintenance should not be difficult. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12
34. What is puppet? ‣ PUPPET TO THE RESCUE vrijdag 27 april 12
35. What is puppet? ‣ Open source configuration management tool. ‣ Written in Ruby ‣ Open source: https://github.com/puppetlabs ‣ Commercial version available (puppet enterprise) vrijdag 27 april 12
36. What is puppet? ‣ Don’t tell HOW to do stuff. ‣ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... vrijdag 27 april 12
37. What is puppet? ‣ Don’t tell HOW to do stuff. ‣ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... “yum install httpd” “apt-get install apache2” “install and run the apache webserver” vrijdag 27 april 12
38. Architectural overview vrijdag 27 april 12
39. Architectural overview Puppet vrijdag 27 april 12
40. Architectural overview Puppet CA Puppet Master Puppet Agent https vrijdag 27 april 12
41. Architectural overview Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent https vrijdag 27 april 12
42. How does it work Puppet master Puppet client vrijdag 27 april 12
43. How does it work Puppet master Puppet client Check credentials vrijdag 27 april 12
44. How does it work Puppet master Puppet client Check credentials Send facts vrijdag 27 april 12
45. How does it work Puppet master Puppet client Check credentials Send facts Returns “catalog” vrijdag 27 april 12
46. How does it work Puppet master Puppet client Check credentials Send facts Returns “catalog” Report results vrijdag 27 april 12
47. Puppet manifests ‣ Manifests are puppet definitions ‣ <filename>.pp ‣ Puppet DSL ‣ De-cla-ra-tive language ‣ Version your manifests! (git/svn) vrijdag 27 april 12
48. Puppet manifests package { “strace” : ensure => present, } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, source => “puppet:///secret.txt”, } vrijdag 27 april 12
49. Puppet manifests ‣ Spot the problem.... package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, require => Package[“httpd”], } vrijdag 27 april 12
50. Puppet manifests ‣ Different distributions, different names Centos / Redhat service: httpd package: httpd config: /etc/httpd/conf/httpd.conf vhosts: /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 config: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available vrijdag 27 april 12
51. Puppet manifests ‣ $operatingsystem is a FACT package { “webserver”: case $operatingsystem { centos, redhat { $apache = “httpd” } debian, ubuntu { $apache = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $apache, ensure => installed, } vrijdag 27 april 12
52. Facter [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 ‣ A simple list with info (also useable in your own tools) vrijdag 27 april 12
53. Puppet manifests node default { $def_packages = [ “mc”, “strace”, “sysstat” ] package { $def_packages : ensure => latest, } } /etc/puppet/manifests/site.pp: ‣ “Main” manifest vrijdag 27 april 12
54. Puppet manifests ‣ Defining nodes - regular expressions node /^web\d+\.example\.local$/ { package { “httpd” : ensure => latest, } } node /^db\d+\.example\.local$/ { package { “mysql-server” : ensure => installed, } } vrijdag 27 april 12
55. Puppet manifests node basenode { user { “jaytaph” : ensure => present, gid => 1000, uid => 1000, home => “/home/jaytaph”, shell => “/bin/sh”, password => “supersecrethashedpassword”, managehome => true, } } node /^.+\.example\.local/ inherits basenode { ... } ‣ Node inheritance vrijdag 27 april 12
56. Puppet manifests ‣ Group together into a class vrijdag 27 april 12
57. Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } ‣ Group together into a class vrijdag 27 april 12
58. Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } file { “vhost_${webserver_name}” : path => “/etc/httpd/conf/10-vhost.conf”, content => template(“vhost.template.erb”), notify => Service[“httpd”], } ‣ Group together into a class vrijdag 27 april 12
59. Puppet manifests ‣ ERB Templates can use custom variables and facts <virtualHost <%= ipaddress %>:80> ServerName <%= webserver_name %> ServerAlias <%= webserver_alias %> DocumentRoot <%= webserver_docroot %> </virtualHost> vhost.template.erb vrijdag 27 april 12
60. Puppet manifests node “web01.example.local” inherits base { $webserver_name = “web01.example.local” $webserver_alias = “www.example.local” $webserver_docroot = “/var/www/web01” import webserver } node “web02.example.local” inherits base { $webserver_name = “web02.example.local” $webserver_alias = “crm.example.local” $webserver_docroot = “/var/www/web02” import webserver } vrijdag 27 april 12
61. Puppet modules ‣ A puppet module is a collection of resources, classes, templates. ‣ Used for easy distribution and code-reuse. ‣ Self-contained, run out-of-the-box vrijdag 27 april 12
62. Puppet modules ‣ puppetforge / github ‣ Create your own (and share!). ‣ Use the ones from puppet enterprise edition. ‣ Use the standard layout / best practices vrijdag 27 april 12
63. Puppet modules class ntp::install { package{"ntpd": ensure => latest } } class ntp::config { File{ require => Class["ntp::install"], notify => Class["ntp::service"], owner => "root", group => "root", mode => 644 } file{"/etc/ntp.conf": source => "puppet:///ntp/ntp.conf"; "/etc/ntp/step-tickers": source => "puppet:///ntp/step-tickers"; } } class ntp::service { service{"ntp": ensure => running, enable => true, require => Class["ntp::config"], } } class ntp { include ntp::install, ntp::config, ntp::service } vrijdag 27 april 12
64. Test your modules ‣ (Unit)test your modules ‣ Test them with: puppet apply --noop ‣ More advanced testing: cucumber / cucumber-puppet (BDD) vrijdag 27 april 12
65. What can puppet manage ‣ http://docs.puppetlabs.com/references/stable/type.html ‣ Almost everything. ‣ standard 48 different resource types ‣ Ranging from “file” to “cron” to “ssh_key” to “user” to “selinux”. ‣ Can control your Cisco routers and windows machines too (sortakinda) vrijdag 27 april 12
66. Confusing puppet things vrijdag 27 april 12
67. Confusing puppet things ‣ Puppet went from v0.25 to v2.6. ‣ REST interface since 2.6. XMLRPC before that. ‣ One binary to rule them all (puppet). ‣ Puppet v2.7 switched from GPLv2 to apache2.0 license. vrijdag 27 april 12
68. Confusing puppet things ‣ --test does not mean dry-run! (--noop does). ‣ It’s not object oriented. (puppet class != php class) ‣ It’s a declarative language. vrijdag 27 april 12
69. Puppet dashboards http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif vrijdag 27 april 12
70. Puppet dashboards http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif vrijdag 27 april 12
71. Live demo | MCollective? vrijdag 27 april 12
72. MCollective ‣ Puppet agent “calls” the master every 30 minutes. ‣ But what about realtime command & control? ‣ “Puppet kick”... (meh) ‣ MCollective (Marionette Collective) vrijdag 27 april 12
73. MCollective ‣ How do we handle large number of nodes? ‣ Which systems running a database and have 16GB or less? ‣ Which systems are using <50% of available memory? ‣ Restart all apache services in timezone GMT+5. vrijdag 27 april 12
74. MCollective ACTIVEMQ Client MCollective Server Node Middleware Client MCollective Server MCollective Server ‣ Middleware takes care of distribution, ‣ queued, broadcast etc.. Collective vrijdag 27 april 12
75. MCollective http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html ‣ The collective vrijdag 27 april 12
76. MCollective ‣ Filter out nodes based on facts $ mc-facts operatingsystem Report for fact: operatingsystem CentOS found 3 times Debian found 14 times Solaris found 4 times $ mc-facts -W operatingsystem=Centos operatingsystemrelease Report for fact: operatingsystemrelease 6.0 found 1 times 5.6 found 2 times vrijdag 27 april 12
77. MCollective - cool stuff ‣ Display all running processes ‣ Run or deploy software ‣ Restart services ‣ Start puppet agent ‣ Upgrade your systems vrijdag 27 april 12
78. Recap -ETOOMUCHINFO Let’s recap vrijdag 27 april 12
79. Recap (1) ‣ Configuration management tool. ‣ Focusses on “what” instead of “how”. ‣ Scales from 1 to 100K+ systems. ‣ Uses descriptive manifests. vrijdag 27 april 12
80. Recap (2) ‣ Useful for sysadmins and developers. ‣ Keeps your infrastructure in sync. ‣ Keeps your infrastructure versioned. ‣ MCollective controls your hosts based on facts, not names. vrijdag 27 april 12
81. Any questions? http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg vrijdag 27 april 12
82. to remove this comic sans font, please rate my talk on: http://joind.in/3781 vrijdag 27 april 12