Puppet for dummies - ZendCon 2011

Slide 1

Slide 1 text

Puppet for Dummies ZendCon - October 2011 Santa Clara - United States http://joind.in/3781 vrijdag 27 april 12

Slide 2

Slide 2 text

Who am I? Joshua Thijssen Senior Software Engineer @ Enrise (Netherlands) Development in PHP, Python, Perl, C, Java, and System & DB admin. Blog: http://adayinthelifeof.nl Email: joshua@enrise.com Twitter: @jaytaph http://www.ﬂickr.com/photos/akrabat/5422369749/in/photostream/ vrijdag 27 april 12

Slide 3

Slide 3 text

Joind.in ‣ http://joind.in/3781 vrijdag 27 april 12

Slide 4

Slide 4 text

The question of the day vrijdag 27 april 12

Slide 5

Slide 5 text

The question of the day What is puppet and why should I care? vrijdag 27 april 12

Slide 6

Slide 6 text

Why should I care? “People are ﬁnally ﬁguring out puppet and how it gets you to the pub by 4pm. Note that I’ve been at this pub since 2pm.” - Jorge Castro vrijdag 27 april 12

Slide 7

Slide 7 text

Why should I care (really)? vrijdag 27 april 12

Slide 8

Slide 8 text

What is puppet? Puppet is a (not necessarily the) solution for the following problem: How do we setup, manage, synchronize, and upgrade our internal and external infrastructure? vrijdag 27 april 12

Slide 9

Slide 9 text

But isn’t that a sysadmin problem? Sysadmin! Y U no ﬁx problem! vrijdag 27 april 12

Slide 10

Slide 10 text

But isn’t that a sysadmin problem? Sysadmin! Y U no ﬁx problem! NO vrijdag 27 april 12

Slide 11

Slide 11 text

What is puppet? LAMP-stack vrijdag 27 april 12

Slide 12

Slide 12 text

What is puppet? LAMP-stack Linux Apache MySQL PHP vrijdag 27 april 12

Slide 13

Slide 13 text

What is puppet? LAMPGMVNMCSTRAH-stack vrijdag 27 april 12

Slide 14

Slide 14 text

What is puppet? LAMPGMVNMCSTRAH-stack Linux Apache MySQL PHP Gearman MongoDB CouchDB Solr Tika Redis ActiveMQ Hadoop Varnish Ngnix Memcache vrijdag 27 april 12

Slide 15

Slide 15 text

How do we manage our infrastructure? vrijdag 27 april 12

Slide 16

Slide 16 text

How do we manage our infrastructure? ‣ Solution 1: We don’t, vrijdag 27 april 12

Slide 17

Slide 17 text

How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, vrijdag 27 april 12

Slide 18

Slide 18 text

How do we manage our infrastructure? ‣ Solution 1: We don’t, ‣ Solution 2: We outsource, ‣ Solution 3: We automate the process. vrijdag 27 april 12

Slide 19

Slide 19 text

How do we manage our infrastructure? (1) ‣ Solution 1: we don’t vrijdag 27 april 12

Slide 20

Slide 20 text

How do we manage our infrastructure? (1) ‣ It’s not funny: you ﬁnd it more often than not. Especially inside small development companies. ‣ Solution 1: we don’t vrijdag 27 april 12

Slide 21

Slide 21 text

How do we manage our infrastructure? (1) ‣ It’s not funny: you ﬁnd it more often than not. Especially inside small development companies. ‣ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ Solution 1: we don’t vrijdag 27 april 12

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

How do we manage our infrastructure? (2) ‣ Solution 2: we outsource vrijdag 27 april 12

Slide 25

Slide 25 text

How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ Solution 2: we outsource vrijdag 27 april 12

Slide 26

Slide 26 text

How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Solution 2: we outsource vrijdag 27 april 12

Slide 27

Slide 27 text

How do we manage our infrastructure? (2) ‣ Expensive $LA’s. ‣ What about INTERNAL servers like your development systems and infrastructure? ‣ Fight between stability and agility. ‣ Solution 2: we outsource vrijdag 27 april 12

Slide 28

Slide 28 text

Slide 29

Slide 29 text

How do we manage our infrastructure? (3) ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12

Slide 30

Slide 30 text

How do we manage our infrastructure? (3) ‣ We are in charge. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12

Slide 31

Slide 31 text

How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12

Slide 32

Slide 32 text

How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Use: cfEngine, chef, puppet. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12

Slide 33

Slide 33 text

How do we manage our infrastructure? (3) ‣ We are in charge. ‣ You can do what you like ‣ Use: cfEngine, chef, puppet. ‣ When done right, maintenance should not be difficult. ‣ Solution 3: we do it ourselves and automate vrijdag 27 april 12

Slide 34

Slide 34 text

What is puppet? ‣ PUPPET TO THE RESCUE vrijdag 27 april 12

Slide 35

Slide 35 text

What is puppet? ‣ Open source conﬁguration management tool. ‣ Written in Ruby ‣ Open source: https://github.com/puppetlabs ‣ Commercial version available (puppet enterprise) vrijdag 27 april 12

Slide 36

Slide 36 text

What is puppet? ‣ Don’t tell HOW to do stuff. ‣ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... vrijdag 27 april 12

Slide 37

Slide 37 text

What is puppet? ‣ Don’t tell HOW to do stuff. ‣ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... “yum install httpd” “apt-get install apache2” “install and run the apache webserver” vrijdag 27 april 12

Slide 38

Slide 38 text

Architectural overview vrijdag 27 april 12

Slide 39

Slide 39 text

Architectural overview Puppet vrijdag 27 april 12

Slide 40

Slide 40 text

Architectural overview Puppet CA Puppet Master Puppet Agent https vrijdag 27 april 12

Slide 41

Slide 41 text

Architectural overview Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent https vrijdag 27 april 12

Slide 42

Slide 42 text

How does it work Puppet master Puppet client vrijdag 27 april 12

Slide 43

Slide 43 text

How does it work Puppet master Puppet client Check credentials vrijdag 27 april 12

Slide 44

Slide 44 text

How does it work Puppet master Puppet client Check credentials Send facts vrijdag 27 april 12

Slide 45

Slide 45 text

How does it work Puppet master Puppet client Check credentials Send facts Returns “catalog” vrijdag 27 april 12

Slide 46

Slide 46 text

How does it work Puppet master Puppet client Check credentials Send facts Returns “catalog” Report results vrijdag 27 april 12

Slide 47

Slide 47 text

Puppet manifests ‣ Manifests are puppet deﬁnitions ‣ <ﬁlename>.pp ‣ Puppet DSL ‣ De-cla-ra-tive language ‣ Version your manifests! (git/svn) vrijdag 27 april 12

Slide 48

Slide 48 text

Puppet manifests package { “strace” : ensure => present, } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, source => “puppet:///secret.txt”, } vrijdag 27 april 12

Slide 49

Slide 49 text

Puppet manifests ‣ Spot the problem.... package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, require => Package[“httpd”], } vrijdag 27 april 12

Slide 50

Slide 50 text

Puppet manifests ‣ Different distributions, different names Centos / Redhat service: httpd package: httpd conﬁg: /etc/httpd/conf/httpd.conf vhosts: /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 conﬁg: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available vrijdag 27 april 12

Slide 51

Slide 51 text

Puppet manifests ‣ $operatingsystem is a FACT package { “webserver”: case $operatingsystem { centos, redhat { $apache = “httpd” } debian, ubuntu { $apache = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $apache, ensure => installed, } vrijdag 27 april 12

Slide 52

Slide 52 text

Facter [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 ‣ A simple list with info (also useable in your own tools) vrijdag 27 april 12

Slide 53

Slide 53 text

Puppet manifests node default { $def_packages = [ “mc”, “strace”, “sysstat” ] package { $def_packages : ensure => latest, } } /etc/puppet/manifests/site.pp: ‣ “Main” manifest vrijdag 27 april 12

Slide 54

Slide 54 text

Puppet manifests ‣ Deﬁning nodes - regular expressions node /^web\d+\.example\.local$/ { package { “httpd” : ensure => latest, } } node /^db\d+\.example\.local$/ { package { “mysql-server” : ensure => installed, } } vrijdag 27 april 12

Slide 55

Slide 55 text

Puppet manifests node basenode { user { “jaytaph” : ensure => present, gid => 1000, uid => 1000, home => “/home/jaytaph”, shell => “/bin/sh”, password => “supersecrethashedpassword”, managehome => true, } } node /^.+\.example\.local/ inherits basenode { ... } ‣ Node inheritance vrijdag 27 april 12

Slide 56

Slide 56 text

Puppet manifests ‣ Group together into a class vrijdag 27 april 12

Slide 57

Slide 57 text

Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } ‣ Group together into a class vrijdag 27 april 12

Slide 58

Slide 58 text

Puppet manifests class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } file { “vhost_${webserver_name}” : path => “/etc/httpd/conf/10-vhost.conf”, content => template(“vhost.template.erb”), notify => Service[“httpd”], } ‣ Group together into a class vrijdag 27 april 12

Slide 59

Slide 59 text

Puppet manifests ‣ ERB Templates can use custom variables and facts :80> ServerName <%= webserver_name %> ServerAlias <%= webserver_alias %> DocumentRoot <%= webserver_docroot %> vhost.template.erb vrijdag 27 april 12

Slide 60

Slide 60 text

Puppet manifests node “web01.example.local” inherits base { $webserver_name = “web01.example.local” $webserver_alias = “www.example.local” $webserver_docroot = “/var/www/web01” import webserver } node “web02.example.local” inherits base { $webserver_name = “web02.example.local” $webserver_alias = “crm.example.local” $webserver_docroot = “/var/www/web02” import webserver } vrijdag 27 april 12

Slide 61

Slide 61 text

Puppet modules ‣ A puppet module is a collection of resources, classes, templates. ‣ Used for easy distribution and code-reuse. ‣ Self-contained, run out-of-the-box vrijdag 27 april 12

Slide 62

Slide 62 text

Puppet modules ‣ puppetforge / github ‣ Create your own (and share!). ‣ Use the ones from puppet enterprise edition. ‣ Use the standard layout / best practices vrijdag 27 april 12

Slide 63

Slide 63 text

Puppet modules class ntp::install { package{"ntpd": ensure => latest } } class ntp::config { File{ require => Class["ntp::install"], notify => Class["ntp::service"], owner => "root", group => "root", mode => 644 } file{"/etc/ntp.conf": source => "puppet:///ntp/ntp.conf"; "/etc/ntp/step-tickers": source => "puppet:///ntp/step-tickers"; } } class ntp::service { service{"ntp": ensure => running, enable => true, require => Class["ntp::config"], } } class ntp { include ntp::install, ntp::config, ntp::service } vrijdag 27 april 12

Slide 64

Slide 64 text

Test your modules ‣ (Unit)test your modules ‣ Test them with: puppet apply --noop ‣ More advanced testing: cucumber / cucumber-puppet (BDD) vrijdag 27 april 12

Slide 65

Slide 65 text

What can puppet manage ‣ http://docs.puppetlabs.com/references/stable/type.html ‣ Almost everything. ‣ standard 48 different resource types ‣ Ranging from “ﬁle” to “cron” to “ssh_key” to “user” to “selinux”. ‣ Can control your Cisco routers and windows machines too (sortakinda) vrijdag 27 april 12

Slide 66

Slide 66 text

Confusing puppet things vrijdag 27 april 12

Slide 67

Slide 67 text

Confusing puppet things ‣ Puppet went from v0.25 to v2.6. ‣ REST interface since 2.6. XMLRPC before that. ‣ One binary to rule them all (puppet). ‣ Puppet v2.7 switched from GPLv2 to apache2.0 license. vrijdag 27 april 12

Slide 68

Slide 68 text

Confusing puppet things ‣ --test does not mean dry-run! (--noop does). ‣ It’s not object oriented. (puppet class != php class) ‣ It’s a declarative language. vrijdag 27 april 12

Slide 69

Slide 69 text

Puppet dashboards http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif vrijdag 27 april 12

Slide 70

Slide 70 text

Puppet dashboards http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif vrijdag 27 april 12

Slide 71

Slide 71 text

Live demo | MCollective? vrijdag 27 april 12

Slide 72

Slide 72 text

MCollective ‣ Puppet agent “calls” the master every 30 minutes. ‣ But what about realtime command & control? ‣ “Puppet kick”... (meh) ‣ MCollective (Marionette Collective) vrijdag 27 april 12

Slide 73

Slide 73 text

MCollective ‣ How do we handle large number of nodes? ‣ Which systems running a database and have 16GB or less? ‣ Which systems are using <50% of available memory? ‣ Restart all apache services in timezone GMT+5. vrijdag 27 april 12

Slide 74

Slide 74 text

MCollective ACTIVEMQ Client MCollective Server Node Middleware Client MCollective Server MCollective Server ‣ Middleware takes care of distribution, ‣ queued, broadcast etc.. Collective vrijdag 27 april 12

Slide 75

Slide 75 text

MCollective http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html ‣ The collective vrijdag 27 april 12

Slide 76

Slide 76 text

MCollective ‣ Filter out nodes based on facts $ mc-facts operatingsystem Report for fact: operatingsystem CentOS found 3 times Debian found 14 times Solaris found 4 times $ mc-facts -W operatingsystem=Centos operatingsystemrelease Report for fact: operatingsystemrelease 6.0 found 1 times 5.6 found 2 times vrijdag 27 april 12

Slide 77

Slide 77 text

MCollective - cool stuff ‣ Display all running processes ‣ Run or deploy software ‣ Restart services ‣ Start puppet agent ‣ Upgrade your systems vrijdag 27 april 12

Slide 78

Slide 78 text

Recap -ETOOMUCHINFO Let’s recap vrijdag 27 april 12

Slide 79

Slide 79 text

Recap (1) ‣ Conﬁguration management tool. ‣ Focusses on “what” instead of “how”. ‣ Scales from 1 to 100K+ systems. ‣ Uses descriptive manifests. vrijdag 27 april 12

Slide 80

Slide 80 text

Recap (2) ‣ Useful for sysadmins and developers. ‣ Keeps your infrastructure in sync. ‣ Keeps your infrastructure versioned. ‣ MCollective controls your hosts based on facts, not names. vrijdag 27 april 12

Slide 81

Slide 81 text

Any questions? http://farm1.static.ﬂickr.com/73/163450213_18478d3aa6_d.jpg vrijdag 27 april 12

Slide 82

Slide 82 text

to remove this comic sans font, please rate my talk on: http://joind.in/3781 vrijdag 27 april 12