Puppet for dummies - 4developers

Puppet is a configuration management tool which allows easy deployment and configuration ranging from 1 to 1 thousand servers (and even more). Even though its common knowledge for devops, puppet is still a strange piece of software for developers. How does it work and what can it do for you as a developer? This talk is about the "other" side of development: the actual deployment of your software. It's not hard to have a VPS up and running at some cloud hosting company but when it comes to management of systems, many things needs to be taken care of. This talk is not so much about how operations and development can work together in a "devops" methodology, but how configuration management tools like Puppet, Vagrant, Veewee etc. can make deployment AND development easier.

Transcript

1. http://joind.in/6328 Puppet for Dummies 4developers - 18 april 2012 Poznań - Poland woensdag 18 april 12
2. Joshua Thijssen Freelance consultant, developer and trainer @ NoxLogic / Techademy Development in PHP, Python, Perl, C, Java and some sysadmin Blog: http://adayinthelifeof.nl Email: jthijssen@noxlogic.nl Twitter: @jaytaph oh hai! 2 woensdag 18 april 12
3. 3 woensdag 18 april 12
4. What is puppet and why should I care? 3 woensdag 18 april 12
5. What is puppet and why should I care? 3 (answer: it’s cool and because I told you so) woensdag 18 april 12
6. “People are finally figuring out puppet and how it gets you to the pub by 4pm. Note that I’ve been at this pub since 2pm.” - Jorge Castro 4 woensdag 18 april 12
7. 5 woensdag 18 april 12
8. Puppet is a (not necessarily the) solution for the following problem: How do we setup, manage, synchronize, and upgrade our internal and external infrastructure? 6 woensdag 18 april 12
9. Sysadmin! Y U no fix problem! 7 woensdag 18 april 12
10. Sysadmin! Y U no fix problem! NO 7 woensdag 18 april 12
11. LAMP-stack 8 woensdag 18 april 12
12. LAMP-stack Linux Apache MySQL PHP 8 woensdag 18 april 12
13. LAMPGMVNMCSTRAH-stack 9 woensdag 18 april 12
14. LAMPGMVNMCSTRAH-stack Linux Apache MySQL PHP Gearman MongoDB CouchDB Solr Tika Redis ActiveMQ Hadoop Varnish Ngnix Memcache 9 woensdag 18 april 12
15. 10 woensdag 18 april 12
16. 10 How do we control our infrastructure? woensdag 18 april 12
17. ➡ Solution 1: We don’t, 10 How do we control our infrastructure? woensdag 18 april 12
18. ➡ Solution 1: We don’t, ➡ Solution 2: We outsource, 10 How do we control our infrastructure? woensdag 18 april 12
19. ➡ Solution 1: We don’t, ➡ Solution 2: We outsource, ➡ Solution 3: We automate the process. 10 How do we control our infrastructure? woensdag 18 april 12
20. ‣ Solution 1: we don’t 11 woensdag 18 april 12
21. ➡ It’s not funny: you find it more often than not. Especially inside small development companies. ‣ Solution 1: we don’t 11 woensdag 18 april 12
22. ➡ It’s not funny: you find it more often than not. Especially inside small development companies. ➡ Internal sysadmin, but he’s too busy with development to do sysadmin. ‣ Solution 1: we don’t 11 woensdag 18 april 12
23. ➡ It’s not funny: you find it more often than not. Especially inside small development companies. ➡ Internal sysadmin, but he’s too busy with development to do sysadmin. ➡ We only act on escalation ‣ Solution 1: we don’t 11 woensdag 18 april 12
24. ➡ It’s not funny: you find it more often than not. Especially inside small development companies. ➡ Internal sysadmin, but he’s too busy with development to do sysadmin. ➡ We only act on escalation ➡ reactive, not proactive ‣ Solution 1: we don’t 11 woensdag 18 april 12
25. ‣ Solution 2: we outsource 12 woensdag 18 april 12
26. ➡ Expensive $LA’s. ‣ Solution 2: we outsource 12 woensdag 18 april 12
27. ➡ Expensive $LA’s. ➡ What about INTERNAL servers like your development systems and infrastructure? ‣ Solution 2: we outsource 12 woensdag 18 april 12
28. ➡ Expensive $LA’s. ➡ What about INTERNAL servers like your development systems and infrastructure? ➡ Fight between stability and agility. ‣ Solution 2: we outsource 12 woensdag 18 april 12
29. ➡ Expensive $LA’s. ➡ What about INTERNAL servers like your development systems and infrastructure? ➡ Fight between stability and agility. ➡ Does your hosting company decide on whether you can use PHP5.3??? ‣ Solution 2: we outsource 12 woensdag 18 april 12
30. ‣ Solution 3: we do it ourselves and automate 13 woensdag 18 april 12
31. ➡ We are in charge. ‣ Solution 3: we do it ourselves and automate 13 woensdag 18 april 12
32. ➡ We are in charge. ➡ You can do what you like ‣ Solution 3: we do it ourselves and automate 13 woensdag 18 april 12
33. ➡ We are in charge. ➡ You can do what you like ➡ Use: cfEngine, chef, puppet. ‣ Solution 3: we do it ourselves and automate 13 woensdag 18 april 12
34. ➡ We are in charge. ➡ You can do what you like ➡ Use: cfEngine, chef, puppet. ➡ When done right, maintenance should not be difficult. ‣ Solution 3: we do it ourselves and automate 13 woensdag 18 april 12
35. PUPPET 14 woensdag 18 april 12
36. ➡ Open source configuration management tool. ➡ Written in Ruby ➡ Open source: https://github.com/puppetlabs ➡ Commercial version available (puppet enterprise) 15 woensdag 18 april 12
37. ➡ Don’t tell HOW to do stuff. ➡ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... 16 woensdag 18 april 12
38. ➡ Don’t tell HOW to do stuff. ➡ Tell WHAT to do. ¹ ¹ It’s not actually true, but good enough for now... “yum install httpd” “apt-get install apache2” “install and run the apache webserver” 16 woensdag 18 april 12
39. 17 Schematic representation of a puppet infrastructure woensdag 18 april 12
40. Puppet 17 Schematic representation of a puppet infrastructure woensdag 18 april 12
41. Puppet CA Puppet Master Puppet Agent https 18 woensdag 18 april 12
42. Puppet CA Puppet Master Puppet Agent Puppet Agent Puppet Agent https 18 woensdag 18 april 12
43. Puppet master Puppet client 19 woensdag 18 april 12
44. Puppet master Puppet client Check credentials 19 woensdag 18 april 12
45. Puppet master Puppet client Check credentials Send facts 19 woensdag 18 april 12
46. Puppet master Puppet client Check credentials Send facts Returns “catalog” 19 woensdag 18 april 12
47. Puppet master Puppet client Check credentials Send facts Returns “catalog” Report results 19 woensdag 18 april 12
48. ➡ Catalogs are “compiled” manifests ➡ Manifests are puppet definitions ➡ <filename>.pp ➡ Puppet DSL ➡ De-cla-ra-tive language ➡ Version your manifests! (git/svn) 20 woensdag 18 april 12
49. package { “strace” : ensure => present, } file { “/home/jaytaph/secret-ingredient.txt” : ensure => present, mode => 0600, user => ‘jaytaph’, group => ‘noxlogic’, content => “beer”, } 21 woensdag 18 april 12
50. package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, } 22 woensdag 18 april 12
51. package { “httpd” : ensure => present, } service { “httpd”: running => true, enable => true, } require => Package[“httpd”], 22 woensdag 18 april 12
52. ‣ Different distributions, different names Centos / Redhat service: httpd package: httpd config: /etc/httpd/conf/httpd.conf vhosts: /etc/httpd/conf.d/*.conf Debian / Ubuntu service: apache2 package: apache2 config: /etc/apache2/httpd.conf vhosts: /etc/apache2/sites-available 23 woensdag 18 april 12
53. $operatingsystem is a FACT package { “webserver”: case $operatingsystem { centos, redhat { $apache = “httpd” } debian, ubuntu { $apache = “apache2” } default : { fail(‘I don’t know this OS/distro’) } } name => $apache, ensure => installed, } 24 woensdag 18 april 12
54. [root@puppetnode1 ~]# facter --puppet architecture => x86_64 fqdn => puppetnode1.noxlogic.local interfaces => eth1,eth2,lo ipaddress_eth1 => 192.168.1.114 ipaddress_eth2 => 192.168.56.200 kernel => Linux kernelmajversion => 2.6 operatingsystem => CentOS operatingsystemrelease => 6.0 processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz puppetversion => 2.6.9 ‣ A simple list with info (also useable in your own tools) 25 woensdag 18 april 12
55. node default { $def_packages = [ “mc”, “strace”, “sysstat” ] package { $def_packages : ensure => latest, } } /etc/puppet/manifests/site.pp: ‣ “Main” manifest 26 woensdag 18 april 12
56. Defining nodes - regular expressions node /^web\d+\.example\.local$/ { package { “httpd” : ensure => latest, } } node /^db\d+\.example\.local$/ { package { “mysql-server” : ensure => installed, } } 27 woensdag 18 april 12
57. node basenode { user { “jaytaph” : ensure => present, gid => 1000, uid => 1000, home => “/home/jaytaph”, shell => “/bin/sh”, password => “supersecrethashedpassword”, managehome => true, } } node /^.+\.example\.local/ inherits basenode { ... } ‣ Node inheritance 28 woensdag 18 april 12
58. ‣ Group together into a class 29 woensdag 18 april 12
59. class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } ‣ Group together into a class 29 woensdag 18 april 12
60. class webserver { service { “apache”: ensure => running, require => Package[“apache”], } package { “apache” : ensure => installed, } } file { “vhost_${webserver_name}” : path => “/etc/httpd/conf/10-vhost.conf”, content => template(“vhost.template.erb”), notify => Service[“httpd”], } ‣ Group together into a class 29 woensdag 18 april 12
61. <virtualHost <%= ipaddress %>:80> ServerName <%= webserver_name %> ServerAlias <%= webserver_alias %> DocumentRoot <%= webserver_docroot %> </virtualHost> vhost.template.erb 30 ‣ ERB templates can contain custom variables and facts woensdag 18 april 12
62. node “web01.example.local” inherits base { $webserver_name = “web01.example.local” $webserver_alias = “www.example.local” $webserver_docroot = “/var/www/web01” include webserver } node “web02.example.local” inherits base { $webserver_name = “web02.example.local” $webserver_alias = “crm.example.local” $webserver_docroot = “/var/www/web02” include webserver } 31 woensdag 18 april 12
63. ➡ A puppet module is a collection of resources, classes, templates. ➡ Used for easy distribution and code-reuse. ➡ Self-contained, run out-of-the-box 32 woensdag 18 april 12
64. ➡ puppetforge / github ➡ Create your own (and share!). ➡ Use the ones from puppet enterprise edition. ➡ Use the standard layout / best practices 33 woensdag 18 april 12
65. class ntp::install { package{"ntpd": ensure => latest } } class ntp::config { File{ require => Class["ntp::install"], notify => Class["ntp::service"], owner => "root", group => "root", mode => 644 } file{"/etc/ntp.conf": source => "puppet:///ntp/ntp.conf"; "/etc/ntp/step-tickers": source => "puppet:///ntp/step-tickers"; } } class ntp::service { service{"ntp": ensure => running, enable => true, require => Class["ntp::config"], } } class ntp { include ntp::install, ntp::config, ntp::service } 34 woensdag 18 april 12
66. ➡ (Unit)test your modules ➡ Test them with: puppet apply --noop ➡ More advanced testing: cucumber / cucumber-puppet (BDD) 35 woensdag 18 april 12
67. http://docs.puppetlabs.com/references/stable/type.html ➡ Almost everything. ➡ standard 48 different resource types ➡ Ranging from “file” to “cron” to “ssh_key” to “user” to “selinux”. ➡ Can control your Cisco routers and windows machines too (sortakinda) 36 woensdag 18 april 12
68. http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif 37 woensdag 18 april 12
69. 38 woensdag 18 april 12
70. 39 ➡ Puppet went from v0.25 to v2.6. ➡ REST interface since 2.6. XMLRPC before that. ➡ One binary to rule them all (puppet). ➡ Puppet v2.7 switched from GPLv2 to apache2.0 license. woensdag 18 april 12
71. ➡ --test does not mean dry-run! (--noop does). ➡ It’s not object oriented. (puppet class ! = php class) ➡ It’s a declarative language. 40 woensdag 18 april 12
72. 41 woensdag 18 april 12
73. ➡ Puppet agent “calls” the master every 30 minutes. ➡ But what about realtime command & control? ➡ “Puppet kick”... (meh) ➡ MCollective (Marionette Collective) 42 woensdag 18 april 12
74. ➡ Which systems running a database and have 16GB or less? ➡ Which systems are using <50% of available memory? ➡ Restart all apache services in timezone GMT+5. 43 woensdag 18 april 12
75. ACTIVEMQ Client MCollective Server Node Middleware Client MCollective Server MCollective Server ‣ Middleware takes care of distribution, ‣ queued, broadcast etc.. Collective 44 woensdag 18 april 12
76. http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html 45 woensdag 18 april 12
77. Filter out nodes based on facts $ mc-facts operatingsystem Report for fact: operatingsystem CentOS found 3 times Debian found 14 times Solaris found 4 times $ mc-facts -W operatingsystem=Centos operatingsystemrelease Report for fact: operatingsystemrelease 6.0 found 1 times 5.6 found 2 times 46 woensdag 18 april 12
78. ➡ Display all running processes ➡ Run or deploy software ➡ Restart services ➡ Start puppet agent ➡ Upgrade your systems 47 woensdag 18 april 12
79. -ETOOMUCHINFO Let’s recap 48 woensdag 18 april 12
80. ➡ Configuration management tool. ➡ Focusses on “what” instead of “how”. ➡ Scales from 1 to 100K+ systems. ➡ Uses descriptive manifests. 49 woensdag 18 april 12
81. ➡ Useful for sysadmins and developers. ➡ Keeps your infrastructure in sync. ➡ Keeps your infrastructure versioned. ➡ MCollective controls your hosts based on facts, not names. 50 woensdag 18 april 12
82. There is no reason NOT to control your infrastructure. Having only 3 servers is NOT a reason. 51 You will be able to join the rest of us in the pub early. woensdag 18 april 12
83. http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg 52 woensdag 18 april 12
84. Please rate my talk on joind.in: http://joind.in/6328 Thank you 53 Find me on twitter: @jaytaph Find me for development and training: www.noxlogic.nl Find me on email: jthijssen@noxlogic.nl Find me for blogs: www.adayinthelifeof.nl woensdag 18 april 12