Puppet for dummies - 4developers

Puppet is a configuration management tool which allows easy deployment and configuration ranging from 1 to 1 thousand servers (and even more). Even though its common knowledge for devops, puppet is still a strange piece of software for developers. How does it work and what can it do for you as a developer? This talk is about the "other" side of development: the actual deployment of your software. It's not hard to have a VPS up and running at some cloud hosting company but when it comes to management of systems, many things needs to be taken care of. This talk is not so much about how operations and development can work together in a "devops" methodology, but how configuration management tools like Puppet, Vagrant, Veewee etc. can make deployment AND development easier.

Transcript

1. http://joind.in/6328
   Puppet for
   Dummies
   4developers - 18 april 2012
   Poznań - Poland
   woensdag 18 april 12
   

   View Slide

2. Joshua Thijssen
   Freelance consultant, developer and
   trainer @ NoxLogic / Techademy
   Development in PHP, Python, Perl,
   C, Java and some sysadmin
   Blog: http://adayinthelifeof.nl
   Email: jthijssen@noxlogic.nl
   Twitter: @jaytaph
   oh hai!
   2
   woensdag 18 april 12
   

   View Slide

3. 3
   woensdag 18 april 12
   

   View Slide

4. What is puppet and why should I care?
   3
   woensdag 18 april 12
   

   View Slide

5. What is puppet and why should I care?
   3
   (answer: it’s cool and because I told you so)
   woensdag 18 april 12
   

   View Slide

6. “People are finally figuring out puppet and
   how it gets you to the pub by 4pm.
   Note that I’ve been at this pub since 2pm.”
   - Jorge Castro
   4
   woensdag 18 april 12
   

   View Slide

7. 5
   woensdag 18 april 12
   

   View Slide

8. Puppet is a (not necessarily the) solution for
   the following problem:
   How do we setup, manage, synchronize,
   and upgrade our internal and external
   infrastructure?
   6
   woensdag 18 april 12
   

   View Slide

9. Sysadmin!
   Y U no fix problem!
   7
   woensdag 18 april 12
   

   View Slide

10. Sysadmin!
    Y U no fix problem!
    NO
    7
    woensdag 18 april 12
    

    View Slide

11. LAMP-stack
    8
    woensdag 18 april 12
    

    View Slide

12. LAMP-stack
    Linux
    Apache
    MySQL
    PHP
    8
    woensdag 18 april 12
    

    View Slide

13. LAMPGMVNMCSTRAH-stack
    9
    woensdag 18 april 12
    

    View Slide

14. LAMPGMVNMCSTRAH-stack
    Linux
    Apache
    MySQL
    PHP
    Gearman
    MongoDB
    CouchDB
    Solr
    Tika
    Redis
    ActiveMQ
    Hadoop
    Varnish
    Ngnix
    Memcache
    9
    woensdag 18 april 12
    

    View Slide

15. 10
    woensdag 18 april 12
    

    View Slide

16. 10
    How do we control our infrastructure?
    woensdag 18 april 12
    

    View Slide

17. ➡ Solution 1: We don’t,
    10
    How do we control our infrastructure?
    woensdag 18 april 12
    

    View Slide

18. ➡ Solution 1: We don’t,
    ➡ Solution 2: We outsource,
    10
    How do we control our infrastructure?
    woensdag 18 april 12
    

    View Slide

19. ➡ Solution 1: We don’t,
    ➡ Solution 2: We outsource,
    ➡ Solution 3: We automate the process.
    10
    How do we control our infrastructure?
    woensdag 18 april 12
    

    View Slide

20. ‣ Solution 1: we don’t
    11
    woensdag 18 april 12
    

    View Slide

21. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12
    

    View Slide

22. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12
    

    View Slide

23. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ➡ We only act on escalation
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12
    

    View Slide

24. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ➡ We only act on escalation
    ➡ reactive, not proactive
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12
    

    View Slide

25. ‣ Solution 2: we outsource
    12
    woensdag 18 april 12
    

    View Slide

26. ➡ Expensive $LA’s.
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12
    

    View Slide

27. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12
    

    View Slide

28. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ➡ Fight between stability and agility.
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12
    

    View Slide

29. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ➡ Fight between stability and agility.
    ➡ Does your hosting company decide on
    whether you can use PHP5.3???
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12
    

    View Slide

30. ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12
    

    View Slide

31. ➡ We are in charge.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12
    

    View Slide

32. ➡ We are in charge.
    ➡ You can do what you like
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12
    

    View Slide

33. ➡ We are in charge.
    ➡ You can do what you like
    ➡ Use: cfEngine, chef, puppet.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12
    

    View Slide

34. ➡ We are in charge.
    ➡ You can do what you like
    ➡ Use: cfEngine, chef, puppet.
    ➡ When done right, maintenance should
    not be difficult.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12
    

    View Slide

35. PUPPET
    14
    woensdag 18 april 12
    

    View Slide

36. ➡ Open source configuration management tool.
    ➡ Written in Ruby
    ➡ Open source: https://github.com/puppetlabs
    ➡ Commercial version available (puppet enterprise)
    15
    woensdag 18 april 12
    

    View Slide

37. ➡ Don’t tell HOW to do stuff.
    ➡ Tell WHAT to do.
    ¹
    ¹ It’s not actually true, but good enough for now...
    16
    woensdag 18 april 12
    

    View Slide

38. ➡ Don’t tell HOW to do stuff.
    ➡ Tell WHAT to do.
    ¹
    ¹ It’s not actually true, but good enough for now...
    “yum install httpd”
    “apt-get install apache2”
    “install and run the apache webserver”
    16
    woensdag 18 april 12
    

    View Slide

39. 17
    Schematic representation of a puppet infrastructure
    woensdag 18 april 12
    

    View Slide

40. Puppet
    17
    Schematic representation of a puppet infrastructure
    woensdag 18 april 12
    

    View Slide

41. Puppet CA
    Puppet
    Master
    Puppet
    Agent
    https
    18
    woensdag 18 april 12
    

    View Slide

42. Puppet CA
    Puppet
    Master
    Puppet
    Agent
    Puppet
    Agent
    Puppet
    Agent
    https
    18
    woensdag 18 april 12
    

    View Slide

43. Puppet
    master
    Puppet
    client
    19
    woensdag 18 april 12
    

    View Slide

44. Puppet
    master
    Puppet
    client
    Check credentials
    19
    woensdag 18 april 12
    

    View Slide

45. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    19
    woensdag 18 april 12
    

    View Slide

46. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    Returns “catalog”
    19
    woensdag 18 april 12
    

    View Slide

47. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    Returns “catalog”
    Report results
    19
    woensdag 18 april 12
    

    View Slide

48. ➡ Catalogs are “compiled” manifests
    ➡ Manifests are puppet definitions
    ➡ <filename>.pp
    ➡ Puppet DSL
    ➡ De-cla-ra-tive language
    ➡ Version your manifests! (git/svn)
    20
    woensdag 18 april 12
    

    View Slide

49. package { “strace” :
    ensure => present,
    }
    file { “/home/jaytaph/secret-ingredient.txt” :
    ensure => present,
    mode => 0600,
    user => ‘jaytaph’,
    group => ‘noxlogic’,
    content => “beer”,
    }
    21
    woensdag 18 april 12
    

    View Slide

50. package { “httpd” :
    ensure => present,
    }
    service { “httpd”:
    running => true,
    enable => true,
    }
    22
    woensdag 18 april 12
    

    View Slide

51. package { “httpd” :
    ensure => present,
    }
    service { “httpd”:
    running => true,
    enable => true,
    }
    require => Package[“httpd”],
    22
    woensdag 18 april 12
    

    View Slide

52. ‣ Different distributions, different names
    Centos / Redhat
    service: httpd
    package: httpd
    config: /etc/httpd/conf/httpd.conf
    vhosts: /etc/httpd/conf.d/*.conf
    Debian / Ubuntu
    service: apache2
    package: apache2
    config: /etc/apache2/httpd.conf
    vhosts: /etc/apache2/sites-available
    23
    woensdag 18 april 12
    

    View Slide

53. $operatingsystem is a FACT
    package { “webserver”:
    case $operatingsystem {
    centos, redhat { $apache = “httpd” }
    debian, ubuntu { $apache = “apache2” }
    default : { fail(‘I don’t know this OS/distro’) }
    }
    name => $apache,
    ensure => installed,
    }
    24
    woensdag 18 april 12
    

    View Slide

54. [root@puppetnode1 ~]# facter --puppet
    architecture => x86_64
    fqdn => puppetnode1.noxlogic.local
    interfaces => eth1,eth2,lo
    ipaddress_eth1 => 192.168.1.114
    ipaddress_eth2 => 192.168.56.200
    kernel => Linux
    kernelmajversion => 2.6
    operatingsystem => CentOS
    operatingsystemrelease => 6.0
    processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz
    puppetversion => 2.6.9
    ‣ A simple list with info (also useable in your own tools)
    25
    woensdag 18 april 12
    

    View Slide

55. node default {
    $def_packages = [ “mc”, “strace”, “sysstat” ]
    package { $def_packages :
    ensure => latest,
    }
    }
    /etc/puppet/manifests/site.pp:
    ‣ “Main” manifest
    26
    woensdag 18 april 12
    

    View Slide

56. Defining nodes - regular expressions
    node /^web\d+\.example\.local$/ {
    package { “httpd” :
    ensure => latest,
    }
    }
    node /^db\d+\.example\.local$/ {
    package { “mysql-server” :
    ensure => installed,
    }
    }
    27
    woensdag 18 april 12
    

    View Slide

57. node basenode {
    user { “jaytaph” :
    ensure => present,
    gid => 1000,
    uid => 1000,
    home => “/home/jaytaph”,
    shell => “/bin/sh”,
    password => “supersecrethashedpassword”,
    managehome => true,
    }
    }
    node /^.+\.example\.local/ inherits basenode {
    ...
    }
    ‣ Node inheritance
    28
    woensdag 18 april 12
    

    View Slide

58. ‣ Group together into a class
    29
    woensdag 18 april 12
    

    View Slide

59. class webserver {
    service { “apache”:
    ensure => running,
    require => Package[“apache”],
    }
    package { “apache” :
    ensure => installed,
    }
    }
    ‣ Group together into a class
    29
    woensdag 18 april 12
    

    View Slide

60. class webserver {
    service { “apache”:
    ensure => running,
    require => Package[“apache”],
    }
    package { “apache” :
    ensure => installed,
    }
    }
    file { “vhost_${webserver_name}” :
    path => “/etc/httpd/conf/10-vhost.conf”,
    content => template(“vhost.template.erb”),
    notify => Service[“httpd”],
    }
    ‣ Group together into a class
    29
    woensdag 18 april 12
    

    View Slide

61. :80>
    ServerName <%= webserver_name %>
    ServerAlias <%= webserver_alias %>
    DocumentRoot <%= webserver_docroot %>
    
    vhost.template.erb
    30
    ‣ ERB templates can contain custom variables and facts
    woensdag 18 april 12
    

    View Slide

62. node “web01.example.local” inherits base {
    $webserver_name = “web01.example.local”
    $webserver_alias = “www.example.local”
    $webserver_docroot = “/var/www/web01”
    include webserver
    }
    node “web02.example.local” inherits base {
    $webserver_name = “web02.example.local”
    $webserver_alias = “crm.example.local”
    $webserver_docroot = “/var/www/web02”
    include webserver
    }
    31
    woensdag 18 april 12
    

    View Slide

63. ➡ A puppet module is a collection of
    resources, classes, templates.
    ➡ Used for easy distribution and code-reuse.
    ➡ Self-contained, run out-of-the-box
    32
    woensdag 18 april 12
    

    View Slide

64. ➡ puppetforge / github
    ➡ Create your own (and share!).
    ➡ Use the ones from puppet enterprise edition.
    ➡ Use the standard layout / best practices
    33
    woensdag 18 april 12
    

    View Slide

65. class ntp::install {
    package{"ntpd":
    ensure => latest
    }
    }
    class ntp::config {
    File{
    require => Class["ntp::install"],
    notify => Class["ntp::service"],
    owner => "root",
    group => "root",
    mode => 644
    }
    file{"/etc/ntp.conf":
    source => "puppet:///ntp/ntp.conf";
    "/etc/ntp/step-tickers":
    source => "puppet:///ntp/step-tickers";
    }
    }
    class ntp::service {
    service{"ntp":
    ensure => running,
    enable => true,
    require => Class["ntp::config"],
    }
    }
    class ntp {
    include ntp::install, ntp::config, ntp::service
    }
    34
    woensdag 18 april 12
    

    View Slide

66. ➡ (Unit)test your modules
    ➡ Test them with:
    puppet apply --noop
    ➡ More advanced testing: cucumber /
    cucumber-puppet (BDD)
    35
    woensdag 18 april 12
    

    View Slide

67. http://docs.puppetlabs.com/references/stable/type.html
    ➡ Almost everything.
    ➡ standard 48 different resource types
    ➡ Ranging from “file” to “cron” to “ssh_key”
    to “user” to “selinux”.
    ➡ Can control your Cisco routers and
    windows machines too (sortakinda)
    36
    woensdag 18 april 12
    

    View Slide

68. http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif
    37
    woensdag 18 april 12
    

    View Slide

69. 38
    woensdag 18 april 12
    

    View Slide

70. 39
    ➡ Puppet went from v0.25 to v2.6.
    ➡ REST interface since 2.6. XMLRPC before
    that.
    ➡ One binary to rule them all (puppet).
    ➡ Puppet v2.7 switched from GPLv2 to
    apache2.0 license.
    woensdag 18 april 12
    

    View Slide

71. ➡ --test does not mean dry-run!
    (--noop does).
    ➡ It’s not object oriented. (puppet class !
    = php class)
    ➡ It’s a declarative language.
    40
    woensdag 18 april 12
    

    View Slide

72. 41
    woensdag 18 april 12
    

    View Slide

73. ➡ Puppet agent “calls” the master every 30 minutes.
    ➡ But what about realtime command & control?
    ➡ “Puppet kick”... (meh)
    ➡ MCollective (Marionette Collective)
    42
    woensdag 18 april 12
    

    View Slide

74. ➡ Which systems running a database and have
    16GB or less?
    ➡ Which systems are using <50% of available
    memory?
    ➡ Restart all apache services in timezone
    GMT+5.
    43
    woensdag 18 april 12
    

    View Slide

75. ACTIVEMQ
    Client
    MCollective
    Server
    Node
    Middleware
    Client
    MCollective
    Server
    MCollective
    Server
    ‣ Middleware takes care of distribution,
    ‣ queued, broadcast etc..
    Collective
    44
    woensdag 18 april 12
    

    View Slide

76. http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html
    45
    woensdag 18 april 12
    

    View Slide

77. Filter out nodes based on facts
    $ mc-facts operatingsystem
    Report for fact: operatingsystem
    CentOS found 3 times
    Debian found 14 times
    Solaris found 4 times
    $ mc-facts -W operatingsystem=Centos operatingsystemrelease
    Report for fact: operatingsystemrelease
    6.0 found 1 times
    5.6 found 2 times
    46
    woensdag 18 april 12
    

    View Slide

78. ➡ Display all running processes
    ➡ Run or deploy software
    ➡ Restart services
    ➡ Start puppet agent
    ➡ Upgrade your systems
    47
    woensdag 18 april 12
    

    View Slide

79. -ETOOMUCHINFO
    Let’s recap
    48
    woensdag 18 april 12
    

    View Slide

80. ➡ Configuration management tool.
    ➡ Focusses on “what” instead of “how”.
    ➡ Scales from 1 to 100K+ systems.
    ➡ Uses descriptive manifests.
    49
    woensdag 18 april 12
    

    View Slide

81. ➡ Useful for sysadmins and developers.
    ➡ Keeps your infrastructure in sync.
    ➡ Keeps your infrastructure versioned.
    ➡ MCollective controls your hosts based
    on facts, not names.
    50
    woensdag 18 april 12
    

    View Slide

82. There is no reason NOT to control your infrastructure.
    Having only 3 servers is NOT a reason.
    51
    You will be able to join the rest of us in the pub early.
    woensdag 18 april 12
    

    View Slide

83. http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg 52
    woensdag 18 april 12
    

    View Slide

84. Please rate my talk on joind.in:
    http://joind.in/6328
    Thank you
    53
    Find me on twitter: @jaytaph
    Find me for development and training: www.noxlogic.nl
    Find me on email: jthijssen@noxlogic.nl
    Find me for blogs: www.adayinthelifeof.nl
    woensdag 18 april 12
    

    View Slide