Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet for dummies - 4developers

Puppet for dummies - 4developers

Puppet is a configuration management tool which allows easy deployment and configuration ranging from 1 to 1 thousand servers (and even more). Even though its common knowledge for devops, puppet is still a strange piece of software for developers. How does it work and what can it do for you as a developer? This talk is about the "other" side of development: the actual deployment of your software. It's not hard to have a VPS up and running at some cloud hosting company but when it comes to management of systems, many things needs to be taken care of. This talk is not so much about how operations and development can work together in a "devops" methodology, but how configuration management tools like Puppet, Vagrant, Veewee etc. can make deployment AND development easier.

Joshua Thijssen

April 18, 2012
Tweet

More Decks by Joshua Thijssen

Other Decks in Programming

Transcript

  1. http://joind.in/6328
    Puppet for
    Dummies
    4developers - 18 april 2012
    Poznań - Poland
    woensdag 18 april 12

    View Slide

  2. Joshua Thijssen
    Freelance consultant, developer and
    trainer @ NoxLogic / Techademy
    Development in PHP, Python, Perl,
    C, Java and some sysadmin
    Blog: http://adayinthelifeof.nl
    Email: [email protected]
    Twitter: @jaytaph
    oh hai!
    2
    woensdag 18 april 12

    View Slide

  3. 3
    woensdag 18 april 12

    View Slide

  4. What is puppet and why should I care?
    3
    woensdag 18 april 12

    View Slide

  5. What is puppet and why should I care?
    3
    (answer: it’s cool and because I told you so)
    woensdag 18 april 12

    View Slide

  6. “People are finally figuring out puppet and
    how it gets you to the pub by 4pm.
    Note that I’ve been at this pub since 2pm.”
    - Jorge Castro
    4
    woensdag 18 april 12

    View Slide

  7. 5
    woensdag 18 april 12

    View Slide

  8. Puppet is a (not necessarily the) solution for
    the following problem:
    How do we setup, manage, synchronize,
    and upgrade our internal and external
    infrastructure?
    6
    woensdag 18 april 12

    View Slide

  9. Sysadmin!
    Y U no fix problem!
    7
    woensdag 18 april 12

    View Slide

  10. Sysadmin!
    Y U no fix problem!
    NO
    7
    woensdag 18 april 12

    View Slide

  11. LAMP-stack
    8
    woensdag 18 april 12

    View Slide

  12. LAMP-stack
    Linux
    Apache
    MySQL
    PHP
    8
    woensdag 18 april 12

    View Slide

  13. LAMPGMVNMCSTRAH-stack
    9
    woensdag 18 april 12

    View Slide

  14. LAMPGMVNMCSTRAH-stack
    Linux
    Apache
    MySQL
    PHP
    Gearman
    MongoDB
    CouchDB
    Solr
    Tika
    Redis
    ActiveMQ
    Hadoop
    Varnish
    Ngnix
    Memcache
    9
    woensdag 18 april 12

    View Slide

  15. 10
    woensdag 18 april 12

    View Slide

  16. 10
    How do we control our infrastructure?
    woensdag 18 april 12

    View Slide

  17. ➡ Solution 1: We don’t,
    10
    How do we control our infrastructure?
    woensdag 18 april 12

    View Slide

  18. ➡ Solution 1: We don’t,
    ➡ Solution 2: We outsource,
    10
    How do we control our infrastructure?
    woensdag 18 april 12

    View Slide

  19. ➡ Solution 1: We don’t,
    ➡ Solution 2: We outsource,
    ➡ Solution 3: We automate the process.
    10
    How do we control our infrastructure?
    woensdag 18 april 12

    View Slide

  20. ‣ Solution 1: we don’t
    11
    woensdag 18 april 12

    View Slide

  21. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12

    View Slide

  22. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12

    View Slide

  23. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ➡ We only act on escalation
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12

    View Slide

  24. ➡ It’s not funny: you find it more often
    than not. Especially inside small
    development companies.
    ➡ Internal sysadmin, but he’s too busy
    with development to do sysadmin.
    ➡ We only act on escalation
    ➡ reactive, not proactive
    ‣ Solution 1: we don’t
    11
    woensdag 18 april 12

    View Slide

  25. ‣ Solution 2: we outsource
    12
    woensdag 18 april 12

    View Slide

  26. ➡ Expensive $LA’s.
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12

    View Slide

  27. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12

    View Slide

  28. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ➡ Fight between stability and agility.
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12

    View Slide

  29. ➡ Expensive $LA’s.
    ➡ What about INTERNAL servers like your
    development systems and
    infrastructure?
    ➡ Fight between stability and agility.
    ➡ Does your hosting company decide on
    whether you can use PHP5.3???
    ‣ Solution 2: we outsource
    12
    woensdag 18 april 12

    View Slide

  30. ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12

    View Slide

  31. ➡ We are in charge.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12

    View Slide

  32. ➡ We are in charge.
    ➡ You can do what you like
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12

    View Slide

  33. ➡ We are in charge.
    ➡ You can do what you like
    ➡ Use: cfEngine, chef, puppet.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12

    View Slide

  34. ➡ We are in charge.
    ➡ You can do what you like
    ➡ Use: cfEngine, chef, puppet.
    ➡ When done right, maintenance should
    not be difficult.
    ‣ Solution 3: we do it ourselves and automate
    13
    woensdag 18 april 12

    View Slide

  35. PUPPET
    14
    woensdag 18 april 12

    View Slide

  36. ➡ Open source configuration management tool.
    ➡ Written in Ruby
    ➡ Open source: https://github.com/puppetlabs
    ➡ Commercial version available (puppet enterprise)
    15
    woensdag 18 april 12

    View Slide

  37. ➡ Don’t tell HOW to do stuff.
    ➡ Tell WHAT to do.
    ¹
    ¹ It’s not actually true, but good enough for now...
    16
    woensdag 18 april 12

    View Slide

  38. ➡ Don’t tell HOW to do stuff.
    ➡ Tell WHAT to do.
    ¹
    ¹ It’s not actually true, but good enough for now...
    “yum install httpd”
    “apt-get install apache2”
    “install and run the apache webserver”
    16
    woensdag 18 april 12

    View Slide

  39. 17
    Schematic representation of a puppet infrastructure
    woensdag 18 april 12

    View Slide

  40. Puppet
    17
    Schematic representation of a puppet infrastructure
    woensdag 18 april 12

    View Slide

  41. Puppet CA
    Puppet
    Master
    Puppet
    Agent
    https
    18
    woensdag 18 april 12

    View Slide

  42. Puppet CA
    Puppet
    Master
    Puppet
    Agent
    Puppet
    Agent
    Puppet
    Agent
    https
    18
    woensdag 18 april 12

    View Slide

  43. Puppet
    master
    Puppet
    client
    19
    woensdag 18 april 12

    View Slide

  44. Puppet
    master
    Puppet
    client
    Check credentials
    19
    woensdag 18 april 12

    View Slide

  45. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    19
    woensdag 18 april 12

    View Slide

  46. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    Returns “catalog”
    19
    woensdag 18 april 12

    View Slide

  47. Puppet
    master
    Puppet
    client
    Check credentials
    Send facts
    Returns “catalog”
    Report results
    19
    woensdag 18 april 12

    View Slide

  48. ➡ Catalogs are “compiled” manifests
    ➡ Manifests are puppet definitions
    ➡ <filename>.pp
    ➡ Puppet DSL
    ➡ De-cla-ra-tive language
    ➡ Version your manifests! (git/svn)
    20
    woensdag 18 april 12

    View Slide

  49. package { “strace” :
    ensure => present,
    }
    file { “/home/jaytaph/secret-ingredient.txt” :
    ensure => present,
    mode => 0600,
    user => ‘jaytaph’,
    group => ‘noxlogic’,
    content => “beer”,
    }
    21
    woensdag 18 april 12

    View Slide

  50. package { “httpd” :
    ensure => present,
    }
    service { “httpd”:
    running => true,
    enable => true,
    }
    22
    woensdag 18 april 12

    View Slide

  51. package { “httpd” :
    ensure => present,
    }
    service { “httpd”:
    running => true,
    enable => true,
    }
    require => Package[“httpd”],
    22
    woensdag 18 april 12

    View Slide

  52. ‣ Different distributions, different names
    Centos / Redhat
    service: httpd
    package: httpd
    config: /etc/httpd/conf/httpd.conf
    vhosts: /etc/httpd/conf.d/*.conf
    Debian / Ubuntu
    service: apache2
    package: apache2
    config: /etc/apache2/httpd.conf
    vhosts: /etc/apache2/sites-available
    23
    woensdag 18 april 12

    View Slide

  53. $operatingsystem is a FACT
    package { “webserver”:
    case $operatingsystem {
    centos, redhat { $apache = “httpd” }
    debian, ubuntu { $apache = “apache2” }
    default : { fail(‘I don’t know this OS/distro’) }
    }
    name => $apache,
    ensure => installed,
    }
    24
    woensdag 18 april 12

    View Slide

  54. [[email protected] ~]# facter --puppet
    architecture => x86_64
    fqdn => puppetnode1.noxlogic.local
    interfaces => eth1,eth2,lo
    ipaddress_eth1 => 192.168.1.114
    ipaddress_eth2 => 192.168.56.200
    kernel => Linux
    kernelmajversion => 2.6
    operatingsystem => CentOS
    operatingsystemrelease => 6.0
    processor0 => Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz
    puppetversion => 2.6.9
    ‣ A simple list with info (also useable in your own tools)
    25
    woensdag 18 april 12

    View Slide

  55. node default {
    $def_packages = [ “mc”, “strace”, “sysstat” ]
    package { $def_packages :
    ensure => latest,
    }
    }
    /etc/puppet/manifests/site.pp:
    ‣ “Main” manifest
    26
    woensdag 18 april 12

    View Slide

  56. Defining nodes - regular expressions
    node /^web\d+\.example\.local$/ {
    package { “httpd” :
    ensure => latest,
    }
    }
    node /^db\d+\.example\.local$/ {
    package { “mysql-server” :
    ensure => installed,
    }
    }
    27
    woensdag 18 april 12

    View Slide

  57. node basenode {
    user { “jaytaph” :
    ensure => present,
    gid => 1000,
    uid => 1000,
    home => “/home/jaytaph”,
    shell => “/bin/sh”,
    password => “supersecrethashedpassword”,
    managehome => true,
    }
    }
    node /^.+\.example\.local/ inherits basenode {
    ...
    }
    ‣ Node inheritance
    28
    woensdag 18 april 12

    View Slide

  58. ‣ Group together into a class
    29
    woensdag 18 april 12

    View Slide

  59. class webserver {
    service { “apache”:
    ensure => running,
    require => Package[“apache”],
    }
    package { “apache” :
    ensure => installed,
    }
    }
    ‣ Group together into a class
    29
    woensdag 18 april 12

    View Slide

  60. class webserver {
    service { “apache”:
    ensure => running,
    require => Package[“apache”],
    }
    package { “apache” :
    ensure => installed,
    }
    }
    file { “vhost_${webserver_name}” :
    path => “/etc/httpd/conf/10-vhost.conf”,
    content => template(“vhost.template.erb”),
    notify => Service[“httpd”],
    }
    ‣ Group together into a class
    29
    woensdag 18 april 12

    View Slide

  61. :80>
    ServerName <%= webserver_name %>
    ServerAlias <%= webserver_alias %>
    DocumentRoot <%= webserver_docroot %>

    vhost.template.erb
    30
    ‣ ERB templates can contain custom variables and facts
    woensdag 18 april 12

    View Slide

  62. node “web01.example.local” inherits base {
    $webserver_name = “web01.example.local”
    $webserver_alias = “www.example.local”
    $webserver_docroot = “/var/www/web01”
    include webserver
    }
    node “web02.example.local” inherits base {
    $webserver_name = “web02.example.local”
    $webserver_alias = “crm.example.local”
    $webserver_docroot = “/var/www/web02”
    include webserver
    }
    31
    woensdag 18 april 12

    View Slide

  63. ➡ A puppet module is a collection of
    resources, classes, templates.
    ➡ Used for easy distribution and code-reuse.
    ➡ Self-contained, run out-of-the-box
    32
    woensdag 18 april 12

    View Slide

  64. ➡ puppetforge / github
    ➡ Create your own (and share!).
    ➡ Use the ones from puppet enterprise edition.
    ➡ Use the standard layout / best practices
    33
    woensdag 18 april 12

    View Slide

  65. class ntp::install {
    package{"ntpd":
    ensure => latest
    }
    }
    class ntp::config {
    File{
    require => Class["ntp::install"],
    notify => Class["ntp::service"],
    owner => "root",
    group => "root",
    mode => 644
    }
    file{"/etc/ntp.conf":
    source => "puppet:///ntp/ntp.conf";
    "/etc/ntp/step-tickers":
    source => "puppet:///ntp/step-tickers";
    }
    }
    class ntp::service {
    service{"ntp":
    ensure => running,
    enable => true,
    require => Class["ntp::config"],
    }
    }
    class ntp {
    include ntp::install, ntp::config, ntp::service
    }
    34
    woensdag 18 april 12

    View Slide

  66. ➡ (Unit)test your modules
    ➡ Test them with:
    puppet apply --noop
    ➡ More advanced testing: cucumber /
    cucumber-puppet (BDD)
    35
    woensdag 18 april 12

    View Slide

  67. http://docs.puppetlabs.com/references/stable/type.html
    ➡ Almost everything.
    ➡ standard 48 different resource types
    ➡ Ranging from “file” to “cron” to “ssh_key”
    to “user” to “selinux”.
    ➡ Can control your Cisco routers and
    windows machines too (sortakinda)
    36
    woensdag 18 april 12

    View Slide

  68. http://media.techtarget.com/digitalguide/images/Misc/puppetDashboard.gif
    37
    woensdag 18 april 12

    View Slide

  69. 38
    woensdag 18 april 12

    View Slide

  70. 39
    ➡ Puppet went from v0.25 to v2.6.
    ➡ REST interface since 2.6. XMLRPC before
    that.
    ➡ One binary to rule them all (puppet).
    ➡ Puppet v2.7 switched from GPLv2 to
    apache2.0 license.
    woensdag 18 april 12

    View Slide

  71. ➡ --test does not mean dry-run!
    (--noop does).
    ➡ It’s not object oriented. (puppet class !
    = php class)
    ➡ It’s a declarative language.
    40
    woensdag 18 april 12

    View Slide

  72. 41
    woensdag 18 april 12

    View Slide

  73. ➡ Puppet agent “calls” the master every 30 minutes.
    ➡ But what about realtime command & control?
    ➡ “Puppet kick”... (meh)
    ➡ MCollective (Marionette Collective)
    42
    woensdag 18 april 12

    View Slide

  74. ➡ Which systems running a database and have
    16GB or less?
    ➡ Which systems are using <50% of available
    memory?
    ➡ Restart all apache services in timezone
    GMT+5.
    43
    woensdag 18 april 12

    View Slide

  75. ACTIVEMQ
    Client
    MCollective
    Server
    Node
    Middleware
    Client
    MCollective
    Server
    MCollective
    Server
    ‣ Middleware takes care of distribution,
    ‣ queued, broadcast etc..
    Collective
    44
    woensdag 18 april 12

    View Slide

  76. http://docs.puppetlabs.com/mcollective/reference/basic/subcollectives.html
    45
    woensdag 18 april 12

    View Slide

  77. Filter out nodes based on facts
    $ mc-facts operatingsystem
    Report for fact: operatingsystem
    CentOS found 3 times
    Debian found 14 times
    Solaris found 4 times
    $ mc-facts -W operatingsystem=Centos operatingsystemrelease
    Report for fact: operatingsystemrelease
    6.0 found 1 times
    5.6 found 2 times
    46
    woensdag 18 april 12

    View Slide

  78. ➡ Display all running processes
    ➡ Run or deploy software
    ➡ Restart services
    ➡ Start puppet agent
    ➡ Upgrade your systems
    47
    woensdag 18 april 12

    View Slide

  79. -ETOOMUCHINFO
    Let’s recap
    48
    woensdag 18 april 12

    View Slide

  80. ➡ Configuration management tool.
    ➡ Focusses on “what” instead of “how”.
    ➡ Scales from 1 to 100K+ systems.
    ➡ Uses descriptive manifests.
    49
    woensdag 18 april 12

    View Slide

  81. ➡ Useful for sysadmins and developers.
    ➡ Keeps your infrastructure in sync.
    ➡ Keeps your infrastructure versioned.
    ➡ MCollective controls your hosts based
    on facts, not names.
    50
    woensdag 18 april 12

    View Slide

  82. There is no reason NOT to control your infrastructure.
    Having only 3 servers is NOT a reason.
    51
    You will be able to join the rest of us in the pub early.
    woensdag 18 april 12

    View Slide

  83. http://farm1.static.flickr.com/73/163450213_18478d3aa6_d.jpg 52
    woensdag 18 april 12

    View Slide

  84. Please rate my talk on joind.in:
    http://joind.in/6328
    Thank you
    53
    Find me on twitter: @jaytaph
    Find me for development and training: www.noxlogic.nl
    Find me on email: [email protected]
    Find me for blogs: www.adayinthelifeof.nl
    woensdag 18 april 12

    View Slide