$30 off During Our Annual Pro Sale. View Details »

Building Data-Driven Infrastructure with Puppet

Building Data-Driven Infrastructure with Puppet

The road to Data Driven Infrastructure is a hard and long one, but a worthwhile endeavor in the lifecycle of Information Technology and Business Enablement. This talk is a snapshot of the voyage of one man's quest to have computers absolutely and completely take over his job, the patterns that have been uncovered, and the things we as an industry need to keep pushing on.

This talk can be viewed online at https://www.youtube.com/watch?v=vEpXJRPITN0&list=PLV86BgbREluU02Ytlz80seDSKAbkx5pRg

James Fryman

August 22, 2013
Tweet

More Decks by James Fryman

Other Decks in Technology

Transcript

  1. Friday, August 23, 13

    View Slide

  2. Data
    Driven
    Infrastructure
    Friday, August 23, 13

    View Slide

  3. Friday, August 23, 13

    View Slide

  4. ?
    Friday, August 23, 13

    View Slide

  5. ?
    http://git.io/UmUdKA
    Friday, August 23, 13

    View Slide

  6. Let me
    Tell you
    a story
    Friday, August 23, 13

    View Slide

  7. What is a
    Cloud
    Friday, August 23, 13

    View Slide

  8. I don’t have to think about IT
    With the Cloud...
    Friday, August 23, 13

    View Slide

  9. Friday, August 23, 13

    View Slide

  10. aaS
    Platform
    Software
    Infrastructure
    Friday, August 23, 13

    View Slide

  11. Friday, August 23, 13

    View Slide

  12. Friday, August 23, 13

    View Slide

  13. Where is
    our cloud?
    Friday, August 23, 13

    View Slide

  14. Friday, August 23, 13

    View Slide

  15. Utopia
    Friday, August 23, 13

    View Slide

  16. James Fryman
    Friday, August 23, 13

    View Slide

  17. Friday, August 23, 13

    View Slide

  18. Automations
    Junkie
    Friday, August 23, 13

    View Slide

  19. Are you
    Crazy?
    Friday, August 23, 13

    View Slide

  20. Goals?
    Data?
    What
    Friday, August 23, 13

    View Slide

  21. Goals?
    Data?
    What
    Friday, August 23, 13

    View Slide

  22. Goals?
    Data?
    What
    Friday, August 23, 13

    View Slide

  23. Machine
    Parsable
    Friday, August 23, 13

    View Slide

  24. There is System
    One
    Friday, August 23, 13

    View Slide

  25. Feedback
    PuppetDB
    Provisioning
    Configuring
    Destroying
    gPanel
    Friday, August 23, 13

    View Slide

  26. Parts
    of the
    System
    Friday, August 23, 13

    View Slide

  27. provisioner
    Friday, August 23, 13

    View Slide

  28. Home
    Grown
    Friday, August 23, 13

    View Slide

  29. Home Grown
    Friday, August 23, 13

    View Slide

  30. controller
    Friday, August 23, 13

    View Slide

  31. Lots O’ Models
    Friday, August 23, 13

    View Slide

  32. file { '/etc/facter/facts.d/quagga_manage_service':
    ensure => present,
    replace => false,
    }
    if $::quagga_manage_service {
    $service_state[ensure] = running
    $service_state[enable] = true
    } else {
    $service_state[ensure] = undef
    $service_state[enable] = undef
    }
    service { 'quagga':
    ensure => $service_state[ensure],
    enable => $service_state[enable],
    }
    Friday, August 23, 13

    View Slide

  33. orchestrator
    Friday, August 23, 13

    View Slide

  34. Chat
    Ops
    Friday, August 23, 13

    View Slide

  35. Chat
    Ops
    Friday, August 23, 13

    View Slide

  36. Feedback
    of the
    System
    Friday, August 23, 13

    View Slide

  37. Metrics&
    Monitoring
    Friday, August 23, 13

    View Slide

  38. begin
    JSON.parse(RestClient.get(url)).each do |cache|
    data["#{cache['target']}"] = 0
    count = 0
    cache["datapoints"].each do |point|
    unless (point[0].nil?)
    data["#{cache['target']}"] += point[0]
    count += 1
    end
    end
    data["#{cache['target']}"] /= count
    data["total"] += data["#{cache['target']}"]
    end
    rescue ZeroDivisionError => e
    puts "UNKNOWN metric not in graphite!"
    exit EXIT_UNKNOWN
    rescue => e
    puts e.message
    exit EXIT_CRITICAL
    end
    Friday, August 23, 13

    View Slide

  39. begin
    JSON.parse(RestClient.get(url)).each do |cache|
    data["#{cache['target']}"] = 0
    count = 0
    cache["datapoints"].each do |point|
    unless (point[0].nil?)
    data["#{cache['target']}"] += point[0]
    count += 1
    end
    end
    data["#{cache['target']}"] /= count
    data["total"] += data["#{cache['target']}"]
    end
    rescue ZeroDivisionError => e
    puts "UNKNOWN metric not in graphite!"
    exit EXIT_UNKNOWN
    rescue => e
    puts e.message
    exit EXIT_CRITICAL
    end
    Friday, August 23, 13

    View Slide

  40. event
    Friday, August 23, 13

    View Slide

  41. event
    log
    Friday, August 23, 13

    View Slide

  42. event
    alert
    log
    Friday, August 23, 13

    View Slide

  43. event
    alert
    log error
    Friday, August 23, 13

    View Slide

  44. Must self-correct
    Friday, August 23, 13

    View Slide

  45. Feedback
    PuppetDB
    Provisioning
    Configuring
    Destroying
    gPanel
    Friday, August 23, 13

    View Slide

  46. Friday, August 23, 13

    View Slide

  47. Deployable
    using text files
    Friday, August 23, 13

    View Slide

  48. Modularity
    Friday, August 23, 13

    View Slide

  49. haproxy::proxy { $es_proxy_name:
    proxy => 'listen',
    mode => 'http',
    ip => $::ipaddress_lo,
    port => '9200',
    config => {
    balance => 'roundrobin',
    },
    }
    Friday, August 23, 13

    View Slide

  50. haproxy::proxy { $es_proxy_name:
    proxy => 'listen',
    mode => 'http',
    ip => $::ipaddress_lo,
    port => '9200',
    config => {
    balance => 'roundrobin',
    },
    }
    Friday, August 23, 13

    View Slide

  51. haproxy::proxy::member { $es_proxy_name:
    hostname => $::ec2_local_ipv4,
    port => '9200',
    param => [
    'weight 1',
    'maxconn 1000',
    'check',
    ],
    }
    Friday, August 23, 13

    View Slide

  52. collectd::plugin { [
    'cpu',
    'load',
    'memory',
    'swap',
    'irq',
    'exec',
    'entropy',
    ]:
    }
    collectd::plugin { [
    'df',
    'interface',
    'protocols',
    'disk',
    ]:
    config => true
    }
    Friday, August 23, 13

    View Slide

  53. Level 4:
    Templates
    Level 5:
    Data Driven
    Friday, August 23, 13

    View Slide

  54. Self
    Authoritative
    Friday, August 23, 13

    View Slide

  55. def self.collect_resources(resource)
    query = [
    "and",
    ["=", "type", "File"],
    ["=", "exported", true],
    ["=", "tag", "nagios::object::#{resource}"],
    ["=", ["node", "active"], true]
    ]
    self.search('/v2/resources', query.to_json)
    end
    Friday, August 23, 13

    View Slide

  56. def self.collect_resources(resource)
    query = [
    "and",
    ["=", "type", "File"],
    ["=", "exported", true],
    ["=", "tag", "nagios::object::#{resource}"],
    ["=", ["node", "active"], true]
    ]
    self.search('/v2/resources', query.to_json)
    end
    Friday, August 23, 13

    View Slide

  57. # Seed the initial file with false, and enable
    # on next run with gh-nagios enable_pager
    file { '/etc/facter/facts.d/enable_pager.txt':
    ensure => file,
    owner => 'root',
    group => 'root',
    mode => '0644',
    content => 'enable_pager=false',
    replace => false,
    }
    Friday, August 23, 13

    View Slide

  58. @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg":
    content => template('nagios/etc/nagios/object.erb'),
    mode => '0444',
    group => 'nagios',
    notify => Service['nagios'],
    }
    Friday, August 23, 13

    View Slide

  59. Tips
    Friday, August 23, 13

    View Slide

  60. Refactoring
    Friday, August 23, 13

    View Slide

  61. Refactoring
    Friday, August 23, 13

    View Slide

  62. least to most
    specific
    Friday, August 23, 13

    View Slide

  63. frymanet.com
    mysql
    nginx rails
    ruby
    common admin
    package repos
    Friday, August 23, 13

    View Slide

  64. graduate to
    params
    Friday, August 23, 13

    View Slide

  65. class ntp::params {
    $defaults = {
    package => {
    version => ‘latest’,
    },
    config => {
    servers => [‘pool.ntp.org’],
    },
    }
    }
    Friday, August 23, 13

    View Slide

  66. externalize
    Friday, August 23, 13

    View Slide

  67. class ntp::params {
    $defaults = {
    package => {
    version =>
    hiera(‘ntp_package_version’),
    },
    config => {
    servers => hiera(‘ntp_servers’),
    },
    }
    }
    Friday, August 23, 13

    View Slide

  68. CloudFormation
    Friday, August 23, 13

    View Slide

  69. CloudFormation
    Friday, August 23, 13

    View Slide

  70. "Resources": {
    "RendererServerGroup": {
    "Type": "AWS::AutoScaling::AutoScalingGroup",
    "Properties": {
    "AvailabilityZones": { "Fn::GetAZs": "" },
    "LaunchConfigurationName": { "Ref": "LaunchConfig" },
    "MinSize": "2",
    "MaxSize": "16",
    "Tags": [
    { "Key": "Environment", "Value": "Production",
    "PropagateAtLaunch": "true" },
    { "Key": "Role", "Value": "renderer",
    "PropagateAtLaunch": "true" }
    ]
    }
    },
    Friday, August 23, 13

    View Slide

  71. Autoloading
    Friday, August 23, 13

    View Slide

  72. # autoloader.pp
    class nagios::autoload_helpers {
    $helpers = get_nagios_helpers()
    case $::puppetversion {
    /^3/: { include $helpers }
    default: {
    nagios::autoload_helpers::import_shim { $helpers: }
    }
    }
    }
    Friday, August 23, 13

    View Slide

  73. # autoloader.pp
    class nagios::autoload_helpers {
    $helpers = get_nagios_helpers()
    case $::puppetversion {
    /^3/: { include $helpers }
    default: {
    nagios::autoload_helpers::import_shim { $helpers: }
    }
    }
    }
    Friday, August 23, 13

    View Slide

  74. module Puppet::Parser::Functions
    newfunction(:get_nagios_helpers,
    :type => :rvalue,
    :doc => "Grab all modules that have nagios helpers for
    import") do
    module_path = File.expand_path('..',
    Puppet::Module.find('nagios',compiler.environment.to_s).path)
    helpers = Dir["#{module_path}/**/nagios/helpers.pp"].map do |d|
    "#{d.split('/')[-4]}::nagios::helpers"
    done
    helpers
    end
    end
    Friday, August 23, 13

    View Slide

  75. Modeling
    Friday, August 23, 13

    View Slide

  76. Be Dynamic
    Friday, August 23, 13

    View Slide

  77. Be Dynamic
    Friday, August 23, 13

    View Slide

  78. <%- if @comment -%>
    ### <%= @comment %>
    <%- end -%>
    <%= @type %> <% if @label %><%=
    @label %><% end %> {
    <%- @config.sort.reverse.each do |
    key,value| -%>
    <%- if value.class == Array -%>
    <%- value.each do |element| -
    %>
    <%= key %>(<%= element %>);
    <%- end -%>
    <%- else -%>
    <%= key %>(<%= value %>);
    <%- end -%>
    <%- end -%>
    };
    Friday, August 23, 13

    View Slide

  79. Fencing Resources
    Friday, August 23, 13

    View Slide

  80. # Seed the initial file with false, and enable
    # on next run with gh-nagios enable_pager
    file { '/etc/facter/facts.d/enable_pager.txt':
    ensure => file,
    owner => 'root',
    group => 'root',
    mode => '0644',
    content => 'enable_pager=false',
    replace => false,
    }
    Friday, August 23, 13

    View Slide

  81. if $::enable_pager {
    @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg":
    content => template('nagios/etc/nagios/object.erb'),
    mode => '0444',
    group => 'nagios',
    notify => Service['nagios'],
    }
    }
    Friday, August 23, 13

    View Slide

  82. Buy it?
    Friday, August 23, 13

    View Slide

  83. Buy it?
    Friday, August 23, 13

    View Slide

  84. It’s About
    Friday, August 23, 13

    View Slide

  85. Missing?
    What’s
    Friday, August 23, 13

    View Slide

  86. Language
    Friday, August 23, 13

    View Slide

  87. Predictive
    Analysis
    Friday, August 23, 13

    View Slide

  88. Coming
    Home
    Friday, August 23, 13

    View Slide

  89. Goals?
    Data?
    What
    Friday, August 23, 13

    View Slide

  90. Systems
    Thinking
    Recap
    Friday, August 23, 13

    View Slide

  91. There is System
    One
    Machine
    Parsable
    Must be a
    Known
    Quantity
    Self
    Authoritative
    Must self-correct
    Friday, August 23, 13

    View Slide

  92. Friday, August 23, 13

    View Slide

  93. jfryman
    [email protected]
    Friday, August 23, 13

    View Slide

  94. jfryman
    [email protected]
    Friday, August 23, 13

    View Slide

  95. what’s next?
    Friday, August 23, 13

    View Slide

  96. what’s next?
    Friday, August 23, 13

    View Slide