Building Data-Driven Infrastructure with Puppet

The road to Data Driven Infrastructure is a hard and long one, but a worthwhile endeavor in the lifecycle of Information Technology and Business Enablement. This talk is a snapshot of the voyage of one man's quest to have computers absolutely and completely take over his job, the patterns that have been uncovered, and the things we as an industry need to keep pushing on.

This talk can be viewed online at https://www.youtube.com/watch?v=vEpXJRPITN0&list=PLV86BgbREluU02Ytlz80seDSKAbkx5pRg

Transcript

1. Friday, August 23, 13
   

   View Slide

2. Data
   Driven
   Infrastructure
   Friday, August 23, 13
   

   View Slide

3. Friday, August 23, 13
   

   View Slide

4. ?
   Friday, August 23, 13
   

   View Slide

5. ?
   http://git.io/UmUdKA
   Friday, August 23, 13
   

   View Slide

6. Let me
   Tell you
   a story
   Friday, August 23, 13
   

   View Slide

7. What is a
   Cloud
   Friday, August 23, 13
   

   View Slide

8. I don’t have to think about IT
   With the Cloud...
   Friday, August 23, 13
   

   View Slide

9. Friday, August 23, 13
   

   View Slide

10. aaS
    Platform
    Software
    Infrastructure
    Friday, August 23, 13
    

    View Slide

11. Friday, August 23, 13
    

    View Slide

12. Friday, August 23, 13
    

    View Slide

13. Where is
    our cloud?
    Friday, August 23, 13
    

    View Slide

14. Friday, August 23, 13
    

    View Slide

15. Utopia
    Friday, August 23, 13
    

    View Slide

16. James Fryman
    Friday, August 23, 13
    

    View Slide

17. Friday, August 23, 13
    

    View Slide

18. Automations
    Junkie
    Friday, August 23, 13
    

    View Slide

19. Are you
    Crazy?
    Friday, August 23, 13
    

    View Slide

20. Goals?
    Data?
    What
    Friday, August 23, 13
    

    View Slide

21. Goals?
    Data?
    What
    Friday, August 23, 13
    

    View Slide

22. Goals?
    Data?
    What
    Friday, August 23, 13
    

    View Slide

23. Machine
    Parsable
    Friday, August 23, 13
    

    View Slide

24. There is System
    One
    Friday, August 23, 13
    

    View Slide

25. Feedback
    PuppetDB
    Provisioning
    Configuring
    Destroying
    gPanel
    Friday, August 23, 13
    

    View Slide

26. Parts
    of the
    System
    Friday, August 23, 13
    

    View Slide

27. provisioner
    Friday, August 23, 13
    

    View Slide

28. Home
    Grown
    Friday, August 23, 13
    

    View Slide

29. Home Grown
    Friday, August 23, 13
    

    View Slide

30. controller
    Friday, August 23, 13
    

    View Slide

31. Lots O’ Models
    Friday, August 23, 13
    

    View Slide

32. file { '/etc/facter/facts.d/quagga_manage_service':
    ensure => present,
    replace => false,
    }
    if $::quagga_manage_service {
    $service_state[ensure] = running
    $service_state[enable] = true
    } else {
    $service_state[ensure] = undef
    $service_state[enable] = undef
    }
    service { 'quagga':
    ensure => $service_state[ensure],
    enable => $service_state[enable],
    }
    Friday, August 23, 13
    

    View Slide

33. orchestrator
    Friday, August 23, 13
    

    View Slide

34. Chat
    Ops
    Friday, August 23, 13
    

    View Slide

35. Chat
    Ops
    Friday, August 23, 13
    

    View Slide

36. Feedback
    of the
    System
    Friday, August 23, 13
    

    View Slide

37. Metrics&
    Monitoring
    Friday, August 23, 13
    

    View Slide

38. begin
    JSON.parse(RestClient.get(url)).each do |cache|
    data["#{cache['target']}"] = 0
    count = 0
    cache["datapoints"].each do |point|
    unless (point[0].nil?)
    data["#{cache['target']}"] += point[0]
    count += 1
    end
    end
    data["#{cache['target']}"] /= count
    data["total"] += data["#{cache['target']}"]
    end
    rescue ZeroDivisionError => e
    puts "UNKNOWN metric not in graphite!"
    exit EXIT_UNKNOWN
    rescue => e
    puts e.message
    exit EXIT_CRITICAL
    end
    Friday, August 23, 13
    

    View Slide

39. begin
    JSON.parse(RestClient.get(url)).each do |cache|
    data["#{cache['target']}"] = 0
    count = 0
    cache["datapoints"].each do |point|
    unless (point[0].nil?)
    data["#{cache['target']}"] += point[0]
    count += 1
    end
    end
    data["#{cache['target']}"] /= count
    data["total"] += data["#{cache['target']}"]
    end
    rescue ZeroDivisionError => e
    puts "UNKNOWN metric not in graphite!"
    exit EXIT_UNKNOWN
    rescue => e
    puts e.message
    exit EXIT_CRITICAL
    end
    Friday, August 23, 13
    

    View Slide

40. event
    Friday, August 23, 13
    

    View Slide

41. event
    log
    Friday, August 23, 13
    

    View Slide

42. event
    alert
    log
    Friday, August 23, 13
    

    View Slide

43. event
    alert
    log error
    Friday, August 23, 13
    

    View Slide

44. Must self-correct
    Friday, August 23, 13
    

    View Slide

45. Feedback
    PuppetDB
    Provisioning
    Configuring
    Destroying
    gPanel
    Friday, August 23, 13
    

    View Slide

46. Friday, August 23, 13
    

    View Slide

47. Deployable
    using text files
    Friday, August 23, 13
    

    View Slide

48. Modularity
    Friday, August 23, 13
    

    View Slide

49. haproxy::proxy { $es_proxy_name:
    proxy => 'listen',
    mode => 'http',
    ip => $::ipaddress_lo,
    port => '9200',
    config => {
    balance => 'roundrobin',
    },
    }
    Friday, August 23, 13
    

    View Slide

50. haproxy::proxy { $es_proxy_name:
    proxy => 'listen',
    mode => 'http',
    ip => $::ipaddress_lo,
    port => '9200',
    config => {
    balance => 'roundrobin',
    },
    }
    Friday, August 23, 13
    

    View Slide

51. haproxy::proxy::member { $es_proxy_name:
    hostname => $::ec2_local_ipv4,
    port => '9200',
    param => [
    'weight 1',
    'maxconn 1000',
    'check',
    ],
    }
    Friday, August 23, 13
    

    View Slide

52. collectd::plugin { [
    'cpu',
    'load',
    'memory',
    'swap',
    'irq',
    'exec',
    'entropy',
    ]:
    }
    collectd::plugin { [
    'df',
    'interface',
    'protocols',
    'disk',
    ]:
    config => true
    }
    Friday, August 23, 13
    

    View Slide

53. Level 4:
    Templates
    Level 5:
    Data Driven
    Friday, August 23, 13
    

    View Slide

54. Self
    Authoritative
    Friday, August 23, 13
    

    View Slide

55. def self.collect_resources(resource)
    query = [
    "and",
    ["=", "type", "File"],
    ["=", "exported", true],
    ["=", "tag", "nagios::object::#{resource}"],
    ["=", ["node", "active"], true]
    ]
    self.search('/v2/resources', query.to_json)
    end
    Friday, August 23, 13
    

    View Slide

56. def self.collect_resources(resource)
    query = [
    "and",
    ["=", "type", "File"],
    ["=", "exported", true],
    ["=", "tag", "nagios::object::#{resource}"],
    ["=", ["node", "active"], true]
    ]
    self.search('/v2/resources', query.to_json)
    end
    Friday, August 23, 13
    

    View Slide

57. # Seed the initial file with false, and enable
    # on next run with gh-nagios enable_pager
    file { '/etc/facter/facts.d/enable_pager.txt':
    ensure => file,
    owner => 'root',
    group => 'root',
    mode => '0644',
    content => 'enable_pager=false',
    replace => false,
    }
    Friday, August 23, 13
    

    View Slide

58. @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg":
    content => template('nagios/etc/nagios/object.erb'),
    mode => '0444',
    group => 'nagios',
    notify => Service['nagios'],
    }
    Friday, August 23, 13
    

    View Slide

59. Tips
    Friday, August 23, 13
    

    View Slide

60. Refactoring
    Friday, August 23, 13
    

    View Slide

61. Refactoring
    Friday, August 23, 13
    

    View Slide

62. least to most
    specific
    Friday, August 23, 13
    

    View Slide

63. frymanet.com
    mysql
    nginx rails
    ruby
    common admin
    package repos
    Friday, August 23, 13
    

    View Slide

64. graduate to
    params
    Friday, August 23, 13
    

    View Slide

65. class ntp::params {
    $defaults = {
    package => {
    version => ‘latest’,
    },
    config => {
    servers => [‘pool.ntp.org’],
    },
    }
    }
    Friday, August 23, 13
    

    View Slide

66. externalize
    Friday, August 23, 13
    

    View Slide

67. class ntp::params {
    $defaults = {
    package => {
    version =>
    hiera(‘ntp_package_version’),
    },
    config => {
    servers => hiera(‘ntp_servers’),
    },
    }
    }
    Friday, August 23, 13
    

    View Slide

68. CloudFormation
    Friday, August 23, 13
    

    View Slide

69. CloudFormation
    Friday, August 23, 13
    

    View Slide

70. "Resources": {
    "RendererServerGroup": {
    "Type": "AWS::AutoScaling::AutoScalingGroup",
    "Properties": {
    "AvailabilityZones": { "Fn::GetAZs": "" },
    "LaunchConfigurationName": { "Ref": "LaunchConfig" },
    "MinSize": "2",
    "MaxSize": "16",
    "Tags": [
    { "Key": "Environment", "Value": "Production",
    "PropagateAtLaunch": "true" },
    { "Key": "Role", "Value": "renderer",
    "PropagateAtLaunch": "true" }
    ]
    }
    },
    Friday, August 23, 13
    

    View Slide

71. Autoloading
    Friday, August 23, 13
    

    View Slide

72. # autoloader.pp
    class nagios::autoload_helpers {
    $helpers = get_nagios_helpers()
    case $::puppetversion {
    /^3/: { include $helpers }
    default: {
    nagios::autoload_helpers::import_shim { $helpers: }
    }
    }
    }
    Friday, August 23, 13
    

    View Slide

73. # autoloader.pp
    class nagios::autoload_helpers {
    $helpers = get_nagios_helpers()
    case $::puppetversion {
    /^3/: { include $helpers }
    default: {
    nagios::autoload_helpers::import_shim { $helpers: }
    }
    }
    }
    Friday, August 23, 13
    

    View Slide

74. module Puppet::Parser::Functions
    newfunction(:get_nagios_helpers,
    :type => :rvalue,
    :doc => "Grab all modules that have nagios helpers for
    import") do
    module_path = File.expand_path('..',
    Puppet::Module.find('nagios',compiler.environment.to_s).path)
    helpers = Dir["#{module_path}/**/nagios/helpers.pp"].map do |d|
    "#{d.split('/')[-4]}::nagios::helpers"
    done
    helpers
    end
    end
    Friday, August 23, 13
    

    View Slide

75. Modeling
    Friday, August 23, 13
    

    View Slide

76. Be Dynamic
    Friday, August 23, 13
    

    View Slide

77. Be Dynamic
    Friday, August 23, 13
    

    View Slide

78. <%- if @comment -%>
    ### <%= @comment %>
    <%- end -%>
    <%= @type %> <% if @label %><%=
    @label %><% end %> {
    <%- @config.sort.reverse.each do |
    key,value| -%>
    <%- if value.class == Array -%>
    <%- value.each do |element| -
    %>
    <%= key %>(<%= element %>);
    <%- end -%>
    <%- else -%>
    <%= key %>(<%= value %>);
    <%- end -%>
    <%- end -%>
    };
    Friday, August 23, 13
    

    View Slide

79. Fencing Resources
    Friday, August 23, 13
    

    View Slide

80. # Seed the initial file with false, and enable
    # on next run with gh-nagios enable_pager
    file { '/etc/facter/facts.d/enable_pager.txt':
    ensure => file,
    owner => 'root',
    group => 'root',
    mode => '0644',
    content => 'enable_pager=false',
    replace => false,
    }
    Friday, August 23, 13
    

    View Slide

81. if $::enable_pager {
    @@file { "/etc/nagios/objects/${type}.d/${filename}.cfg":
    content => template('nagios/etc/nagios/object.erb'),
    mode => '0444',
    group => 'nagios',
    notify => Service['nagios'],
    }
    }
    Friday, August 23, 13
    

    View Slide

82. Buy it?
    Friday, August 23, 13
    

    View Slide

83. Buy it?
    Friday, August 23, 13
    

    View Slide

84. It’s About
    Friday, August 23, 13
    

    View Slide

85. Missing?
    What’s
    Friday, August 23, 13
    

    View Slide

86. Language
    Friday, August 23, 13
    

    View Slide

87. Predictive
    Analysis
    Friday, August 23, 13
    

    View Slide

88. Coming
    Home
    Friday, August 23, 13
    

    View Slide

89. Goals?
    Data?
    What
    Friday, August 23, 13
    

    View Slide

90. Systems
    Thinking
    Recap
    Friday, August 23, 13
    

    View Slide

91. There is System
    One
    Machine
    Parsable
    Must be a
    Known
    Quantity
    Self
    Authoritative
    Must self-correct
    Friday, August 23, 13
    

    View Slide

92. Friday, August 23, 13
    

    View Slide

93. jfryman
    fryman@github.com
    Friday, August 23, 13
    

    View Slide

94. jfryman
    fryman@github.com
    Friday, August 23, 13
    

    View Slide

95. what’s next?
    Friday, August 23, 13
    

    View Slide

96. what’s next?
    Friday, August 23, 13
    

    View Slide