Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hacking The Data out of Puppet

Puppet Labs
October 25, 2012

Hacking The Data out of Puppet

"Hacking The Data out of Puppet" by Dan Bode, Integration Specialist at Puppet Labs, from PuppetConf 2012.

Dan has worked in the technology industry as a consultant and software developer for almost a decade. He has spent most of the last 4 years building infrastructure automation solutions and teaching people how to use Puppet. He currently works in Business Development at PuppetLabs where he spends his time researching technologies and figuring out how they can be integrated with Puppet to provide value.

Watch PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

Puppet Labs

October 25, 2012
Tweet

More Decks by Puppet Labs

Other Decks in Technology

Transcript

  1. Dan Bode| Puppet Labs
    [email protected]
    Hacking
    The Data
    out of Puppet

    View Slide

  2. # puppetconf # puppetize @ puppetlabs
    Who is this talk for?
    •  People who already understand Puppet
    •  Developers or people who are dev-curious

    View Slide

  3. # puppetconf # puppetize @ puppetlabs
    Shameless plug

    View Slide

  4. # puppetconf # puppetize @ puppetlabs
    What is it about?
    •  Deconstructing Puppet to data

    View Slide

  5. Dissecting a Puppet Run
    Puppet as Data

    View Slide

  6. # puppetconf # puppetize @ puppetlabs
    Facter, who am I?

    Agent

    Hi! your facts are:



    kernel=linux

    ipaddress=10.0.0.3

    macaddress=…

    View Slide

  7. # puppetconf # puppetize @ puppetlabs
    Hi Mr. Master,

    I need a catalog. Here
    are my facts

    http://www.dgcomputers.org/testimonials.php

    Agent

    facts

    View Slide

  8. # puppetconf # puppetize @ puppetlabs
    facts

    Agent

    Thanks for you facts.

    I’ll just store them in
    PuppetDB

    PuppetDB

    View Slide

  9. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    ENC

    Agent

    Mr. ENC, is this host
    defined as an external
    node?

    Yep, he should be an
    apache server. Here is
    the definition

    nodes

    View Slide

  10. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    Just compiled your
    catalog. One sec while
    I store it in PuppetDB.

    catalog

    View Slide

  11. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    catalog

    Here is your
    catalog. Send me
    a report and let
    me know how it
    went!

    catalog

    View Slide

  12. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    catalog

    I hate to be a
    bother, but can
    you compute
    the md5sums of
    a few files?

    catalog

    View Slide

  13. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    Just finished applying.
    Here are the results.

    report

    catalog

    View Slide

  14. Interacting with Puppet’s Data
    Termini and the indirector

    View Slide

  15. # puppetconf # puppetize @ puppetlabs
    facts find from
    terminus facter

    Agent

    View Slide

  16. # puppetconf # puppetize @ puppetlabs
    catalog find from
    terminus rest

    http://www.dgcomputers.org/testimonials.php

    Agent

    facts

    View Slide

  17. # puppetconf # puppetize @ puppetlabs
    facts

    Agent

    facts save to terminus
    puppetdb

    PuppetDB

    View Slide

  18. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    ENC

    Agent

    node find from
    terminus exec (or
    ldap)

    nodes

    View Slide

  19. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    catalog find from
    terminus compiler

    catalog

    View Slide

  20. # puppetconf # puppetize @ puppetlabs
    PuppetDB

    facts

    Agent

    catalog save to
    terminus puppetdb

    catalog

    View Slide

  21. # puppetconf # puppetize @ puppetlabs
    Facter
    ENC

    Disecting a Puppet Run
    Compiler

    Config
    Catalogs

    Nodes/
    Manifest

    Reports

    Facts

    View Slide

  22. # puppetconf # puppetize @ puppetlabs
    CLI commands
    puppet
    facts
    find

    puppet
    node
    find

    puppet
    catalog
    find

    View Slide

  23. # puppetconf # puppetize @ puppetlabs
    CLI Puppet Facts
    # mkdir –p /tmp/yaml/facts
    # puppet facts find node_name --render-as yaml \
    > /tmp/yaml/facts/node_name.yaml

    View Slide

  24. # puppetconf # puppetize @ puppetlabs
    Creating a node (optional):
    # puppet node find node_name \
    --node_terminus=exec \
    --external_nodes=/etc/puppet/nodes.sh \
    --facts_terminus=yaml \
    --clientyamldir=/tmp/yaml/ --render-as=yaml \
    > /tmp/yaml/nodes/node_name.yaml

    View Slide

  25. # puppetconf # puppetize @ puppetlabs
    Creating a catalog:
    # puppet catalog find node_name \
    --facts_terminus=yaml \
    # puppet catalog find node_name \
    --node_terminus=yaml \
    --manifest=/etc/puppet/manifest/site.pp \
    --modulepath=/etc/puppet/modules/
    --clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml

    View Slide

  26. # puppetconf # puppetize @ puppetlabs
    Fun with IRB
    Puppet::Node::Facts.indirection.find
    facts

    Puppet::Node.new
    nodes

    Puppet::Catalog.indirection.find
    catalog

    View Slide

  27. # puppetconf # puppetize @ puppetlabs
    IRB Facts
    irb:> require ‘puppet/face’
    > facts=Puppet::Face[:facts, :current].find('node’)

    View Slide

  28. # puppetconf # puppetize @ puppetlabs
    Access a Fact value (irb):

    > facts.values['ipaddress']
    => "10.0.2.15"

    View Slide

  29. # puppetconf # puppetize @ puppetlabs
    Creating a node (from irb):

    > node=Puppet::Node.new('node_name',
    {:classes => {:foo => {:bar => :baz}}})
    >node.merge(facts.values)

    View Slide

  30. # puppetconf # puppetize @ puppetlabs
    Creating a catalog:

    irb> catalog=Puppet::Resource::Catalog.indirection.\
    find('node_name’, :node => node)

    View Slide

  31. Interacting with Puppet’s Data
    Use Cases

    View Slide

  32. # puppetconf # puppetize @ puppetlabs
    Inspecting the catalog:
    •  What types are in the catalog?
    irb> catalog.resources.collect do |r| r.type end.uniq
    •  Gimme a resource:
    irb>catalog.resource(‘Package[httpd]’)

    View Slide

  33. # puppetconf # puppetize @ puppetlabs
    Rspec Puppet:
    let :facts do
    {:operatingsystem => ‘Redhat’}
    end
    let :params do
    {:bind_address => ‘0.0.0.0’
    end
    it { should contain_file(‘/tmp/foo.conf’) }

    View Slide

  34. # puppetconf # puppetize @ puppetlabs
    Thundering Herd
    Pre-compile catalogs for faster auto-scaling

    View Slide

  35. # puppetconf # puppetize @ puppetlabs
    Applying pre-compiled
    catalogs:
    puppet apply --catalog /tmp/catalog.json –server
    puppet-fileserver

    View Slide

  36. # puppetconf # puppetize @ puppetlabs
    DMZ
    tcp over USB

    View Slide

  37. Interacting with Puppet’s Data
    Use Cases

    View Slide

  38. # puppetconf # puppetize @ puppetlabs
    Hacking reports
    Everything in Puppet is a state transition
    User[‘dan’] : absent -> present
    User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash

    View Slide

  39. # puppetconf # puppetize @ puppetlabs
    Setting up the agent:
    [agent]
    report=true

    View Slide

  40. # puppetconf # puppetize @ puppetlabs
    Archive reports in your
    yamldir
    [master]
    reports = store

    View Slide

  41. # puppetconf # puppetize @ puppetlabs
    Puppet reports
    $ cd `puppet config print reportdir`
    $ ls
    node1 node2 node3
    $ ls node1

    View Slide

  42. # puppetconf # puppetize @ puppetlabs
    Every report from every run
    ever
    $ ls node1
    201206060256.yaml 201206060303.yaml
    201206060519.yaml 201206122349.yaml
    201206122354.yaml 201206130002.yaml

    View Slide

  43. # puppetconf # puppetize @ puppetlabs
    Lets crack one open!
    Irb > require ‘yaml’
    >reports=YAML.load_file('201206130002.yaml')

    View Slide

  44. # puppetconf # puppetize @ puppetlabs
    Have a look
    >(reports.methods - Object.methods).sort
    Notice the following methods:

    View Slide

  45. # puppetconf # puppetize @ puppetlabs
    High level data
    > reports.exit_status
    ⇒ 0
    > reports.status
    => "unchanged"
    > reports.host
    ⇒ ”node1”

    View Slide

  46. # puppetconf # puppetize @ puppetlabs
    metrics
    > reports.metrics.keys
    ⇒ ["resources", "events", "changes", "time"]
    > reports.metrics['resources']
    ⇒ [‘failed’, 0],[ ‘changed’, ‘7’]

    View Slide

  47. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    > reports.resource_statuses.keys
    => ["Package[xinetd]", "File[/srv/node/1]",
    "Package[swift]", "Exec[compile fragments]",
    "Package[swift-container]", "File[/var/opt/lib/pe-
    puppet/concat/_etc_swift_object-server.conf]",
    "File[/etc/rsync.d/frag-account]”]

    View Slide

  48. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    > status = reports.resource_statuses
    > status.keys
    => ["Package[xinetd]", "File[/srv/node/1]",
    "Package[swift]", "Exec[compile fragments]",
    "Package[swift-container]", "File[/var/opt/lib/pe-
    puppet/concat/_etc_swift_object-server.conf]",
    "File[/etc/rsync.d/frag-account]”]

    View Slide

  49. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    >events = status["File[/etc/swift/swift.conf]"].events
    > events.first.status
    ⇒ "success”
    > events.first.desired_value
    ⇒ :present
    > events.first.previous_value
    => :absent

    View Slide

  50. Thank You
    Dan Bode| Puppet Labs
    [email protected]

    View Slide

  51. View Slide