Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Hacking The Data out of Puppet

A8dd98306ac2cc522f9bff45872a33a7?s=47 Puppet Labs
October 25, 2012

Hacking The Data out of Puppet

"Hacking The Data out of Puppet" by Dan Bode, Integration Specialist at Puppet Labs, from PuppetConf 2012.

Dan has worked in the technology industry as a consultant and software developer for almost a decade. He has spent most of the last 4 years building infrastructure automation solutions and teaching people how to use Puppet. He currently works in Business Development at PuppetLabs where he spends his time researching technologies and figuring out how they can be integrated with Puppet to provide value.

Watch PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

A8dd98306ac2cc522f9bff45872a33a7?s=128

Puppet Labs

October 25, 2012
Tweet

Transcript

  1. Dan Bode| Puppet Labs dan@puppetlabs.com Hacking The Data out of

    Puppet
  2. # puppetconf # puppetize @ puppetlabs Who is this talk

    for? •  People who already understand Puppet •  Developers or people who are dev-curious
  3. # puppetconf # puppetize @ puppetlabs Shameless plug

  4. # puppetconf # puppetize @ puppetlabs What is it about?

    •  Deconstructing Puppet to data
  5. Dissecting a Puppet Run Puppet as Data

  6. # puppetconf # puppetize @ puppetlabs Facter, who am I?

    Agent Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…
  7. # puppetconf # puppetize @ puppetlabs Hi Mr. Master, I

    need a catalog. Here are my facts http://www.dgcomputers.org/testimonials.php Agent facts
  8. # puppetconf # puppetize @ puppetlabs facts Agent Thanks for

    you facts. I’ll just store them in PuppetDB PuppetDB
  9. # puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent

    Mr. ENC, is this host defined as an external node? Yep, he should be an apache server. Here is the definition nodes
  10. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just

    compiled your catalog. One sec while I store it in PuppetDB. catalog
  11. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog

    Here is your catalog. Send me a report and let me know how it went! catalog
  12. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog

    I hate to be a bother, but can you compute the md5sums of a few files? catalog
  13. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just

    finished applying. Here are the results. report catalog
  14. Interacting with Puppet’s Data Termini and the indirector

  15. # puppetconf # puppetize @ puppetlabs facts find from terminus

    facter Agent
  16. # puppetconf # puppetize @ puppetlabs catalog find from terminus

    rest http://www.dgcomputers.org/testimonials.php Agent facts
  17. # puppetconf # puppetize @ puppetlabs facts Agent facts save

    to terminus puppetdb PuppetDB
  18. # puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent

    node find from terminus exec (or ldap) nodes
  19. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog

    find from terminus compiler catalog
  20. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog

    save to terminus puppetdb catalog
  21. # puppetconf # puppetize @ puppetlabs Facter ENC Disecting a

    Puppet Run Compiler Config Catalogs Nodes/ Manifest Reports Facts
  22. # puppetconf # puppetize @ puppetlabs CLI commands puppet facts

    find puppet node find puppet catalog find
  23. # puppetconf # puppetize @ puppetlabs CLI Puppet Facts #

    mkdir –p /tmp/yaml/facts # puppet facts find node_name --render-as yaml \ > /tmp/yaml/facts/node_name.yaml
  24. # puppetconf # puppetize @ puppetlabs Creating a node (optional):

    # puppet node find node_name \ --node_terminus=exec \ --external_nodes=/etc/puppet/nodes.sh \ --facts_terminus=yaml \ --clientyamldir=/tmp/yaml/ --render-as=yaml \ > /tmp/yaml/nodes/node_name.yaml
  25. # puppetconf # puppetize @ puppetlabs Creating a catalog: #

    puppet catalog find node_name \ --facts_terminus=yaml \ # puppet catalog find node_name \ --node_terminus=yaml \ --manifest=/etc/puppet/manifest/site.pp \ --modulepath=/etc/puppet/modules/ --clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml
  26. # puppetconf # puppetize @ puppetlabs Fun with IRB Puppet::Node::Facts.indirection.find

    facts Puppet::Node.new nodes Puppet::Catalog.indirection.find catalog
  27. # puppetconf # puppetize @ puppetlabs IRB Facts irb:> require

    ‘puppet/face’ > facts=Puppet::Face[:facts, :current].find('node’)
  28. # puppetconf # puppetize @ puppetlabs Access a Fact value

    (irb): … > facts.values['ipaddress'] => "10.0.2.15"
  29. # puppetconf # puppetize @ puppetlabs Creating a node (from

    irb): … > node=Puppet::Node.new('node_name', {:classes => {:foo => {:bar => :baz}}}) >node.merge(facts.values)
  30. # puppetconf # puppetize @ puppetlabs Creating a catalog: …

    irb> catalog=Puppet::Resource::Catalog.indirection.\ find('node_name’, :node => node)
  31. Interacting with Puppet’s Data Use Cases

  32. # puppetconf # puppetize @ puppetlabs Inspecting the catalog: • 

    What types are in the catalog? irb> catalog.resources.collect do |r| r.type end.uniq •  Gimme a resource: irb>catalog.resource(‘Package[httpd]’)
  33. # puppetconf # puppetize @ puppetlabs Rspec Puppet: let :facts

    do {:operatingsystem => ‘Redhat’} end let :params do {:bind_address => ‘0.0.0.0’ end it { should contain_file(‘/tmp/foo.conf’) }
  34. # puppetconf # puppetize @ puppetlabs Thundering Herd Pre-compile catalogs

    for faster auto-scaling
  35. # puppetconf # puppetize @ puppetlabs Applying pre-compiled catalogs: puppet

    apply --catalog /tmp/catalog.json –server puppet-fileserver
  36. # puppetconf # puppetize @ puppetlabs DMZ tcp over USB

  37. Interacting with Puppet’s Data Use Cases

  38. # puppetconf # puppetize @ puppetlabs Hacking reports Everything in

    Puppet is a state transition User[‘dan’] : absent -> present User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
  39. # puppetconf # puppetize @ puppetlabs Setting up the agent:

    [agent] report=true
  40. # puppetconf # puppetize @ puppetlabs Archive reports in your

    yamldir [master] reports = store
  41. # puppetconf # puppetize @ puppetlabs Puppet reports $ cd

    `puppet config print reportdir` $ ls node1 node2 node3 $ ls node1
  42. # puppetconf # puppetize @ puppetlabs Every report from every

    run ever $ ls node1 201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml
  43. # puppetconf # puppetize @ puppetlabs Lets crack one open!

    Irb > require ‘yaml’ >reports=YAML.load_file('201206130002.yaml')
  44. # puppetconf # puppetize @ puppetlabs Have a look >(reports.methods

    - Object.methods).sort Notice the following methods:
  45. # puppetconf # puppetize @ puppetlabs High level data >

    reports.exit_status ⇒ 0 > reports.status => "unchanged" > reports.host ⇒ ”node1”
  46. # puppetconf # puppetize @ puppetlabs metrics > reports.metrics.keys ⇒ ["resources",

    "events", "changes", "time"] > reports.metrics['resources'] ⇒ [‘failed’, 0],[ ‘changed’, ‘7’]
  47. # puppetconf # puppetize @ puppetlabs And the awesome sauce

    > reports.resource_statuses.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
  48. # puppetconf # puppetize @ puppetlabs And the awesome sauce

    > status = reports.resource_statuses > status.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
  49. # puppetconf # puppetize @ puppetlabs And the awesome sauce

    >events = status["File[/etc/swift/swift.conf]"].events > events.first.status ⇒ "success” > events.first.desired_value ⇒ :present > events.first.previous_value => :absent
  50. Thank You Dan Bode| Puppet Labs dan@puppetlabs.com

  51. None