Hacking The Data out of Puppet

"Hacking The Data out of Puppet" by Dan Bode, Integration Specialist at Puppet Labs, from PuppetConf 2012.

Dan has worked in the technology industry as a consultant and software developer for almost a decade. He has spent most of the last 4 years building infrastructure automation solutions and teaching people how to use Puppet. He currently works in Business Development at PuppetLabs where he spends his time researching technologies and figuring out how they can be integrated with Puppet to provide value.

Watch PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

Transcript

1. Dan Bode| Puppet Labs dan@puppetlabs.com Hacking The Data out of Puppet
2. # puppetconf # puppetize @ puppetlabs Who is this talk for? •  People who already understand Puppet •  Developers or people who are dev-curious
3. # puppetconf # puppetize @ puppetlabs Shameless plug
4. # puppetconf # puppetize @ puppetlabs What is it about? •  Deconstructing Puppet to data
5. Dissecting a Puppet Run Puppet as Data
6. # puppetconf # puppetize @ puppetlabs Facter, who am I? Agent Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…
7. # puppetconf # puppetize @ puppetlabs Hi Mr. Master, I need a catalog. Here are my facts http://www.dgcomputers.org/testimonials.php Agent facts
8. # puppetconf # puppetize @ puppetlabs facts Agent Thanks for you facts. I’ll just store them in PuppetDB PuppetDB
9. # puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent Mr. ENC, is this host defined as an external node? Yep, he should be an apache server. Here is the definition nodes
10. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just compiled your catalog. One sec while I store it in PuppetDB. catalog
11. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog Here is your catalog. Send me a report and let me know how it went! catalog
12. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog I hate to be a bother, but can you compute the md5sums of a few files? catalog
13. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just finished applying. Here are the results. report catalog
14. Interacting with Puppet’s Data Termini and the indirector
15. # puppetconf # puppetize @ puppetlabs facts find from terminus facter Agent
16. # puppetconf # puppetize @ puppetlabs catalog find from terminus rest http://www.dgcomputers.org/testimonials.php Agent facts
17. # puppetconf # puppetize @ puppetlabs facts Agent facts save to terminus puppetdb PuppetDB
18. # puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent node find from terminus exec (or ldap) nodes
19. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog find from terminus compiler catalog
20. # puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog save to terminus puppetdb catalog
21. # puppetconf # puppetize @ puppetlabs Facter ENC Disecting a Puppet Run Compiler Config Catalogs Nodes/ Manifest Reports Facts
22. # puppetconf # puppetize @ puppetlabs CLI commands puppet facts find puppet node find puppet catalog find
23. # puppetconf # puppetize @ puppetlabs CLI Puppet Facts # mkdir –p /tmp/yaml/facts # puppet facts find node_name --render-as yaml \ > /tmp/yaml/facts/node_name.yaml
24. # puppetconf # puppetize @ puppetlabs Creating a node (optional): # puppet node find node_name \ --node_terminus=exec \ --external_nodes=/etc/puppet/nodes.sh \ --facts_terminus=yaml \ --clientyamldir=/tmp/yaml/ --render-as=yaml \ > /tmp/yaml/nodes/node_name.yaml
25. # puppetconf # puppetize @ puppetlabs Creating a catalog: # puppet catalog find node_name \ --facts_terminus=yaml \ # puppet catalog find node_name \ --node_terminus=yaml \ --manifest=/etc/puppet/manifest/site.pp \ --modulepath=/etc/puppet/modules/ --clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml
26. # puppetconf # puppetize @ puppetlabs Fun with IRB Puppet::Node::Facts.indirection.find facts Puppet::Node.new nodes Puppet::Catalog.indirection.find catalog
27. # puppetconf # puppetize @ puppetlabs IRB Facts irb:> require ‘puppet/face’ > facts=Puppet::Face[:facts, :current].find('node’)
28. # puppetconf # puppetize @ puppetlabs Access a Fact value (irb): … > facts.values['ipaddress'] => "10.0.2.15"
29. # puppetconf # puppetize @ puppetlabs Creating a node (from irb): … > node=Puppet::Node.new('node_name', {:classes => {:foo => {:bar => :baz}}}) >node.merge(facts.values)
30. # puppetconf # puppetize @ puppetlabs Creating a catalog: … irb> catalog=Puppet::Resource::Catalog.indirection.\ find('node_name’, :node => node)
31. Interacting with Puppet’s Data Use Cases
32. # puppetconf # puppetize @ puppetlabs Inspecting the catalog: •  What types are in the catalog? irb> catalog.resources.collect do |r| r.type end.uniq •  Gimme a resource: irb>catalog.resource(‘Package[httpd]’)
33. # puppetconf # puppetize @ puppetlabs Rspec Puppet: let :facts do {:operatingsystem => ‘Redhat’} end let :params do {:bind_address => ‘0.0.0.0’ end it { should contain_file(‘/tmp/foo.conf’) }
34. # puppetconf # puppetize @ puppetlabs Thundering Herd Pre-compile catalogs for faster auto-scaling
35. # puppetconf # puppetize @ puppetlabs Applying pre-compiled catalogs: puppet apply --catalog /tmp/catalog.json –server puppet-fileserver
36. # puppetconf # puppetize @ puppetlabs DMZ tcp over USB
37. Interacting with Puppet’s Data Use Cases
38. # puppetconf # puppetize @ puppetlabs Hacking reports Everything in Puppet is a state transition User[‘dan’] : absent -> present User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
39. # puppetconf # puppetize @ puppetlabs Setting up the agent: [agent] report=true
40. # puppetconf # puppetize @ puppetlabs Archive reports in your yamldir [master] reports = store
41. # puppetconf # puppetize @ puppetlabs Puppet reports $ cd `puppet config print reportdir` $ ls node1 node2 node3 $ ls node1
42. # puppetconf # puppetize @ puppetlabs Every report from every run ever $ ls node1 201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml
43. # puppetconf # puppetize @ puppetlabs Lets crack one open! Irb > require ‘yaml’ >reports=YAML.load_file('201206130002.yaml')
44. # puppetconf # puppetize @ puppetlabs Have a look >(reports.methods - Object.methods).sort Notice the following methods:
45. # puppetconf # puppetize @ puppetlabs High level data > reports.exit_status ⇒ 0 > reports.status => "unchanged" > reports.host ⇒ ”node1”
46. # puppetconf # puppetize @ puppetlabs metrics > reports.metrics.keys ⇒ ["resources", "events", "changes", "time"] > reports.metrics['resources'] ⇒ [‘failed’, 0],[ ‘changed’, ‘7’]
47. # puppetconf # puppetize @ puppetlabs And the awesome sauce > reports.resource_statuses.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
48. # puppetconf # puppetize @ puppetlabs And the awesome sauce > status = reports.resource_statuses > status.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]
49. # puppetconf # puppetize @ puppetlabs And the awesome sauce >events = status["File[/etc/swift/swift.conf]"].events > events.first.status ⇒ "success” > events.first.desired_value ⇒ :present > events.first.previous_value => :absent
50. Thank You Dan Bode| Puppet Labs dan@puppetlabs.com