Hacking The Data out of Puppet

Slide 1

Slide 1 text

Dan Bode| Puppet Labs [email protected] Hacking The Data out of Puppet

Slide 2

Slide 2 text

# puppetconf # puppetize @ puppetlabs Who is this talk for? •  People who already understand Puppet •  Developers or people who are dev-curious

Slide 3

Slide 3 text

# puppetconf # puppetize @ puppetlabs Shameless plug

Slide 4

Slide 4 text

# puppetconf # puppetize @ puppetlabs What is it about? •  Deconstructing Puppet to data

Slide 5

Slide 5 text

Dissecting a Puppet Run Puppet as Data

Slide 6

Slide 6 text

# puppetconf # puppetize @ puppetlabs Facter, who am I? Agent Hi! your facts are: kernel=linux ipaddress=10.0.0.3 macaddress=…

Slide 7

Slide 7 text

# puppetconf # puppetize @ puppetlabs Hi Mr. Master, I need a catalog. Here are my facts http://www.dgcomputers.org/testimonials.php Agent facts

Slide 8

Slide 8 text

# puppetconf # puppetize @ puppetlabs facts Agent Thanks for you facts. I’ll just store them in PuppetDB PuppetDB

Slide 9

Slide 9 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent Mr. ENC, is this host deﬁned as an external node? Yep, he should be an apache server. Here is the deﬁnition nodes

Slide 10

Slide 10 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just compiled your catalog. One sec while I store it in PuppetDB. catalog

Slide 11

Slide 11 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog Here is your catalog. Send me a report and let me know how it went! catalog

Slide 12

Slide 12 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog I hate to be a bother, but can you compute the md5sums of a few ﬁles? catalog

Slide 13

Slide 13 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent Just ﬁnished applying. Here are the results. report catalog

Slide 14

Slide 14 text

Interacting with Puppet’s Data Termini and the indirector

Slide 15

Slide 15 text

# puppetconf # puppetize @ puppetlabs facts ﬁnd from terminus facter Agent

Slide 16

Slide 16 text

# puppetconf # puppetize @ puppetlabs catalog ﬁnd from terminus rest http://www.dgcomputers.org/testimonials.php Agent facts

Slide 17

Slide 17 text

# puppetconf # puppetize @ puppetlabs facts Agent facts save to terminus puppetdb PuppetDB

Slide 18

Slide 18 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts ENC Agent node ﬁnd from terminus exec (or ldap) nodes

Slide 19

Slide 19 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog ﬁnd from terminus compiler catalog

Slide 20

Slide 20 text

# puppetconf # puppetize @ puppetlabs PuppetDB facts Agent catalog save to terminus puppetdb catalog

Slide 21

Slide 21 text

# puppetconf # puppetize @ puppetlabs Facter ENC Disecting a Puppet Run Compiler Conﬁg Catalogs Nodes/ Manifest Reports Facts

Slide 22

Slide 22 text

# puppetconf # puppetize @ puppetlabs CLI commands puppet facts find puppet node find puppet catalog find

Slide 23

Slide 23 text

# puppetconf # puppetize @ puppetlabs CLI Puppet Facts # mkdir –p /tmp/yaml/facts # puppet facts find node_name --render-as yaml \ > /tmp/yaml/facts/node_name.yaml

Slide 24

Slide 24 text

# puppetconf # puppetize @ puppetlabs Creating a node (optional): # puppet node find node_name \ --node_terminus=exec \ --external_nodes=/etc/puppet/nodes.sh \ --facts_terminus=yaml \ --clientyamldir=/tmp/yaml/ --render-as=yaml \ > /tmp/yaml/nodes/node_name.yaml

Slide 25

Slide 25 text

# puppetconf # puppetize @ puppetlabs Creating a catalog: # puppet catalog find node_name \ --facts_terminus=yaml \ # puppet catalog find node_name \ --node_terminus=yaml \ --manifest=/etc/puppet/manifest/site.pp \ --modulepath=/etc/puppet/modules/ --clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml

Slide 26

Slide 26 text

# puppetconf # puppetize @ puppetlabs Fun with IRB Puppet::Node::Facts.indirection.ﬁnd facts Puppet::Node.new nodes Puppet::Catalog.indirection.ﬁnd catalog

Slide 27

Slide 27 text

# puppetconf # puppetize @ puppetlabs IRB Facts irb:> require ‘puppet/face’ > facts=Puppet::Face[:facts, :current].find('node’)

Slide 28

Slide 28 text

# puppetconf # puppetize @ puppetlabs Access a Fact value (irb): … > facts.values['ipaddress'] => "10.0.2.15"

Slide 29

Slide 29 text

# puppetconf # puppetize @ puppetlabs Creating a node (from irb): … > node=Puppet::Node.new('node_name', {:classes => {:foo => {:bar => :baz}}}) >node.merge(facts.values)

Slide 30

Slide 30 text

# puppetconf # puppetize @ puppetlabs Creating a catalog: … irb> catalog=Puppet::Resource::Catalog.indirection.\ find('node_name’, :node => node)

Slide 31

Slide 31 text

Interacting with Puppet’s Data Use Cases

Slide 32

Slide 32 text

# puppetconf # puppetize @ puppetlabs Inspecting the catalog: •  What types are in the catalog? irb> catalog.resources.collect do |r| r.type end.uniq •  Gimme a resource: irb>catalog.resource(‘Package[httpd]’)

Slide 33

Slide 33 text

# puppetconf # puppetize @ puppetlabs Rspec Puppet: let :facts do {:operatingsystem => ‘Redhat’} end let :params do {:bind_address => ‘0.0.0.0’ end it { should contain_file(‘/tmp/foo.conf’) }

Slide 34

Slide 34 text

# puppetconf # puppetize @ puppetlabs Thundering Herd Pre-compile catalogs for faster auto-scaling

Slide 35

Slide 35 text

# puppetconf # puppetize @ puppetlabs Applying pre-compiled catalogs: puppet apply --catalog /tmp/catalog.json –server puppet-fileserver

Slide 36

Slide 36 text

# puppetconf # puppetize @ puppetlabs DMZ tcp over USB

Slide 37

Slide 37 text

Interacting with Puppet’s Data Use Cases

Slide 38

Slide 38 text

# puppetconf # puppetize @ puppetlabs Hacking reports Everything in Puppet is a state transition User[‘dan’] : absent -> present User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash

Slide 39

Slide 39 text

# puppetconf # puppetize @ puppetlabs Setting up the agent: [agent] report=true

Slide 40

Slide 40 text

# puppetconf # puppetize @ puppetlabs Archive reports in your yamldir [master] reports = store

Slide 41

Slide 41 text

# puppetconf # puppetize @ puppetlabs Puppet reports $ cd `puppet config print reportdir` $ ls node1 node2 node3 $ ls node1

Slide 42

Slide 42 text

# puppetconf # puppetize @ puppetlabs Every report from every run ever $ ls node1 201206060256.yaml 201206060303.yaml 201206060519.yaml 201206122349.yaml 201206122354.yaml 201206130002.yaml

Slide 43

Slide 43 text

# puppetconf # puppetize @ puppetlabs Lets crack one open! Irb > require ‘yaml’ >reports=YAML.load_file('201206130002.yaml')

Slide 44

Slide 44 text

# puppetconf # puppetize @ puppetlabs Have a look >(reports.methods - Object.methods).sort Notice the following methods:

Slide 45

Slide 45 text

# puppetconf # puppetize @ puppetlabs High level data > reports.exit_status ⇒ 0 > reports.status => "unchanged" > reports.host ⇒ ”node1”

Slide 46

Slide 46 text

# puppetconf # puppetize @ puppetlabs metrics > reports.metrics.keys ⇒ ["resources", "events", "changes", "time"] > reports.metrics['resources'] ⇒ [‘failed’, 0],[ ‘changed’, ‘7’]

Slide 47

Slide 47 text

# puppetconf # puppetize @ puppetlabs And the awesome sauce > reports.resource_statuses.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

Slide 48

Slide 48 text

# puppetconf # puppetize @ puppetlabs And the awesome sauce > status = reports.resource_statuses > status.keys => ["Package[xinetd]", "File[/srv/node/1]", "Package[swift]", "Exec[compile fragments]", "Package[swift-container]", "File[/var/opt/lib/pe- puppet/concat/_etc_swift_object-server.conf]", "File[/etc/rsync.d/frag-account]”]

Slide 49

Slide 49 text

# puppetconf # puppetize @ puppetlabs And the awesome sauce >events = status["File[/etc/swift/swift.conf]"].events > events.first.status ⇒ "success” > events.first.desired_value ⇒ :present > events.first.previous_value => :absent