Hacking The Data out of Puppet

"Hacking The Data out of Puppet" by Dan Bode, Integration Specialist at Puppet Labs, from PuppetConf 2012.

Dan has worked in the technology industry as a consultant and software developer for almost a decade. He has spent most of the last 4 years building infrastructure automation solutions and teaching people how to use Puppet. He currently works in Business Development at PuppetLabs where he spends his time researching technologies and figuring out how they can be integrated with Puppet to provide value.

Watch PuppetConf videos you missed out on here: http://www.puppetlabs.com/community/videos/puppetconf

Be the first to know about PuppetConf 2013:
http://info.puppetlabs.com/puppetconf2013-notification.html

Transcript

1. Dan Bode| Puppet Labs
   dan@puppetlabs.com
   Hacking
   The Data
   out of Puppet
   

   View Slide

2. # puppetconf # puppetize @ puppetlabs
   Who is this talk for?
   •  People who already understand Puppet
   •  Developers or people who are dev-curious
   

   View Slide

3. # puppetconf # puppetize @ puppetlabs
   Shameless plug
   

   View Slide

4. # puppetconf # puppetize @ puppetlabs
   What is it about?
   •  Deconstructing Puppet to data
   

   View Slide

5. Dissecting a Puppet Run
   Puppet as Data
   

   View Slide

6. # puppetconf # puppetize @ puppetlabs
   Facter, who am I?
   
   Agent
   
   Hi! your facts are:
   
   
   
   kernel=linux
   
   ipaddress=10.0.0.3
   
   macaddress=…
   
   

   View Slide

7. # puppetconf # puppetize @ puppetlabs
   Hi Mr. Master,
   
   I need a catalog. Here
   are my facts
   
   http://www.dgcomputers.org/testimonials.php
   
   Agent
   
   facts
   
   

   View Slide

8. # puppetconf # puppetize @ puppetlabs
   facts
   
   Agent
   
   Thanks for you facts.
   
   I’ll just store them in
   PuppetDB
   
   PuppetDB
   
   

   View Slide

9. # puppetconf # puppetize @ puppetlabs
   PuppetDB
   
   facts
   
   ENC
   
   Agent
   
   Mr. ENC, is this host
   defined as an external
   node?
   
   Yep, he should be an
   apache server. Here is
   the definition
   
   nodes
   
   

   View Slide

10. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    Just compiled your
    catalog. One sec while
    I store it in PuppetDB.
    
    catalog
    
    

    View Slide

11. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    catalog
    
    Here is your
    catalog. Send me
    a report and let
    me know how it
    went!
    
    catalog
    
    

    View Slide

12. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    catalog
    
    I hate to be a
    bother, but can
    you compute
    the md5sums of
    a few files?
    
    catalog
    
    

    View Slide

13. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    Just finished applying.
    Here are the results.
    
    report
    
    catalog
    
    

    View Slide

14. Interacting with Puppet’s Data
    Termini and the indirector
    

    View Slide

15. # puppetconf # puppetize @ puppetlabs
    facts find from
    terminus facter
    
    Agent
    
    

    View Slide

16. # puppetconf # puppetize @ puppetlabs
    catalog find from
    terminus rest
    
    http://www.dgcomputers.org/testimonials.php
    
    Agent
    
    facts
    
    

    View Slide

17. # puppetconf # puppetize @ puppetlabs
    facts
    
    Agent
    
    facts save to terminus
    puppetdb
    
    PuppetDB
    
    

    View Slide

18. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    ENC
    
    Agent
    
    node find from
    terminus exec (or
    ldap)
    
    nodes
    
    

    View Slide

19. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    catalog find from
    terminus compiler
    
    catalog
    
    

    View Slide

20. # puppetconf # puppetize @ puppetlabs
    PuppetDB
    
    facts
    
    Agent
    
    catalog save to
    terminus puppetdb
    
    catalog
    
    

    View Slide

21. # puppetconf # puppetize @ puppetlabs
    Facter
    ENC
    
    Disecting a Puppet Run
    Compiler
    
    Config
    Catalogs
    
    Nodes/
    Manifest
    
    Reports
    
    Facts
    
    

    View Slide

22. # puppetconf # puppetize @ puppetlabs
    CLI commands
    puppet
    facts
    find
    
    puppet
    node
    find
    
    puppet
    catalog
    find
    
    

    View Slide

23. # puppetconf # puppetize @ puppetlabs
    CLI Puppet Facts
    # mkdir –p /tmp/yaml/facts
    # puppet facts find node_name --render-as yaml \
    > /tmp/yaml/facts/node_name.yaml
    

    View Slide

24. # puppetconf # puppetize @ puppetlabs
    Creating a node (optional):
    # puppet node find node_name \
    --node_terminus=exec \
    --external_nodes=/etc/puppet/nodes.sh \
    --facts_terminus=yaml \
    --clientyamldir=/tmp/yaml/ --render-as=yaml \
    > /tmp/yaml/nodes/node_name.yaml
    

    View Slide

25. # puppetconf # puppetize @ puppetlabs
    Creating a catalog:
    # puppet catalog find node_name \
    --facts_terminus=yaml \
    # puppet catalog find node_name \
    --node_terminus=yaml \
    --manifest=/etc/puppet/manifest/site.pp \
    --modulepath=/etc/puppet/modules/
    --clientyamldir=/tmp/yaml/ > /tmp/catalog.yaml
    

    View Slide

26. # puppetconf # puppetize @ puppetlabs
    Fun with IRB
    Puppet::Node::Facts.indirection.find
    facts
    
    Puppet::Node.new
    nodes
    
    Puppet::Catalog.indirection.find
    catalog
    
    

    View Slide

27. # puppetconf # puppetize @ puppetlabs
    IRB Facts
    irb:> require ‘puppet/face’
    > facts=Puppet::Face[:facts, :current].find('node’)
    

    View Slide

28. # puppetconf # puppetize @ puppetlabs
    Access a Fact value (irb):
    …
    > facts.values['ipaddress']
    => "10.0.2.15"
    

    View Slide

29. # puppetconf # puppetize @ puppetlabs
    Creating a node (from irb):
    …
    > node=Puppet::Node.new('node_name',
    {:classes => {:foo => {:bar => :baz}}})
    >node.merge(facts.values)
    

    View Slide

30. # puppetconf # puppetize @ puppetlabs
    Creating a catalog:
    …
    irb> catalog=Puppet::Resource::Catalog.indirection.\
    find('node_name’, :node => node)
    

    View Slide

31. Interacting with Puppet’s Data
    Use Cases
    

    View Slide

32. # puppetconf # puppetize @ puppetlabs
    Inspecting the catalog:
    •  What types are in the catalog?
    irb> catalog.resources.collect do |r| r.type end.uniq
    •  Gimme a resource:
    irb>catalog.resource(‘Package[httpd]’)
    

    View Slide

33. # puppetconf # puppetize @ puppetlabs
    Rspec Puppet:
    let :facts do
    {:operatingsystem => ‘Redhat’}
    end
    let :params do
    {:bind_address => ‘0.0.0.0’
    end
    it { should contain_file(‘/tmp/foo.conf’) }
    

    View Slide

34. # puppetconf # puppetize @ puppetlabs
    Thundering Herd
    Pre-compile catalogs for faster auto-scaling
    

    View Slide

35. # puppetconf # puppetize @ puppetlabs
    Applying pre-compiled
    catalogs:
    puppet apply --catalog /tmp/catalog.json –server
    puppet-fileserver
    

    View Slide

36. # puppetconf # puppetize @ puppetlabs
    DMZ
    tcp over USB
    

    View Slide

37. Interacting with Puppet’s Data
    Use Cases
    

    View Slide

38. # puppetconf # puppetize @ puppetlabs
    Hacking reports
    Everything in Puppet is a state transition
    User[‘dan’] : absent -> present
    User[‘dan’][‘shell’] -> ‘/sbin/nologin’ -> /bin/bash
    

    View Slide

39. # puppetconf # puppetize @ puppetlabs
    Setting up the agent:
    [agent]
    report=true
    

    View Slide

40. # puppetconf # puppetize @ puppetlabs
    Archive reports in your
    yamldir
    [master]
    reports = store
    

    View Slide

41. # puppetconf # puppetize @ puppetlabs
    Puppet reports
    $ cd `puppet config print reportdir`
    $ ls
    node1 node2 node3
    $ ls node1
    

    View Slide

42. # puppetconf # puppetize @ puppetlabs
    Every report from every run
    ever
    $ ls node1
    201206060256.yaml 201206060303.yaml
    201206060519.yaml 201206122349.yaml
    201206122354.yaml 201206130002.yaml
    

    View Slide

43. # puppetconf # puppetize @ puppetlabs
    Lets crack one open!
    Irb > require ‘yaml’
    >reports=YAML.load_file('201206130002.yaml')
    

    View Slide

44. # puppetconf # puppetize @ puppetlabs
    Have a look
    >(reports.methods - Object.methods).sort
    Notice the following methods:
    

    View Slide

45. # puppetconf # puppetize @ puppetlabs
    High level data
    > reports.exit_status
    ⇒ 0
    > reports.status
    => "unchanged"
    > reports.host
    ⇒ ”node1”
    

    View Slide

46. # puppetconf # puppetize @ puppetlabs
    metrics
    > reports.metrics.keys
    ⇒ ["resources", "events", "changes", "time"]
    > reports.metrics['resources']
    ⇒ [‘failed’, 0],[ ‘changed’, ‘7’]
    

    View Slide

47. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    > reports.resource_statuses.keys
    => ["Package[xinetd]", "File[/srv/node/1]",
    "Package[swift]", "Exec[compile fragments]",
    "Package[swift-container]", "File[/var/opt/lib/pe-
    puppet/concat/_etc_swift_object-server.conf]",
    "File[/etc/rsync.d/frag-account]”]
    

    View Slide

48. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    > status = reports.resource_statuses
    > status.keys
    => ["Package[xinetd]", "File[/srv/node/1]",
    "Package[swift]", "Exec[compile fragments]",
    "Package[swift-container]", "File[/var/opt/lib/pe-
    puppet/concat/_etc_swift_object-server.conf]",
    "File[/etc/rsync.d/frag-account]”]
    

    View Slide

49. # puppetconf # puppetize @ puppetlabs
    And the awesome sauce
    >events = status["File[/etc/swift/swift.conf]"].events
    > events.first.status
    ⇒ "success”
    > events.first.desired_value
    ⇒ :present
    > events.first.previous_value
    => :absent
    

    View Slide

50. Thank You
    Dan Bode| Puppet Labs
    dan@puppetlabs.com
    

    View Slide

51. View Slide