DevOps Sydney - Puppet Hiera Data Extraction

A talk I prepared for Devops Sydney Meetup group, to try and help people get their head around what problems Hiera addresses, how to implement, test and debug Hiera from command line, and how best to use it in a Puppet manifest.

I hope it's useful.

:)

Transcript

1. It’s all about data separation! (A crash course in Hiera) Devops Sydney Meetup Jan 2014 Stephen Wallace
2. Who are you, and why the funny accent? Stephen Wallace 20 years IT experience Predom *nix / sysadmin background Senior management - teams up to 20 Large, heavily integrated systems Health, Online Gaming, Recruitment ICE Systems 2 yrs....one of the Puppet go to guys stephen.wallace@icesystems.com.au
3. The Scenario… We wish to write a module to install a webserver “We only use Redhat Enterprise Linux!” •  `yum install httpd` - easy eh •  Done and dusted…I’m off to the pub
4. A quick word later…. We wish to write a module to install a webserver We use Redhat Enterprise Linux! …..and a bit of Oracle Linux ………and a bit of Ubuntu come of think of it! `yum install httpd` - Couple of Debian challenges?
5. An ideal solution? OK…so we need to be able to accommodate multiple operating systems…….with multiple package names…..yuck! platform='unknown’ real_webserver=‘unknown’ unamestr=`uname` if [[ "$unamestr" == 'Linux' ]]; then platform='linux’ real_webserver=’httpd’ ……sometimes…. elif [[ "$unamestr" == 'FreeBSD' ]]; then platform='freebsd’ fi …………ETC…….AAAARGH!!!!!
6. Just as easy to make the same mistakes in Puppet! You could do this... centos, redhat, oel, oraclelinux, linux: { $supported = true $webserver = [ "apache2" ] $svc_name = "apache2" $config = "/etc/apache2/httpd.conf" if $::operatingsystemrelease =~ /^5/ { $config_tpl = 'httpd.conf.el.erb' } elsif $::operatingsystemrelease =~ /^6/ { $config_tpl = 'httpd.conf.el6.erb' } else { fail("the webserver module doesn't know what template to use for your $ {::operatingsystemrelease}")
7. Thinking caps on! Wouldn’t it be nice if we could externalise, and centralise the data from the code?
8. Who am I talking to? §  Puppet newbs! §  The guilty….you know who you are……!
9. So What Is This Hiera Thing?! §  A simple pluggable, hierarchical database §  Yaml §  MySQL §  Json §  LDAP, etc. Write your own! §  Included with Puppet Enterprise, gem install for opensource §  Allows data to be separated from code §  Simplified, more supportable manifests
10. How does Hiera work? hiera.yaml…you know conf files...right?! :hierarchy: - nodes/%{::fqdn} - %{::operatingsystem} - common :backends: - yaml - eyaml :yaml: :datadir: '/etc/puppetlabs/hieradata’
11. Hiera from CLI :hierarchy: - nodes/"%{::fqdn}" - "%{::operatingsystem}” - common -------------------- bash-3.2# facter operatingsystem Darwin bash-3.2# cat Darwin.yaml webserver: httpd-4-apple bash-3.2# hiera webserver apache2 bash-3.2# eh? Get you %{::facts} on the table!
12. Hiera from CLI…cont :hierarchy: - nodes/"%{::fqdn}" - "%{::operatingsystem}” <******** Need to feed hiera this fact when testing - "%{::environment}" - common bash-3.2# hiera webserver apache2 bash-3.2# hiera webserver operatingsystem=Darwin httpd-4-apple J
13. Hiera debugging! Hiera is not giving me the expected answer! bash-3.2# hiera webserver operatingsystem=Darwin --debug DEBUG: Thu Jan 16 17:09:05 +1100 2014: Hiera YAML backend starting DEBUG: Thu Jan 16 17:09:05 +1100 2014: Looking up webserver in YAML backend DEBUG: Thu Jan 16 17:09:05 +1100 2014: Looking for data source nodes/"" DEBUG: Thu Jan 16 17:09:05 +1100 2014: Cannot find datafile /etc/hieradb/ nodes/"".yaml, skipping DEBUG: Thu Jan 16 17:09:05 +1100 2014: Looking for data source Darwin DEBUG: Thu Jan 16 17:09:05 +1100 2014: Found webserver in Darwin httpd-4-apple …..nice.
14. Hiera/YAML Arrays? my_adminusers: simon: uid: 1001 gid: simon managehome: true groups: - wheel stephen: uid: 1003 gid: stephen my_admingroups: simon: gid: 1001 chris: gid: 1002 stephen: gid: 1003
15. Hiera Arrays…done! bash-3.2# hiera my_adminusers {"simon"=> {"uid"=>1001, "managehome"=>true, "gid"=>"simon", "groups"=>["wheel"]}, "stephen"=>{"uid"=>1003, "gid"=>"stephen"}} bash-3.2# hiera my_admingroups {"simon"=>{"gid"=>1001}, "chris"=>{"gid"=>1002}, "stephen"=>{"gid"=>1003}} bash-3.2#
16. Hiera n Puppet Automated Data Bindings in Puppet 3! # Automatically get the structured data: $real_webserver = hiera(’webserver’) …and with defaults # Get the structured data, and add a default: $real_webserver = hiera(’webserver’, nil) (If $webserver != ‘nil’…..then install the package!)
17. The Hiera/Puppet Magic? How do I use Puppet and Hiera to modify lists of resources? For example; •  How do I install a list of packages? •  How do I assure a list of packages are always uninstalled? •  How to install a list of users •  How to uninstall a list of users ……..I’m glad you asked J
18. A Sneaky Clue…… Using the adminusers and groups from the previous YAML file….. class my_adminusers { $groups = hiera('my_admingroups') $users = hiera('my_adminusers') if ! $groups { fail('No admin groups available') } if ! $users { fail('No admin groups available') } create_resources(user,$users) create_resources(group,$groups) }
19. Ah ha…but I have secrets! OK…so you don’t want to publish some data in plain text YAML! Encrypt the entire file? gem install hiera-gpg rootpwd: jhns732ns # gpg --trust-model=always --homedir=/etc/puppet/keyrings/live --encrypt -o mysql.gpg - r puppet.live.mycorp.com mysql.yaml # rm mysql.yaml # ls mysql.yaml.gpg # hiera -c /etc/puppet/hiera.yaml rootpwd calling_module=mysql env=live Jhns732ns ……………….OR, just the data items you want, using hiera-eyaml
20. In closing… Can Hiera Help? If you find yourself doing if, then, else, elseif…. If you love the concept of centralising all of your data…. If you want to simplify your manifests, make them more supportable…AND impress your friends and family….
21. Thanks! Linkedin: http://au.linkedin.com/in/stephenwallace