Configuration Management or: How I Learned to Stop Worrying and Love the Production Environment

Introduction to configuration management / puppet given at the 2013 Astronomical Data Center Technologies meeting (
http://www.astdatcentech.org/).

Transcript

1. Configuration Management or: How I Learned to Stop Worrying and Love the Production Environment Joshua Hoblitt National Optical Astronomy Observatory AstDatCenTec April 18-19, 2013
2. Overview • Identify the problem space • Personal experience with CM tools • Brief description of a few currently popular CM tools • Puppet Audience • Those that are not indoctrinated into DevOps culture • No experience with Puppet or other CM tools
3. Problem Space (and space is vast vast...)
4. Common Problems with manual configuration • Primates are error prone – Often “things” are not setup the way you think they are • Configuration verification is labor intensive – Most installation tools/scripts are brain dead • “Business logic” is often redeveloped over and over again – I know I configured X just last year... • Configuration data ends up being: – ephemeral – unversioned – unreproducible
5. Common Problems with manual configuration (cont.) • New software deployment across many systems ends up costing ~O(N) in labor • Software updates can be even more labor intensive than an initial deployment • Disaster recovery time == help! • By hand configuration scales at best linearly – FTEs = ~servers / admin_efficiency • Automated configuration has the potential to scale based on configuration complexity instead of the number of nodes – FTEs = (resources * resource_complexity) / admin_efficiency
6. Personal Experience with CM tools “I call it the law of the instrument, and it may be formulated as follows: Give a small boy a hammer, and he will find that everything he encounters needs pounding." – Abraham Kaplan – Abraham Kaplan
7. Timeline • 2000 : CFEngine 2.something – Never got it doing anything useful • ~2003 bcfg – Never got it working • ~2003 CFEngine 2.something – Still hadn't learned my lesson • ~2006-2009 rolled my own – Perl, ssh, rsync, make, and unionfs mounts – Static per host configs created by unioning <site>:<group>:<hostname> – Currently still in use managing ~100 hosts
8. Timeline (cont.) • 2010 Puppet – Desire to reduce complicated kickstart %postscripts – Update IDL license files on workstations – Started using TheForeman as an ENC • 2011 Puppet & TheForeman (1st project @ NOAO) – Image and configure brand new cluster hardware – Incrementally brought up across most of the server environment for common configs (ntp, resolv.conf, ssh, etc.) – Production applications currently being brought under umbrella
9. Configuration Management Tools "I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail." – – Abraham Maslow Abraham Maslow
10. The DevOps Influence • Adapting Software Engineering practices to IT operations – code reuse (write a function instead of cut'n'paste) → configuration automation via reusable modules/cookbooks/recipes/formulas/etc. – source version control → configuration data version control – unit & integration tests → configuration testing (and even CI) • The holy grail is to be able to treat infrastructure “as code”
11. Lots of tools in this problem space • CFEngine – Small, fast, scalable, few dependencies – Started 1993 – https://en.wikipedia.org/wiki/Promise_theory (you have been warned) – commercial support http://cfengine.com • Puppet – Server/agent model – DSL – Started 2005 – Commercial support http://puppetlabs.com
12. Lots of tools in this problem space (cont.) • Chef – Client/server/agent model – Ruby DSL – Started 2009 – Commercial support http://opscode.com • Salt – Publisher/subscriber – Started 2011 – Templated YAML – Commercial support http://saltstack.com
13. Lots of tools in this problem space (cont.) • Ansible – Manages over ssh* (temp daemon) – YAML – State machine for multinode dependencies – Started 2012 – Commercial support http://ansible.cc/
14. Puppet “When I was a kid, I never saw a puppet show. I never played with puppets or had any interest in them.” – Jim Henson – Jim Henson
15. Features of Puppet • Idempotent Idempotent – “relating to or being a mathematical quantity which when applied to itself under a given binary operation (as multiplication) equals itself; also : relating to or being an operation under which a mathematical quantity is idempotent “ - m-w.com Time –> Host state Manifest New resource defined State modified State converged State changed outside of puppet (sudo abuser;)
16. Features of Puppet (cont.) • Resource abstraction Resource abstraction – Resources are modeled with “Types” – One or more “Providers” per Type user { 'bob': uid => 567, ensure => present, } – The user Type has many Providers • useradd Provider on systems that include it with certain capabilities (linux) • directoryservice Provider on darwin • Providers for aix, hpux, solaris, ldap, windows, etc.
17. Features of Puppet (cont.) • Graph based relationships – Resource dependency is modeled as a DAG
18. Master/Agent Model Shamelessly stolen from: https://puppetlabs.com/wp-content/uploads/2012/03/PL_dataflow_notitle.png
19. Local execution mode $ sudo puppet resource user puppet user { 'puppet': ensure => 'present', comment => 'Puppet', gid => '52', home => '/var/lib/puppet', password => '!!', password_max_age => '-1', password_min_age => '-1', shell => '/sbin/nologin', uid => '52', } $ puppet apply --noop -e "file{ '/tmp/foo': }" Notice: Finished catalog run in 0.12 seconds $ puppet apply --noop ./users.pp Notice: Finished catalog run in 0.11 seconds
20. Facter • Surprisingly, it produces facts • Incredibly easy to add new facts in Ruby $ facter osfamily facterversion puppetversion virtual augeasversion uptime augeasversion => 0.10.0 facterversion => 2.0.0-rc4 osfamily => Gentoo puppetversion => 3.1.1 uptime => 27 days virtual => physical
21. Obligatory “Hello World” $ puppet apply -e "notify{ 'hello world': }" Notice: hello world Notice: /Stage[main]//Notify[hello world]/message: defined 'message' as 'hello world' Notice: Finished catalog run in 0.18 seconds
22. Basic Syntax • All resource declarations follow the same basic format <type> { '<title>': <attribute> => <value> <attribute> => <value> ... } user { 'puppet': ensure => 'present', # this is a metaparam comment => 'added by portage for puppet', gid => '101', home => '/var/lib/puppet', shell => '/bin/bash', uid => '112', }
23. Resource Ordering • The Puppet DSL is not an iterative language. Resources are applied based on dependency ordering. Declaration order does not matter. user{ 'bob': uid =. 123, gid => 'bob', ensure => 'present', require => Group['bob'], # <-- metaparam } group{ 'bob': gid => 123, ensure => 'present', }
24. Autorequire • Some Types automatically declare dependencies – This can not be done for Classes and defined Types in the Puppet DSL but it can be done for Types implemented in Ruby file {'/tmp/b/c': } file {'/tmp/': ensure => directory } file {'/tmp/d/': ensure => directory } file {'/tmp/d/e': } file {'/tmp/b/': ensure => directory } file {'/tmp/a': } file {'/tmp/d/f': }
25. Variables • Variables, like all declarations in the Puppet DSL are idempotent $uid = 567 user { 'bob': ensure => 'present', uid => $uid, } $ cat bar.pp $foo = "bar" $foo = "baz" $ puppet apply bar.pp Error: Cannot reassign variable foo at [...]/bar.pp:2 on node Error: Cannot reassign variable foo at [...]/bar.pp:2 on node
26. Classes • Classes are similar to a Type but may be declared only once in a manifest • If something needs to be declared multiple times with different titles, you need to use a defined Type (no example). class foo ($arg1 = 'default', $arg2) { user{ $arg1: ensure => $arg2, } } class{ 'foo': arg1 => 'bob', arg2 => 'present, }
27. Modules • Classes, defined Types, custom facts, and new Types & providers can be packaged into a module • Class/define name resolution is dependent upon the directory structure autofsck/ |-- Changelog |-- Gemfile |-- LICENSE |-- manifests | `-- init.pp |-- Modulefile |-- Rakefile |-- README.md |-- spec | |-- classes | | `-- autofsck_spec.rb | `-- spec_helper.rb `-- tests `-- init.pp http://forge.puppetlabs.com/jhoblitt/autofsck
28. Testing $ rake lint manifests/gmetad.pp - ERROR: trailing whitespace found on line 53 rake aborted! /home/jhoblitt/.gem/ruby/1.9.1/gems/puppet-lint-0.3.2/lib/puppet-lint/tasks/p /home/jhoblitt/.gem/ruby/1.9.1/gems/puppet-lint-0.3.2/lib/puppet-lint/tasks/p Tasks: TOP => lint (See full trace by running task with --trace) $ rake spec Cloning into 'spec/fixtures/modules/stdlib'... [...] HEAD is now at bebecd3 Merge branch 'update_gemspec' for 4.0.2 /usr/bin/ruby -S rspec spec/classes/gmetad_spec.rb spec/classes/gmond_spe dnsdomainname: Name or service not known ... Finished in 2.24 seconds 3 examples, 0 failures
29. Puppet Forge • Loosely modeled on CPAN, rubygems, etc. • Uses a two tier name space of <author>/<module name>
30. Forge Modules • The 1K module mark was hit earlier in this month (April)
31. CI
32. A few Puppet Links • http://docs.puppetlabs.com/ • http://forge.puppetlabs.com/ • http://cloudsmith.github.io/geppetto/ • https://travis-ci.org/ • https://github.com/rodjek/vim-puppet • #puppet & #puppet-dev on freenode `
33. Questions?