A year with Puppet

Transcript

1. learning by mistakes & pain a year with Puppet
2. @codec Patrick Otto Chief Puppeteer (Sysadmin) Mayflower GmbH Operations & development background In — with Puppet since ’09
3. 70 employees (mostly developers) 3 Sysadmins to rule them all PHP & JavaScript, Web- application development — Open Source
4. Corporate services Mail Network VoIP *aaS legacy systems
5. 49 modules Infrastructure in git ~900 MB ~5000 lines of manifests ~50 nodes 2630 commits
6. the workflow Hack, hack, hack. commit push deploy run Puppet
7. the workflow Hack, hack, hack. commit push deploy run Puppet Finished: FAILURE manifests/nodes/git.dmz.muc.mayflower.de.pp: 49:trailing_whitespace:ERROR:trailing whitespace found Fix (typo|whitespace|...), rage ...
8. the workflow Hack, hack, hack. commit push deploy run Puppet Finished: SUCCESS Triggering a new build of puppet-admin deployment #15422 Fix (typo|whitespace|...), rage ...
9. the workflow Hack, hack, hack. commit push deploy run Puppet
10. the workflow Everything is in production work happens on master (mainline-style) refactoring becomes a pain
11. the workflow Everything is ON FIRE ... easily use named branches & pull requests use branches as environments
12. the workflow Testing Jenkins only validates style & syntax before deployment rspec-puppet is your best bet we suck at this
13. deb31f8 fix syntax error 63501d9 syntax fix fa17b28 module syntax phu d02dbdc deprecated syntax 96d37fa wrong syntax de4509b syntax.. 649168f fix syntax b13c676 stupid syntax the workflow lint & validate
14. deb31f8 fix syntax error 63501d9 syntax fix fa17b28 module syntax phu d02dbdc deprecated syntax 96d37fa wrong syntax de4509b syntax.. 649168f fix syntax b13c676 stupid syntax the workflow lint & validate it’s your job - not Jenkins’ use a pre-commit hook use an easy to setup local environment
15. the workflow modules git submodules are madness hack, commit & push module pull & push super-repo # Hack on your module $ git commit -m ‘fix some stuff’ $ git push # Update your super-repo $ cd ../puppet-admin $ git rm modules/dhcp $ vi .gitmodules $ git commit -m ‘removed dhcp for update’ $ git submodule add ... modules/dhcp $ git add modules/dhcp .gitmodules $ git commit -m ‘updated dhcp’ $ git push
16. the workflow modules git submodules are madness hack, commit & push module pull & push super-repo librarian-puppet to the rescue “Puppetfile” in your repository install from forge & git like bundler for Ruby # Puppetfile forge "http://forge.puppetlabs.com" # From the forge mod "puppetlabs/stdlib" mod "puppetlabs/apt" mod "puppetlabs/mysql" mod "puppetlabs/rabbitmq" mod "puppetlabs/git" mod "puppetlabs/rsync" mod "puppetlabs/xinetd" mod "puppetlabs/apache" mod "puppetlabs/mcollective" mod "puppetlabs/nova" mod "puppetlabs/glance" mod "puppetlabs/swift" mod "puppetlabs/keystone" mod "puppetlabs/horizon" mod "puppetlabs/openstack" # Forked by Mayflower mod "virt", :git => "https://github.com/Mayflower/puppet-virt.git", :ref => 'lxc-lvm' mod "users", :git => "git@git.dmz.muc.mayflower.de:puppet-users.git" mod "php", :git => "https://github.com/Mayflower/puppet-php.git" mod "bind", :git => "https://github.com/codec/puppet-bind.git" mod "dhcp", :git => "https://github.com/Mayflower/puppet-dhcp.git" mod "radvd", :git => "https://github.com/fpletz/puppet-radvd.git" mod "nginx", :git => "https://github.com/Mayflower/puppetlabs-nginx.git" mod "xmpp", :git => "https://github.com/codec/puppet-xmpp.git"
17. the workflow deployment lint & validate SSH to master git pull git submodule sync git submodule update ...
18. the workflow deployment lint & validate SSH to master git pull git submodule sync git submodule update ... Keep it simple.
19. Puppet in action Exported resources glue things together Modules in Puppet fork, fix & abstract Infrastructure as code is just as hard as software
20. infrastructure as code Design think in applications, profiles & roles node manifests with hundreds of lines are unmaintainable Local environment keep it easy to set up and run (avoid Cowboys on production) handle your repo like a software project
21. environment Vagrant Multi-VM support, plugin for DNS, run your own master VirtualBox for now ... “node definitions” can be hacked ## Hackety, hack. mocked_nodes = ENV['VAGRANT_NODES'] ? ENV['VAGRANT_NODES'].split(',') : ['node'] mocked_nodes.each_with_index do |host,index| index += 5 config.vm.define host do |config| config.vm.host_name = "#{host.to_s}.dev" # FIXME: this shouldn't include the tld config.vm.provision :puppet_server do |puppet| puppet.options = ['--verbose', '--no-daemonize', '--onetime'] puppet.puppet_server = 'puppet' end config.vm.network :hostonly, "192.168.172.#{index.to_s}" # FIXME: hackhackhack end end
22. Export Define resources ... define webserver::forward (...) { # ... @@webserver::vhost::proxy { $name: ensure => present, aliases => $aliases, backend => $backend ? { undef => "http://${::ipaddress}", default => $backend, }, rewrites => $rewrites ? { undef => undef, default => $rewrite_hash, }, protocol => $protocol, read_timeout => $read_timeout, tag => "webserver_${frontend}", } } node ‘jira’ { # ... webserver::forward { 'bugs.example.com': frontend => "proxy.example.com", backend => "http://${::fqdn}:8080", protocol => 'ssl' } }
23. Collect ... and collect across nodes node ‘proxy.example.com’ { # ... Webserver::Vhost::Proxy <<| tag == "webserver_${::fqdn}" |>> Nginx::Resource::Vhost <| tag == "webserver_${::fqdn}_ssl" |> { ssl_cert => "/etc/nginx/ssl/${::fqdn}.crt", ssl_key => "/etc/nginx/ssl/${::fqdn}.key", } } Glue things together Nagios databases DHCP / DNS SSH keys ...
24. Modules in puppet Plenty of modules in the sea, but ... hard to find i.e., GitHub abandonded code with pull requests & duplicated effort quality is a problem too specialized ಠ_ಠ I’m looking at you, PuppetLabs!
25. Modules in puppet The Puppet Forge only a few modules, ~6 months ago much better since the last few weeks publish your modules! Writing modules check example42 & puppetlabs modules
26. Questions?