From Zero to Puppet - PuppetConf 2012

Switching from nothing to everything in Puppet config management and how we use it for automating failover at Server Density.

Presented at http://puppetconf.com 2012 by Pedro Pessoa.

Video at https://www.youtube.com/watch?v=4fvFSzWEgvc

Transcript

1. From zero to Puppet http://www.rankpop.com/you-need-to-start-structuring-your-blog-posts-asap/
3. +15TB / mth +1bn docs /mth 2-5k inserts/s @ 3ms 10K RPM @ 140ms
4. Servers HTTP Load Balancer – 5x Apache - 14x Build - 2x MongoDB - 19x data, 13x routing, 6x configuration, 6x arbiter,
5. HTTP Load Balancer $globalIPs_array = split($globalIPs, ',') <% globalIPs_array.each do |globalIP| ­%> ListenHTTP # primary public IP address Address <%= globalIP %> Port 80
6. Apache
7. file { 'vhost': path => "/etc/apache2/sites­enabled/${::siteDomain}", ensure => file, content => template('apache­php/vhost.erb'), notify => Service['apache2'], } Apache
8. <VirtualHost *:80> ServerName <%= siteDomain %> DocumentRoot <%= documentRoot %> ... ErrorLog /var/log/apache2/error­<%= siteDomain %>.log <% if @requestLogging and requestLogging == "yes" %> CustomLog /var/log/apache2/access­<%= siteDomain %>.log vhost_combined <% end %> <% if enableSSL == "yes" %> <VirtualHost *:443> SSLEngine On SSLCertificateFile /var/www/ssl/<%= siteDomain %>.crt SSLCertificateKeyFile /var/www/ssl/<%= siteDomain %>.key ... Apache
9. $excess_bagage = [ "ppp", "bind9­host", "laptop­detect", "open­iscsi", "libnss3:i386" ] package { $excess_bagage: ensure => purged, }
10. exec { 'ufw allow is­1': command => '/usr/sbin/ufw allow from 184.173.178.67', unless => '/usr/sbin/ufw status verbose | grep "184.173.178.67"', } exec { 'ufw enable': command => '/usr/sbin/ufw enable', unless => '/usr/sbin/ufw status | grep "Status: active"', Require => [Exec['ufw allow is­1']], notify => Exec['ufw reload'], } exec { 'ufw reload': command => '/usr/sbin/ufw reload', require => Exec['ufw enable'], refreshonly => true, }
11. SSH file { 'sshd_config': path => '/etc/ssh/sshd_config', ensure => file, content => template('sshd/sshd_config.erb'), } Port 22 Protocol 2 AllowUsers david pessoa chris rob wes Compression yes ... <% if @duoSecKey and @duoIntKey and @duoHost %> ForceCommand /usr/sbin/login_duo PermitTunnel no AllowTcpForwarding no <% end %>
12. System updates CANARIES=" \ aws­prod­apac­singapore­exm­a1 \ a2.wdc.sl \ exm1.wdc.sl \ mtx­web1.wdc.sl \ sdapp­web1.wdc.sl \ sdapi­web1.wdc.sl \ " function canaries { echo "REMOVE *­web1.wdc.sl FROM ROTATION!" echo "press ENTER" read echo "have you REALLY removed *­web1.wdc.sl from rotation?" echo "press ENTER" read for i in $CANARIES do echo $i mco rpc puppetral ­I $i create type=exec \ title="/bin/bash ­c 'apt­get dist­upgrade ­y'" done }
13. System updates function reboot_canaries { echo "REMOVE *­web1.wdc.sl FROM ROTATION!" echo "press ENTER" read echo "have you REALLY removed *­web1.wdc.sl from rotation?" echo "press ENTER" read for i in $CANARIES do echo $i mco rpc puppetral ­I $i create type=exec \ title="/bin/bash ­c 'reboot'" echo "­­­­­­­­­­­­­­" done }
14. What to Reboot mco rpc puppetral create type=exec \ title="/bin/bash ­c 'ls /var/run/reboot­required'" \ | grep ­B 1 "Resource was created" \ | grep ".sl" exm­md1a.wdc.sl Status: Resource was created Resource: {"tags"=>["exec"], "title"=>"/bin/bash ­c 'ls /var/run/reboot­required'", "type"=>"Exec", "parameters"=>{:returns=>:notrun}, "exported"=>false} exm1.wdc.sl Status: change from notrun to 0 failed: /bin/bash ­c 'ls /var/run/reboot­required' returned 2 instead of one of [0]
15. Live Management - Emergency # Metrics <%= metricsIP %> metrics­svc # Infrastructure services 184.173.178.66 puppet ...
16. Live Management - Emergency
17. Live Management - Emergency
18. Live Management - Emergency
19. Live Management - Emergency
20. Pedro Pessoa pessoa@serverdensity.com www.serverdensity.com