PuppetCamp NYC 2012: Puppet In the Cloud

1. PUPPET IN THE CLOUD Puppet Camp 2012: New York City April 27, 2012 Robert Terhaar rob@atlanticdynamic.com Atlantic Dynamic New York, NY 1 Saturday, April 28, 12
2. COMPANY & PERSONAL BIO • Built custom cloud & IT management systems for: • Finance // Bio-Tech // Start-ups // Advertising • Sysadmin since 1998 • Amazon EC2 for 5+ years & Puppet for 4+ years • Based in NYC 2 Saturday, April 28, 12
3. MANAGE YOUR CLOUD INFRASTRUCTURE... WITH PUPPET 3 Saturday, April 28, 12
4. CLOUD SERVER LIFE CYCLE 4 Saturday, April 28, 12
5. http://www.wired.com/dangerroom/2010/09/revealed-pentagons-craziest-powerpoint-slide-ever/ 5 Saturday, April 28, 12
6. KEEP IT SIMPLE. 6 Saturday, April 28, 12
7. CLOUD SERVER LIFE CYCLE Need Create Manage Destroy • Create • Provision & classify new instances • Maintain • Configure, manage change • Servers perform their function • Inception • “We need new servers!” • Destroy • Servers are decommissioned 7 Saturday, April 28, 12
8. IN DEPTH: CREATE / MANAGE Launch Install Classify Puppetize! • Install Puppet • msiexec.exe -i puppet.msi • Classify • Drop a fact onto the node • or... plug it into an ENC • Launch Instances • ~*~*API MAGIC*~*~ • Configure • puppet agent -t • (config finishes in one run because your code is AWESOME) 8 Saturday, April 28, 12
9. STEP ZERO: Provision a Puppet Master (in the cloud) 9 Saturday, April 28, 12
10. Puppet, without a Puppet Master? 10 Saturday, April 28, 12
11. How do I configure my Puppet Master... ? (to run in the cloud) 11 Saturday, April 28, 12
12. PROVISION A PUPPET MASTER • Don’t use the OS Packages (unless they’re >2.7) • Use Gem, apt.puppetlabs.com, or build your own packages • Google for: FPM • Use Puppet Enterprise! • Don’t use Ruby 1.8.5 (RHEL 5) • Apache/Nginx and passenger (puppetmasterd is for dev!) • DNS is important for the Puppetmaster! 12 Saturday, April 28, 12
13. • Hostname and FQDN config: •$ hostname •-> puppet •$ hostname -f •-> puppet.myfqdn.com PROVISION A PUPPET MASTER 13 Saturday, April 28, 12
14. • DNS: • For EC2: • Assign an Elastic IP to your Puppet Master • Create a DNS CNAME entry • puppet.myfqdn.com => ec2-23-20-187-61.compute-1.amazonaws.com • For Others (Linode, Rackspace, et al.): • Your (future) nodes need to find puppet.myfqdn.com PROVISION A PUPPET MASTER 14 Saturday, April 28, 12
15. STEP ONE: Start some instances! 15 Saturday, April 28, 12
16. What is so complicated about creating instances... in the cloud? 16 Saturday, April 28, 12
17. • Vendor’s web control panel • puppet node bootstrap (Cloud Provisioner) • 3rd Party GUI Management platforms: • StackHammer / Scalr / Foreman / Rightscale • CLI Tools: • StarCluster • ec2-tools & euca-tools • do-it.sh • EC2 Auto-scaling Group STARTING INSTANCES 17 Saturday, April 28, 12
18. STARTING INSTANCES • Vendor’s web control panel • Good for getting started • Don’t build gold images • (don’t rebuild images with Puppet pre-installed) 18 Saturday, April 28, 12
19. STARTING INSTANCES • puppet node bootstrap • Great for periodically creating a few nodes 19 Saturday, April 28, 12
20. STARTING INSTANCES • 3rd Party GUI Management platforms: • Use them for provisioning, not instance configuration • (It’s Puppet’s job to configure your servers) • Hopefully the tool you use has Puppet support • Depends on your workflow 20 Saturday, April 28, 12
21. STARTING INSTANCES • Other CLI Tools • Only limited by your imagination! • AWS Cloud Formation • Need to create and provision 1,000 instances in < 15mns? • (Use StarCluster) 21 Saturday, April 28, 12
22. How to install the Puppet client on instances after they boot? 22 Saturday, April 28, 12
23. INSTALL PUPPET CLIENT • cloud_init (Ubuntu and AWS AMIs) • https://help.ubuntu.com/community/CloudInit • StarCluster • puppet node bootstrap • Chef (yes, really...) • 3rd Party Management Platform • Your own custom tool that SSHs onto the instances 23 Saturday, April 28, 12
24. CONFIGURE PUPPET CLIENT • Puppet versions on client and master should match • Don’t use OS Packages 24 Saturday, April 28, 12
25. How do nodes connect to their Puppet Master... in the cloud? 25 Saturday, April 28, 12
26. CONNECT INSTANCES • Generic instance boots & Puppet is installed • Puppet is configured on your new instance • echo "pluginsync=true" >> /etc/puppet/puppet.conf • echo "certname=some_unique_id" >> /etc/puppet/puppet.conf • echo "server=puppet.myfqdn" >> /etc/puppet/puppet.conf • mkdir -p /etc/facter/facts.d • echo server_role=web > /etc/facter/facts.d/server_role.txt • Puppet is run for the first time • puppet agent -t 26 Saturday, April 28, 12
27. CONNECT INSTANCES AUTO-SIGNING • “Certificate auto-signing is a great solution” • Fast! • Easy! • Fun! • “Certificate auto-signing is an awful solution!” • It’s insecure! (use a firewall!!!) • It’s insane! 27 Saturday, April 28, 12
28. CONNECT INSTANCES AUTO-SIGNING • Here’s how to do it anyway (on your Puppet Master) • echo "*.myfqdn.com" > /etc/puppet/autosign.conf 28 Saturday, April 28, 12
29. CERTNAME, HUH? How to name my servers in the cloud? 29 Saturday, April 28, 12
30. SERVER NAMES: DISTINCT OR “CLOUDY” • Distinct: • db1.prod.myapp.com • (role.environment.domain) • Cloudy: • web-<uuid>.prod.myapp.com • web-6ba7b810-9dad-11d1-80b4-00c04fd430c8.prod.myapp.com • (role-<uuid>.environment.domain) • RFC-4122 30 Saturday, April 28, 12
31. UUIDS FOR HOSTNAMES??? WHAT ABOUT DNS? @@host {"${fqdn}-private": name => $fqdn, host_aliases => $hostname, ip => $ipaddress_eth0, tag => "cloud_server", } Host <<| tag == "cloud_server" |>> 31 Saturday, April 28, 12
32. Storeconfigs http://garretthoneycutt.com/index.php/Public_Speaking 32 Saturday, April 28, 12
33. WHAT ABOUT DNS? EXAMPLE /ETC/HOSTS $ cat /etc/hosts # HEADER: This file was autogenerated at Thu Apr 05 00:06:56 +0000 2012 # HEADER: by puppet. While it can still be managed manually, it # HEADER: is definitely not recommended. 127.0.0.1 localhost.localdomain localhost 10.193.200.175 puppet.dev.myfqdn.com puppet 10.90.181.232 demo.dev.myfqdn.com demo 10.252.41.131 queue1.dev.myfqdn.com queue1 10.242.243.2 webapp-1305060569.dev.myfqdn.com webapp-1305060569 10.90.192.85 ci.dev.myfqdn.com ci 10.122.66.163 db1.dev.myfqdn.com db1 10.87.87.130 webapp-1333394643.dev.myfqdn.com webapp-1333394643 10.252.170.34 solr1.dev.myfqdn.com solr1 10.80.192.68 solr2.dev.myfqdn.com solr2 10.123.47.35 db-slave-1326995752.dev.myfqdn.com db-slave-1326995752 10.222.241.2 webapp-1305061505.dev.myfqdn.com webapp-1305061505 10.252.148.26 webapp-1305064507.dev.myfqdn.com webapp-1305064507 33 Saturday, April 28, 12
34. DETOUR: NAGIOS & STORECONFIGS class apache2::monitor { @@nagios_service { "check_http_${hostname}": check_command => $http_check_command, use => 'generic-service', host_name => "$fqdn", notification_period => '24x7', service_description => "${hostname}_check_http", contact_groups => 'admins', notify => Service['nagios3'], require => Class['nagios'], } } class nagios::realize { Nagios_host <<| |>> Nagios_service <<| |>> } 34 Saturday, April 28, 12
35. ON YOUR PUPPET MASTER thin_storeconfigs = true 35 Saturday, April 28, 12
36. NODES.PP case $server_role { "web": { include role::web } "database":{ include role::db } default: { fail("Role is undefined") } } node /^db1(?:\.\w+)?\.myapp\.com$/ inherits 'basenode' { include mysql::server include users::admins include kitchen::sink } Do this: Not this (in the cloud) 36 Saturday, April 28, 12
37. WHERE DID $server_role COME FROM? https://github.com/puppetlabs/puppetlabs-stdlib facter_dot_d: # mkdir -p /etc/facter/facts.d # echo "server_role=web" > /etc/facter/facts.d/server_role.txt On your nodes 37 Saturday, April 28, 12
38. How to find my servers in the cloud? 38 Saturday, April 28, 12
39. ORGANIZE YOUR CLOUD • Mcollective & mc-ssh • Puppet Dashboard / Foreman • Run multiple Puppet Masters • One master per cluster, or application • One master per region • Build loosely-coupled systems • Your provisioning system should tag your instances • cluster, environment, server_role, etc... 39 Saturday, April 28, 12
40. How do I clean up references to my servers, when I’m done using them? 40 Saturday, April 28, 12
41. CLEAN OLD NODES • puppet node clean • allow_duplicate_certs = true 41 Saturday, April 28, 12
42. Questions? Robert Terhaar rob@atlanticdynamic.com Atlantic Dynamic New York, NY 42 Saturday, April 28, 12