Being a Puppet Master: Automating Amazon EC2 with Puppet & Friends

Infrastructure as a Service (IaaS) has been a huge market for 'Cloud Computing', especially for the likes of Amazon and Rackspace. Despite the ubiquity of solutions to provide a company with quick-to-deploy virtual machines, the lack of a configuration management solution leaves most administrators wondering what the advantage to this all really is.

Puppet is an open source data center automation and configuration management framework. Puppet provides system administrators with a simplified platform that allows for consistent, transparent, and flexible systems management.

Puppet lets System Administrators spend less time on mundane tasks and instead focus on managing their infrastructure strategically.

Transcript

1. Being a Puppet Master Automating Amazon EC2 with Puppet & Friends Mark Stanislav <mark.stanislav@gmail.com>
2. Puppet: A Quick Overview Stop administrating your environment and start developing it... Re-usable code for managing your software & configurations Provides a Domain Specific Language (DSL) to script with Classes, conditionals, selectors, variables, basic math, etc. Supports Linux, Solaris, BSD, OS X; Windows in process! Project ran by Luke Kanies; Founder/CEO of Puppet Labs $5M Series B in July 2010; ~$7M total funding CFEngine & Chef are similar projects; both are quality, too. Sun, Stanford, Match.com, Media Temple, & Digg all use it!
3. High-Level Puppet Overview Puppet Master Modules Configuration Puppet Clients Monitoring Nagios/Munin DNS BIND Nameserver Development Apache, Tomcat, Passenger Testing Apache, Tomcat, Passenger Review Apache, Tomcat, Passenger Production Apache, Tomcat, Passenger Syslog rsyslog Server LDAP OpenLDAP Server Software Development Environments General Cloud Infrastructure
4. Puppet Network Overview Puppet Master Puppet Clients 8140/TCP Client Initiated puppetd -t 8139/TCP Server Initiated puppetrun Configuration allows for manual synchronizations or a set increment Client or server initiated synchronizations Client/Server configuration leverages a Certificate Authority (CA) on the Puppet Master to sign client certificates to verify authenticity Transmissions of all data between a master & client are encrypted SSL
5. Why EC2 IaaS is Tiring... An Amazon Machine Image (AMI) is very inflexible Building and deploying a new AMI is time consuming “What do you mean you want to update a file? We can’t just do that...” Auto-scaling is fantastic but managing the scaling hosts is not Time to deploy & configure offsets benefits of IaaS =
6. Puppet is an EC2 Superhero Deployment of a “base” EC2 AMI - just what you always need on any standard image Leverage EC2 security groups to give context to a new instance Puppet knows what you want out of the box; configure a new instance without interaction Update a package or configuration file at anytime
7. New EC2 Puppet Client Flow EC2 Instance With Puppet Spawned Puppet Service Starts For First Time Client Generates SSL Certificate Client Sends SSL Certificate to Master Master Signs SSL Certificate Puppet Client Synchronizes Puppet Master can allow certain domain scopes (*.example.com) to be auto-signed when asked by a valid hostname Create a crontab script that executes every minute looking for new SSL certificates in a certain directory and signs them Auto-sign everything you are asked to sign without question Manually sign each certificate when you add a new Puppet Client Methods to Sign Client SSL Certificates:
8. Puppet Module Structure Module Folder manifests/ Tell the module how to work files/ Static files needed for deployment templates/ Dynamic Ruby-based templates lib/ Relevant Ruby-based libraries
9. A Partial List of Puppet ‘types’ Files & Directories Users & Groups Services Packages Crontabs /etc/hosts Mail Aliases Mount Points Nagios SELinux SSH Keys Yum Repos Packages: Supports 23 different package providers Abstracted for your OS automatically Specify ‘installed’, ‘absent’, or ‘latest’ for desired state Change from ‘installed’ to ‘latest’ and deploy for quick upgrade Services: Supports 10 different ‘init’ frameworks Control whether a service starts on boot or is required to be running always A service can be notified to restart if a configuration file has been changed Files/Directories: Specify ownership & permissions Load content from ‘files/’, ‘templates/’ or custom strings Create symlinks Supports 5 types to verify a file checksum Purge a directory of files not ‘maintained’
10. General Puppet Syntax Class Configuration: If-Else Conditionals: if ($ec2_security_groups == ‘DNS’) { include bind::server } else { include bind::client } Set a Variable: $lib_path = “/usr/local/lib64/” Case Statements: case $ec2_security_groups { Monitoring: { include nagios } Developer: { include mercurial } } Single Class: class ntp { ... } Inherited Class: class sftp inherits ssh { ... } Nested Class: class foo { class bar { ... } } Scoped Class: class ntp::base { ... } Basic Math: $file_size = $bytes * 1024 Selectors: $admin = $user_id ? { ‘0’ => ‘root’, }
11. A Simple NTP Puppet Module ntpd/manifests/init.pp: class ntp { package { "ntp": ensure => latest } service { "ntpd": ensure => running, enable => true, hasrestart => true, hasstatus => true, require => Package["ntp"], } file { "/etc/ntp.conf": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntp.conf", notify => Service["ntpd"]; "/etc/sysconfig/ntpd": ensure => present, owner => root, group => root, mode => 0644, source => "puppet:///modules/ntp/ntpd", notify => Service["ntpd"]; } } ntpd/files/ntp.conf: restrict default kod nomodify notrap nopeer noquery restrict 127.0.0.1 server nist.netservicesgroup.com server time.nist.gov server time-a.nist.gov server time-b.nist.gov server 127.127.1.0 fudge 127.127.1.0 stratum 10 driftfile /var/lib/ntp/drift keys /etc/ntp/keys ntpd/files/ntpd: OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid" SYNC_HWCLOCK=yes NTPDATE_OPTIONS="-g -x"
12. EC2 Security Group Magic EC2 security groups are a named set of inbound firewall rules for a given instance Puppet can learn about EC2 meta-data very easily Tell Puppet to configure instances based on their security group Scales for 1 instance or 100 Rinse and repeat for each service group you have
13. EC2 Security Groups + Puppet 22/TCP for SSH for remote access 53/{TCP,UDP} for DNS nameserver ‘DNS’ EC2 Security Group: Inbound Firewall Rules: ssh - SSH server configuration bind - BIND nameserver Puppet Modules Enabled: The Puppet type ‘file’ allows for variable-replacement in filenames and use-on-first-match file { "/etc/ssh/sshd_config": source => [ “puppet:///modules/ssh/{$ec2_security_groups}-sshd_config”, "puppet:///modules/ssh/sshd_config" ]; } Puppet will use ‘DNS-sshd_config’ if it exists. If the file does not exist, it will use ‘sshd_config’
14. Client Meta-Data with Facter Retrieve useful ‘facts’ about a client host to determine how to interact with it. ‘facter’ quickly inventories all system metrics! Examples: $architecture - Create files that are based on architecture $hostname/$ip_address_eth0 - Create an /etc/hosts entry $uptime_days - Update all packages after 30 days uptime $selinux - Configure packages based on SELinux contexts $operatingsystemrelease - Run OS version specific tasks $is_virtual - Configure hosts based on VM vs. Physical $ec2_ami_id - Update configuration for the EC2 AMI used
15. Nagios ‘Type’ Support Puppet natively supports creating Nagios configuration Easily generate specific configuration for n hosts automatically Never again manually include hosts/services in groupings @@nagios_service { "load_check_${hostname}": service_description => "Load Averages", check_command => "load_check!3!5", host_name => "$fqdn", use => "generic-service"; } Nagios Service: @@nagios_hostgroup { "load_balancers": alias => "Load Balancers"; } Nagios Host Group: @@nagios_servicegroup { "apache_servers": alias => "Apache Servers"; } Nagios Service Group: @@nagios_host { $fqdn: ensure => present, hostgroups => "ldap", use => "generic-host"; } Nagios Host:
16. Puppet Generated Host/Service Checks
17. Puppet Generated Munin Metrics/Groupings
18. The Foreman: A Heavy Lifter Easy-to-use Puppet web interface for many tasks Review Puppet reports regarding your hosts easily Edit host facts and groups LDAP authentication Statistical graphs for metrics Execute puppetrun on hosts Provision hosts from the web
19. The Foreman ‘Overview’ Page
20. Interact with ‘Facter Facts’
21. Evaluate Puppet Efficiency with Reports
22. General Statistics for Puppet Clients
23. Marionette Collective Manage/Control/Execute: Services Packages Process Information Facter Facts Pings Decide which hosts you act upon by any Facter Fact Easily manage a large amount of diverse hosts
24. View Any Service’s Status Across Hosts
25. Check Versions That Are Installed
26. View Processes On Hosts Matching a ‘Fact’
27. Quickly Retrieve a List of MCollective Hosts
28. Consider This Scenario 1. You reserve 10 Elastic IPs for a network of hosts 2. Each instance starts and Puppet gives it an elastic IP 3. Based on an ‘IP -> NEED’ map, each new instance is created for a specific need (DNS, WWW, IMAP, etc.) 4. Hosts that become ‘WWW’ servers automatically are added to the Elastic Load Balancer (ELB) instance 5. Nagios & Munin configuration is done automatically 6. If an instance dies, the next time a new instance starts it is given the old host’s IP and that service is fulfilled again ...most importantly, you’ve done nothing :)
29. Take Your Environment Back Puppet: Provides you with the means to handle ad-hoc EC2 instance scaling with granular updates/configuration changes based on any ‘Fact’ you can supply. The Foreman: Manage your hosts from a well designed front-end. View reports, check for deployment efficiency, get the ‘big picture’ on your infrastructure; even deploy hosts from scratch! MCollective: Handle your mass administrative tasks with consistency and structure. Utilize ‘Facter’ to intelligently execute tasks only against certain sub-sets of hosts. Nagios/Munin: Automatically deploy full monitoring & metrics for hosts without ever hand configuring a file.
30. Thanks! Questions? mark.stanislav@gmail.com uncompiled.com @markstanislav