Keeping the Cloud Promise: Infrastructure Agility with a DevOps Toolchain

The agility of cloud computing can often be hampered by system administration techniques and infrastructure technologies that aren’t able to keep-up at an equal pace. The Development Operations (DevOps) movement helps to address many of the pace issues in cloud computing by bringing new life to team culture and implemented technologies. A mixture of pre-created and live demonstrations will be given for how a “DevOps Toolchain” can be implemented to build and maintain infrastructure, both cloud and virtualized.

A wide variety of software will address provisioning, configuration management, orchestration, monitoring, metrics, and other aspects of a fully-functional operations model. The presentation will focus on Amazon Web Services and make benefit of technologies such as Puppet, mCollective, Nagios, Munin, and many others.

Transcript

1. KEEPING THE CLOUD PROMISE Infrastructure Agility with a DevOps Toolchain Mark Stanislav mark.stanislav@gmail.com
2. Me Senior Consultant at NetWorks Group (InfoSec MSP) Previously managed cloud computing client projects for MNX Solutions (Linux MSP) Previously the cloud computing subject matter expert for ePrize (Digital Promotions) Deployed two Super Bowl promotions with AWS in 2010 Hold the Cloud Security Alliance's CCSK certification
3. Caveat Emptor This talk is not intended to tell you to use any certain product, vendor, platform, or otherwise Questions are probably best held until the end as there’s a lot to cover. I am more than happy to chat after the presentation or via e-mail for any questions! I’ll be mixing Virtualization in with ‘Cloud Computing’ to make my life a bit easier/cheaper to show the tools I can’t promise these technologies will scale for your environment or platform, but I’ve had solid success!
4. DevOps Definition? Plenty of discussion about the correct definition of DevOps online (read: Holy War) What do I think? I really like this one... “DevOps is all about trying to avoid that epic failure and working smarter and more efficiently at the same time. It is a framework of ideas and principles designed to foster cooperation, learning and coordination between development and operational groups.” - James Turnbull, Puppet Labs http://www.kartar.net/2010/02/what-devops-means-to-me/
5. What’s a DevOps Tool? Generally applied to software which is utilized in environments heavily focused on DevOps There’s no DevOps tool “certification process” In my mind, a DevOps tool should: Make a process more efficient and consistent Help to enable an environment to engage DevOps methodologies rather than distract from them Allow for a single technology need to be handled well
6. So Cloud and DevOps and DevOps Tools... Okay? Cloud Computing (in its truest definitions) enables a break-neck speed at deploying resources (*aaS) If our environments and technologies are too slow to keep up, then it is our fault if cloud “doesn’t do what they said it would” A DevOps toolchain allows us to start catching up with the fire-hose of awesome that is Cloud Computing Let’s take on the challenge, utilize all of this crazy technology we are handed, and do amazing things!
7. The Toolchain Metadata: Facter Provisioning: Puppet Dashboard, Vagrant + Veewee Configuration Management: Puppet Virtualization: Virtual Box Orchestration: mCollective Monitoring: Nagios Metrics: Munin
8. Facter Gathers information (meta-data) about nodes in order to understand what the system is beyond “a server” Each meta-data is consider a ‘fact’ and can range from hardware (e.g. architecture) to software (e.g. kernel) Facter allows for custom facts to provide unlimited usage possibilities in defining what a host is The Amazon AMI (~CentOS 6) has 113 facts by default Facter helps allow Puppet, mCollective, Nagios, and Munin in acting intelligently in scoping as well as function
9. Facter in Action Install Platform Specific Puppet Modules Install a certain 64-bit RPM when the system is x86_64 and the 32-bit RPM for i386 Virtualization Scoped mCollective Commands Execute a command/plugin only on systems that are identified as running VirtualBox Deploy Monitoring Based on Distribution Nagios and Munin intelligently deploy for RHEL v. Debian v. Windows systems
10. Puppet Configuration Management - “Desired State” Build modules which deploy/manage services, users, groups, SELinux policy, files, cron, and MUCH more Template-based configuration files can take advantage of Facter (e.g. create an Apache dynamically w/ FQDN) Build configuration and run it on many environments and platforms, utilizing Facter metadata to decide how each environment needs to be specifically deployed Build ‘Infrastructure as Code’ ; now you can version, audit, test, and fork your infrastructure (pull requests!)
11. Puppet Dashboard View reports of node synchronization, e.g. what actions were taken to bring a node ‘into compliance’ with modules Search for nodes based on Facter facts Inspect user, group, host, and package resources of nodes and compare/clone configuration of other nodes Execute system actions such as installing a package, forcing Puppet to synchronize, restarting a service, and more! Add classes (modules) to a given node and group them together for easier compliance management
14. mCollective Plugin-based framework that allows for consistent, parallel, secure commands/output to be generated across systems Intelligently act upon nodes based on Facter data, Puppet classes assigned, agent, or identity information Utilizes a queuing system (RabbitMQ/ActiveMQ) Supports PSK/SSL/AES plugins for privacy/authentication More than an ‘ssh for-loop’ replacement; provides reliability and management to tasks which should have such aspects!
15. mCollective in Action Check Apache’s status on systems running SELinux Install a package ad-hoc on all RHEL-based systems Ping only systems which run Puppet 2.7.18 Delete a file only on systems in us-east-1 of AWS Force Puppet to run on all x86_64 nodes Query process information for all Xen servers
17. Virtualization Vagrant is a Ruby-based tool which provides consistency and reduced effort to deploy virtual machines (using in our case, Virtual Box) Simplified port forwarding, seamless SSH authentication, configuration management supported (e.g. Puppet/Chef natively), packages a ‘box’ file to distribute to others Veewee is an extension to Vagrant that provides template- based, automated virtual machine creation “I want a CentOS 6.2 64-bit server with [...] packages installed, 15GB of HDD space, 4GB of RAM” -- but I don’t want to do any of it myself! :)
18. Using Vagrant/Veewee Copy Existing Template For Editing vagrant basebox define MyVM CentOS-6.2-x86_64-minimal Build Vagrant Box From Template vagrant basebox build 'MyVM' Export Box For Sharing vagrant basebox export MyVM Add a Box to Be Used In Vagrant vagrant box add 'MyVM' 'MyVM.box'
19. Building a base box (creating a VM, bootstrapping)
20. Installation of base packages, VirtualBox Additions
21. Export VM into ‘.box’ file, add to Vagrant
22. Using Vagrant/Veewee Initializing a Vagrant Box For Deployment vagrant init 'CentOS-VM' Boot VM w/ Port Forwarding, Disk Shares, Puppet vagrant up Execute Puppet (or Chef/CFEngine) Modules vagrant provision Destroy Virtual Machine vagrant destroy
23. Munin Graph metrics for system functions, network interfaces, and applications with an easy-to-use web interface Create thresholds for basic alerting/monitoring Add plugins as-needed to provide graphs for any number of “I wish we knew...” situations With Puppet, automatically deploy plugins and configuration for new systems based on Facter metadata Never again forget to add certain metrics until its too late
25. Nagios Long-standing monitoring platform with a variety of open- source plugins to monitor just about everything Various alert methods, escalations, grouping, time periods, and most of the other things you need to monitor systems Problem: Configuration can often be a headache to maintain past a few dozen hosts and it’s easy to forget to enable certain monitoring for a given host -- what if you add a service but don’t add monitoring? With Puppet, again, never forget to add monitoring for a new service or think you had properly deployed an alert
28. Our Environment Puppet Master Test Instance Amazon EC2 Instances My Laptop Puppet: 8140/TCP mCollective: 61613/TCP Nagios: 5666/TCP Munin: 4949/TCP Puppet: 8140/TCP mCollective: 61613/TCP Nagios: 5666/TCP Munin: 4949/TCP Nagios/Munin Virtual Box Virtual Box
29. DEMOS 1) Provision an Amazon EC2 instance using Puppet’s cloud provisioner (auto install puppet, sign certificate) 2) Show Nagios has deployed monitoring 1) Add syslogd monitoring in via Puppet Dashboard 3) Show Munin has deployed metrics 4) Install a new Puppet module via ‘Forge’ and show function 5) Install a package via mCollective through Puppet Dashboard 6) Show various default mCollective plugins 7) Show general Puppet Dashboard functionality -- Resources, Inventory, Reporting, Classes, Groups
30. Alternative Technologies Provisioning: Foreman Configuration Management: Chef, CFEngine, Bcfg2 Virtualization: Xen, KVM, VMWare Metadata: Ohai Orchestration: Run Deck, Salt Monitoring: Zenoss, Xymon, Metrics: Ganglia, Collectd, Statsd
31. The Real Challenge Bringing your team together in a way that you can accomplish the overhaul likely needed to actually take advantage of these technologies and cloud computing Getting management buy-in... Automated monitoring and metrics Quicker and more consistent deployments Ability to deploy/manage ALL of your environments More time to spent on areas that add value!
32. Additional Slide Decks Being a Puppet Master: Automating EC2 with Puppet & Friends http://www.slideshare.net/markstanislav/being-a-puppet- master-automating-amazon-ec2-with-puppet-friends Development Operations: Take Back Your Infrastructure http://www.slideshare.net/markstanislav/development- operations-take-back-your-infrastructure
33. Helpful Links http://docs.puppetlabs.com/pe/2.5/index.html http://docs.puppetlabs.com/mcollective/ http://www.youtube.com/user/PuppetLabsInc/videos?view=1 http://vagrantup.com/v1/docs/index.html https://github.com/jedi4ever/veewee/ http://aws.amazon.com/documentation/ http://nagios.sourceforge.net/docs/nagioscore/3/en/ http://munin-monitoring.org/wiki/Documentation
34. Thanks! Questions? mark.stanislav@gmail.com @markstanislav http://www.uncompiled.com