Chef: Automation for the Cloud

Introduction to Chef for automating your cloud. This was originally given at a Rackspace inteRACTIVE event.

Transcript

1. Chef: Automation for the Cloud Nathen Harvey nharvey@opscode.com @nathenharvey
2. Nathen Harvey • Technical Community Manager at Opscode • Co-host of the Food Fight Show Podcast • Meetup Organizer • @nathenharvey
3. We have a problem...
4. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity
5. Items of Manipulation (Resources) • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems
6. Application A tale of growth...
7. Application Application Database Add a database
8. Application App Databases Make database redundant
9. App Servers App Databases Application server redundancy
10. App LB App Servers App Databases Add a load balancer
11. App LBs App Servers App Databases Webscale!
12. App LBs App Servers App DB Cache App DBs Now we need a caching layer
13. App LBs App Servers App DB Cache App DBs Infrastructure has a Topology
14. Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your Infrastructure is a Snowflake
15. App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring??
16. ™
17. The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/
18. The Chef Tool(s) • ohai • chef-client • chef-shell • knife • The Ruby language Omnibus - Full Stack Native Packages
19. The Chef API • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
20. The Chef Community • Apache License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com
21. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg
22. Chef is Infrastructure as Code • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
23. Programs • Chef generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/
24. Declarative Interface to Resources • Define Policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
25. That Looks Like This package "apache2" template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end service "apache2" do action [:enable,:start] supports :reload => true end
26. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search
27. Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625
28. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So when this...
29. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite ...becomes this
30. Memcache Postgres Slaves Postgres Master Nagios Graphite Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite ...this can happen automatically
31. Nagios Graphite Nagios Graphite Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App
32. Landscape of Chef-managed Infrastructure
33. knife
34. knife - with the Chef Server • knife node • create / edit / delete • list • knife cookbook ... • knife role ... • knife environment ...
35. • SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef! $ knife bootstrap SERVER -r ‘role[webserver]’ knife bootstrap
36. $ knife rackspace knife rackspace ** RACKSPACE COMMANDS ** knife rackspace flavor list (options) knife rackspace image list (options) knife rackspace network create (options) knife rackspace network delete NETWORK_ID [NETWORK_ID] (options) knife rackspace network list (options) knife rackspace server create (options) knife rackspace server delete SERVER_ID [SERVER_ID] (options) knife rackspace server list (options)
37. $ knife rackspace flavor list Rackspace Flavors ID Name VCPUs RAM Disk 2 512MB Standard Instance 1 512 20 GB 3 1GB Standard Instance 1 1024 40 GB 4 2GB Standard Instance 2 2048 80 GB 5 4GB Standard Instance 2 4096 160 GB 6 8GB Standard Instance 4 8192 320 GB 7 15GB Standard Instance 6 15360 620 GB 8 30GB Standard Instance 8 30720 1200 GB
38. $ knife rackspace image list Rackspace Images ID Name 01b1691b-c7b9-4bc0-a655-530f51dbfdb4 Arch 2013.8 25a5f2e8-f522-4fe0-b0e0-dbaa62405c25 CentOS 6.4 b7511a0b-dd00-4246-9095-4ade53a72af4 Fedora 19 (Schrodinger's Cat) 9620c99d-9eeb-4142-b88f-7ec004901ffb OpenSUSE 12.3 e0c3a14d-b1dc-451b-ade4-d419501aa121 Red Hat Enterprise Linux 6.4 c6f9c411-e708-4952-91e5-62ded5ea4d3e Ubuntu 12.10 (Quantal Quetzal) 1bbc5e56-ca2c-40a5-94b8-aa44822c3947 Ubuntu 13.04 (Raring Ringtail) 8e5b96b6-25f7-4c10-ad31-c10afa02ed43 Windows Server 2008 R2 SP1 ...
39. Web Server Create a simple Web Server
40. $ knife rackspace server create -N web01 -f 2 -I 1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[my_app]" Create a Server Instance ID: 21329a45-1fd3-4dad-a48b-7e8aadedd9e0 Name: web01 Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
41. Web Servers Add another Web Server
42. $ knife rackspace server create -N web02 -f 2 -I 1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[my_app]" Create a Server Instance ID: 19fcf019-3126-4ebd-878b-1fe7b90dbecc Name: web02 Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
43. App LB Web Servers Add a Load Balancer
44. $ knife rackspace server create -N haproxy -f 2 -I 1bbc5e56-ca2c-40a5-94b8-aa44822c3947 -r "role[haproxy]" Create a Load Balancer Instance ID: 54daecaf-dcb8-4c8b-8b3f-d6ede4ae5689 Name: haproxy Flavor: 512MB Standard Instance Image: Ubuntu 13.04 (Raring Ringtail) Metadata: [] RackConnect Wait: no ServiceLevel Wait: no ...
45. App LB Web Servers Stitch together the configuration
46. $ knife search "run_list:role\[demo_app\]" Search with Knife 2 items found Node Name: web01 FQDN: web01 IP: 192.237.214.213 Run List: role[demo_app] Node Name: web02 Environment: _default FQDN: web02 IP: 192.237.219.98 Run List: role[demo_app]
47. pool_members = search("node", "role:#{node['haproxy']['app_server_role']}" <% @pool_members.each do |member| %> server <%= member[:hostname] %> <%= member[:ipaddress] %>:<%= node['haproxy']['member_port'] %> weight 1 <% end %> Search in Cookbooks • Search for all web servers • Add each to the Load Balancer configuration
48. App LB Web Servers Add another web server
49. $ knife rackspace server list List Servers Instance ID Name Public IP Private IP Flavor Image State 46383a09... web02 192.237.219.98 10.183.21.146 2 c6f9c41... active 795d968b... web01 192.237.214.213 10.183.21.152 2 c6f9c41... active d159c852... haproxy 192.237.222.93 10.183.25.200 2 c6f9c41... active
50. $ knife search "*:*" -a platform -a platform_version More fun with knife 3 items found web01: platform: ubuntu platform_version: 12.10 web02: platform: ubuntu platform_version: 12.10 haproxy: platform: ubuntu platform_version: 12.10
51. $ knife openstack knife openstack knife openstack flavor list (options) knife openstack group list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options)
52. $ knife openstack group list knife openstack group list Name Protocol From To CIDR Description default tcp 22 22 0.0.0.0/0 default default icmp -1 -1 0.0.0.0/0 default haproxy tcp 22002 22002 0.0.0.0/0 22022
53. $ knife openstack image list knife openstack image list ID Name Snapshot 03860dc3-f4b5-4ecf-bb13-804d6618cf15 canonical-ubuntu-10.04-amd64 no 663656ce-2fe4-4164-b842-214f221cff55 canonical-ubuntu-12.04-amd64 no ad8a6e48-ea86-4afc-8aee-f427c02eb3ce canonical-ubuntu-13.04-amd64 no 6efbafc0-fcb1-4623-9f7a-17125bba413a centos-6.2 no e0184596-577f-4eb0-9887-d70117c6b77b debian-6.0.4-amd64 no
54. $ knife openstack server list knife openstack server list Instance ID Name Public IP Private IP Flavor Image Keypair State 08f2d9f7-...-63aed8f096cc os-45... 50.56.12.229 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- 43c6bbf5-...-392d955ce5b1 os-99... 50.56.12.232 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- c1b9e3df-...-ed998b516608 os-55... 50.56.12.230 2 737969f8-...-f3cf63bd25c5 rs-demo a-...- f3edc5da-...-d957e09809e3 os-07... 50.56.12.231 2 737969f8-...-f3cf63bd25c5 rs-demo active
55. $ knife openstack server create -a -f 2 -I 737...25c5 -S rs- demo -x ubuntu -r "role[base]" knife openstack server create Instance Name: os-45539345723309377 Instance ID: 08f2d9f7-eeb0-45e7-8562-63aed8f096cc Waiting for server......... Flavor: 2 Image: 737969f8-6091-4896-ba9c-f3cf63bd25c5 SSH Identity File: /Users/mray/.ssh/rs-demo.pem SSH Keypair: rs-demo Public IP Address: 10.241.0.12 Floating IP Address: 50.56.12.229 Waiting for sshd.....done Bootstrapping Chef on 50.56.12.229 Instance Name: os-45539345723309377
56. • Instant infrastructure • Unlimited capacity • Autoscaling • No commitment • Immediate replacement Why the Cloud?
57. • Reliability • Performance • Security • Price Why not the Cloud?
58. Data Gravity
59. Know our escape plan for every infrastructure provider
60. Chef for Infrastructure Portability • knife rackspace • knife openstack • knife azure • knife cloudstack • knife ec2 • knife google • knife hp • knife vsphere • ...and many others
61. • AWS • Rackspace • HP • Google • Azure • many others ™ Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal
62. • AWS • Rackspace • HP • Google • Azure • many others Desktop, Virtualization & Cloud • Vagrant • VMware • CloudStack • Eucalyptus • OpenStack • bare metal ™
63. What does this all mean? •Every infrastructure is a unique snowflake •Understand the costs associated with the features of your platform(s) of choice. •Chef enables Infrastructure Portability •Use the same infrastructure code for wherever you deploy •"Data Gravity" is a concern ™
64. There’s a lot more... • Attributes • Environments • Roles • Chef Handlers • LWRPs • Workflow Helpers • Testing Frameworks • and even more awesome
65. Continuous Delivery
66. Thank You! • What questions do you have? • http://bit.ly/interacktive-chef - Slides • http://bit.ly/nhinteracktive - Code • nharvey@opscode.com • @nathenharvey