Intro to Chef - Ruby Hangout

A quick introduction to Chef - the automation framework. This was presented at the April 2013 Ruby Hangout. Video is available on YouTube here - http://www.youtube.com/watch?v=2yjJAStSNBY&t=8m55s

Transcript

1. Introduction to Chef Ruby Hangout Nathen Harvey nharvey@opscode.com @nathenharvey
2. Nathen • Technical Community Manager, Opscode • Co-host of Food Fight Show podcast • Co-organizer • DevOpsDC • MongoDC • Co-moderator - Ruby Hangout (usually) • Formerly - Web Operations at CustomInk
3. Chef is an automation platform for developers & systems engineers to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
4. http://www.flickr.com/photos/sbh/462754460/ Infrastructure
5. Heroku
6. When should I use an automation framework? • After you outgrow Heroku? • When you need to set-up the “third machine”
7. Evolving towards an automation framework • Just build it • Keep notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Build from template / Golden Images • Automation framework
8. See Node Application Server
9. See Nodes Application Server Application Database
10. See Nodes Grow Application Server Application Databases
11. Application Servers Application Databases See Nodes Grow
12. Application Servers Application Databases Load Balancer See Nodes Grow
13. See Nodes Grow Application Servers Application Databases Load Balancers
14. See Nodes Grow Application Servers Application Database Cache Load Balancers Application Databases
15. Tied together with Config Application Servers Application Database Cache Load Balancers Application Databases
16. Infrastructure is a Snowflake Application Servers Application Database Cache Load Balancers Floating IP? Application Databases
17. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache Database
18. Complexity Grows Quickly DC1 DC3 DC2
19. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
20. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
21. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical Infrastructure
22. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
23. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
24. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
25. • Don’t break anything! • Bob just got fired =( 5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
26. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Invalid configs! Different IP Addresses?
27. Configuration Desperation Code Sample http://www.flickr.com/photos/francoforeshock/5716969942/
28. • But you already guessed that, didn’t you? Chef Solves this Problem
29. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
30. • Chef-Client generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
31. Collections of Resources • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
32. Declarative Interface to Resources • Define policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
33. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
34. Recipes and Cookbooks • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
35. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search for nodes with Roles • Find configuration data Search
36. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
37. # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
38. Jboss App Memcache Postgres Slaves Postgres Master So when this Nagios Graphite
39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
40. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates can be automatic
41. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
42. Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
43. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything • Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
44. Code Sample Landscape of Chef-managed Infrastructure
45. knife
46. Code Sample Knife is the command-line tool used by Chefs
47. knife with the Chef Server • knife node • create/delete/edit • list • knife cookbook ... • knife role ... • knife environment ...
48. knife search • What operating systems are running? • What version of ruby is running? • How much memory do you have on each machine?
49. Ohai!
50. "memory": { "swap": { "cached": "0kB", "total": "4128760kB", "free": "4128760kB" }, "total": "2055676kB", "free": "1646524kB", "buffers": "35032kB", "cached": "210276kB", "active": "125336kB", "inactive": "142884kB", "dirty": "8kB", "writeback": "0kB", "anon_pages": "22976kB", "mapped": "8416kB", "slab": "121512kB", "slab_reclaimable": "41148kB", "slab_unreclaim": "80364kB", "page_tables": "1784kB", "nfs_unstable": "0kB", "bounce": "0kB", "commit_limit": "5156596kB", "committed_as": "74980kB", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "vmalloc_chunk": "34359449936kB" }, Ohai! "block_device": { "ram0": { "size": "32768", "removable": "0" }, "ram1": { "size": "32768", "removable": "0" }, "ram2": { "size": "32768", "removable": "0" }, "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" }, "fe80::20c:29ff:fe43:26c5": { "family": "inet6", "prefixlen": "64", "scope": "Link" } },
51. knife search
52. knife search knife search “*:*” -a platform knife search “*:*” -a languages.ruby.version knife search “*:*” -a memory.total
53. knife ssh $ knife ssh "roles:rails-web" "sudo chef-client"
54. knife bootstrap knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa • SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
55. knife ec2 $ knife ec2 Available ec2 subcommands: (for details, knife SUB-COMMAND -- help) ** EC2 COMMANDS ** knife ec2 flavor list (options) knife ec2 instance data (options) knife ec2 server create (options) knife ec2 server delete SERVER [SERVER] (options) knife ec2 server list (options) $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
56. knife openstack $ knife openstack Available openstack subcommands: (for details, knife SUB- COMMAND --help) ** OPENSTACK COMMANDS ** knife openstack flavor list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options) $ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
57. Chef for Infrastructure Portability • knife ec2 • knife rackspace • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
58. Test Driven Development!
59. Testing Tools • knife cookbook test - Verify ruby syntax • Foodcritic - Cookbook linter • Chefspec - Unit testing recipes • Fauxhai - Mock all the things • Minitest Chef Handler - post-converge tests • Why-run - Best guess
60. Moar Testing Tools • Vagrant - Local development and testing • Test Kitchen - Cross-platform testing • Cucumber Chef - acceptance & integration testing
61. Community
62. The Chef Community • Apache License, Version 2.0 • 1200+ Individual contributors • 200+ Corporate contributors • Google, HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 800+ cookbooks • http://community.opscode.com
64. Learning Chef •docs.opscode.com •learnchef.com •opscode.eventbrite.com •lists.opscode.com
65. Obligatory Self-promotion
66. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
67. Food Fight Show • The Phoenix Project • elastic search • LWRPs, Libraries, and Definitions • Zookeeper • Adaptive Fault Detection • What’s New in Chef 11 • ...and many more!
68. #ChefConf 2013 Tex
69. Yep, we’re hiring!
70. Thanks! What Questions Do You Have? Nathen Harvey nharvey@opscode.com @nathenharvey