Intro to Chef - Charm City Linux

A brief introduction to Chef presented at the Charm City Linux group.

Transcript

2. Introduction to Chef Charm City Linux Nathen Harvey nharvey@opscode.com @nathenharvey
3. Nathen • Technical Community Manager, Opscode • Co-host of Food Fight Show podcast • Co-organizer • DevOpsDC • MongoDC • Co-moderator - Ruby Hangout (usually) • Formerly - Web Operations at CustomInk
4. Chef is an automation platform for developers & systems engineers to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
5. http://www.flickr.com/photos/sbh/462754460/ Infrastructure
6. Evolving towards an automation framework • Just build it • Keep notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Build from template / Golden Images • Automation framework
7. See Node Application Server
8. See Nodes Application Server Application Database
9. See Nodes Grow Application Server Application Databases
10. Application Servers Application Databases See Nodes Grow
11. Application Servers Application Databases Load Balancer See Nodes Grow
12. See Nodes Grow Application Servers Application Databases Load Balancers
13. See Nodes Grow Application Servers Application Database Cache Load Balancers Application Databases
14. Tied together with Config Application Servers Application Database Cache Load Balancers Application Databases
15. Infrastructure is a Snowflake Application Servers Application Database Cache Load Balancers Floating IP? Application Databases
16. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache Database
17. Complexity Grows Quickly DC1 DC3 DC2
18. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
19. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
20. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical Infrastructure
21. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
22. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
23. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
24. • Don’t break anything! • Bob just got fired =( 5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
25. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Invalid configs! Different IP Addresses?
26. Configuration Desperation http://www.flickr.com/photos/francoforeshock/5716969942/
27. • But you already guessed that, didn’t you? Chef Solves this Problem
28. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
29. • Chef-Client generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
30. Collections of Resources • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
31. Declarative Interface to Resources • Define policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
32. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
33. Recipes and Cookbooks • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
34. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search for nodes with Roles • Find configuration data Search
35. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
36. # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
37. Jboss App Memcache Postgres Slaves Postgres Master So when this Nagios Graphite
38. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates can be automatic
40. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
41. Build anything • Simple internal applications • Complex external applications • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
42. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything • Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
43. Code Sample Landscape of Chef-managed Infrastructure
44. knife
45. Knife is the command-line tool used by Chefs
46. knife with the Chef Server • knife node • create/delete/edit • list • knife cookbook ... • knife role ... • knife environment ...
47. knife search • What operating systems are running? • What version of ruby is running? • How much memory do you have on each machine?
48. Ohai!
49. "memory": { "swap": { "cached": "0kB", "total": "4128760kB", "free": "4128760kB" }, "total": "2055676kB", "free": "1646524kB", "buffers": "35032kB", "cached": "210276kB", "active": "125336kB", "inactive": "142884kB", "dirty": "8kB", "writeback": "0kB", "anon_pages": "22976kB", "mapped": "8416kB", "slab": "121512kB", "slab_reclaimable": "41148kB", "slab_unreclaim": "80364kB", "page_tables": "1784kB", "nfs_unstable": "0kB", "bounce": "0kB", "commit_limit": "5156596kB", "committed_as": "74980kB", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "vmalloc_chunk": "34359449936kB" }, Ohai! "block_device": { "ram0": { "size": "32768", "removable": "0" }, "ram1": { "size": "32768", "removable": "0" }, "ram2": { "size": "32768", "removable": "0" }, "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" }, "fe80::20c:29ff:fe43:26c5": { "family": "inet6", "prefixlen": "64", "scope": "Link" } },
50. knife search
51. knife search • knife search “*:*” -a platform • knife search “*:*” -a languages.ruby.version • knife search “*:*” -a memory.total
52. knife ssh $ knife ssh "roles:rails-web" "sudo chef-client"
53. knife bootstrap • knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/ id_rsa • SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
54. knife ec2 • $ knife ec2 • Available ec2 subcommands: (for details, knife SUB-COMMAND --help) • ** EC2 COMMANDS ** • knife ec2 flavor list (options) • knife ec2 instance data (options) • knife ec2 server create (options) • knife ec2 server delete SERVER [SERVER] (options) • knife ec2 server list (options) • $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
55. knife openstack • $ knife openstack • Available openstack subcommands: (for details, knife SUB- COMMAND --help) • ** OPENSTACK COMMANDS ** • knife openstack flavor list (options) • knife openstack image list (options) • knife openstack server create (options) • knife openstack server delete SERVER [SERVER] (options) • knife openstack server list (options) • $ knife openstack server create -S keypair -i ~/.ssh/id_rsa • -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
56. Chef for Infrastructure Portability • knife ec2 • knife rackspace • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
57. Test Driven Development!
58. Testing Tools • knife cookbook test - Verify ruby syntax • Foodcritic - Cookbook linter • Chefspec - Unit testing recipes • Fauxhai - Mock all the things • Minitest Chef Handler - post-converge tests • Why-run - Best guess
59. Moar Testing Tools • Vagrant - Local development and testing • Test Kitchen - Cross-platform testing • Cucumber Chef - acceptance & integration testing
60. Community
61. The Chef Community • Apache License, Version 2.0 • 1200+ Individual contributors • 200+ Corporate contributors • Google, HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 800+ cookbooks • http://community.opscode.com
63. Learning Chef • docs.opscode.com • learnchef.com • opscode.eventbrite.com • lists.opscode.com
64. Obligatory Self-promotion
65. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps Chef Do Battle • Regular updates about new Cookbooks, Knife- plugins, and more • Best Practices for working with Chef
66. Food Fight Show • The Phoenix Project • elastic search • LWRPs, Libraries, and Definitions • Zookeeper • Adaptive Fault Detection • What’s New in Chef 11 • ...and many more!
67. Yep, we’re hiring!
68. Thanks! What Questions Do You Have? Nathen Harvey nharvey@opscode.com @nathenharvey