$30 off During Our Annual Pro Sale. View Details »

Intro to Chef - Ruby Hangout

Intro to Chef - Ruby Hangout

A quick introduction to Chef - the automation framework. This was presented at the April 2013 Ruby Hangout. Video is available on YouTube here - http://www.youtube.com/watch?v=2yjJAStSNBY&t=8m55s

Nathen Harvey

April 04, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Introduction to Chef Ruby Hangout Nathen Harvey nharvey@opscode.com @nathenharvey

  2. Nathen • Technical Community Manager, Opscode • Co-host of Food

    Fight Show podcast • Co-organizer • DevOpsDC • MongoDC • Co-moderator - Ruby Hangout (usually) • Formerly - Web Operations at CustomInk
  3. Chef is an automation platform for developers & systems engineers

    to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
  4. http://www.flickr.com/photos/sbh/462754460/ Infrastructure

  5. Heroku

  6. When should I use an automation framework? • After you

    outgrow Heroku? • When you need to set-up the “third machine”
  7. Evolving towards an automation framework • Just build it •

    Keep notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Build from template / Golden Images • Automation framework
  8. See Node Application Server

  9. See Nodes Application Server Application Database

  10. See Nodes Grow Application Server Application Databases

  11. Application Servers Application Databases See Nodes Grow

  12. Application Servers Application Databases Load Balancer See Nodes Grow

  13. See Nodes Grow Application Servers Application Databases Load Balancers

  14. See Nodes Grow Application Servers Application Database Cache Load Balancers

    Application Databases
  15. Tied together with Config Application Servers Application Database Cache Load

    Balancers Application Databases
  16. Infrastructure is a Snowflake Application Servers Application Database Cache Load

    Balancers Floating IP? Application Databases
  17. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache

    Database Cache Database
  18. Complexity Grows Quickly DC1 DC3 DC2

  19. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/

  20. Golden Images are not the answer • Gold is heavy

    • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
  21. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical

    Infrastructure
  22. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
  23. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
  24. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
  25. • Don’t break anything! • Bob just got fired =(

    5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
  26. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Invalid configs! Different IP Addresses?
  27. Configuration Desperation Code Sample http://www.flickr.com/photos/francoforeshock/5716969942/

  28. • But you already guessed that, didn’t you? Chef Solves

    this Problem
  29. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and

    configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
  30. • Chef-Client generates configurations directly on nodes from their run

    list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
  31. Collections of Resources • Networking • Files • Directories •

    Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
  32. Declarative Interface to Resources • Define policy • Say what,

    not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
  33. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev

    rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
  34. Recipes and Cookbooks • Recipes are collections of Resources •

    Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
  35. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search

    for nodes with Roles • Find configuration data Search
  36. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root"

    group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
  37. # Set up application listeners here. listen application 0.0.0.0:80 balance

    roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
  38. Jboss App Memcache Postgres Slaves Postgres Master So when this

    Nagios Graphite
  39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes

    this
  40. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates

    can be automatic
  41. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves

    • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
  42. Build anything • Simple internal applications • Complex external applications

    • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
  43. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything •

    Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
  44. Code Sample Landscape of Chef-managed Infrastructure

  45. knife

  46. Code Sample Knife is the command-line tool used by Chefs

  47. knife with the Chef Server • knife node • create/delete/edit

    • list • knife cookbook ... • knife role ... • knife environment ...
  48. knife search • What operating systems are running? • What

    version of ruby is running? • How much memory do you have on each machine?
  49. Ohai!

  50. "memory": { "swap": { "cached": "0kB", "total": "4128760kB", "free": "4128760kB"

    }, "total": "2055676kB", "free": "1646524kB", "buffers": "35032kB", "cached": "210276kB", "active": "125336kB", "inactive": "142884kB", "dirty": "8kB", "writeback": "0kB", "anon_pages": "22976kB", "mapped": "8416kB", "slab": "121512kB", "slab_reclaimable": "41148kB", "slab_unreclaim": "80364kB", "page_tables": "1784kB", "nfs_unstable": "0kB", "bounce": "0kB", "commit_limit": "5156596kB", "committed_as": "74980kB", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "vmalloc_chunk": "34359449936kB" }, Ohai! "block_device": { "ram0": { "size": "32768", "removable": "0" }, "ram1": { "size": "32768", "removable": "0" }, "ram2": { "size": "32768", "removable": "0" }, "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" }, "fe80::20c:29ff:fe43:26c5": { "family": "inet6", "prefixlen": "64", "scope": "Link" } },
  51. knife search

  52. knife search knife search “*:*” -a platform knife search “*:*”

    -a languages.ruby.version knife search “*:*” -a memory.total
  53. knife ssh $ knife ssh "roles:rails-web" "sudo chef-client"

  54. knife bootstrap knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/id_rsa •

    SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
  55. knife ec2 $ knife ec2 Available ec2 subcommands: (for details,

    knife SUB-COMMAND -- help) ** EC2 COMMANDS ** knife ec2 flavor list (options) knife ec2 instance data (options) knife ec2 server create (options) knife ec2 server delete SERVER [SERVER] (options) knife ec2 server list (options) $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
  56. knife openstack $ knife openstack Available openstack subcommands: (for details,

    knife SUB- COMMAND --help) ** OPENSTACK COMMANDS ** knife openstack flavor list (options) knife openstack image list (options) knife openstack server create (options) knife openstack server delete SERVER [SERVER] (options) knife openstack server list (options) $ knife openstack server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
  57. Chef for Infrastructure Portability • knife ec2 • knife rackspace

    • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
  58. Test Driven Development!

  59. Testing Tools • knife cookbook test - Verify ruby syntax

    • Foodcritic - Cookbook linter • Chefspec - Unit testing recipes • Fauxhai - Mock all the things • Minitest Chef Handler - post-converge tests • Why-run - Best guess
  60. Moar Testing Tools • Vagrant - Local development and testing

    • Test Kitchen - Cross-platform testing • Cucumber Chef - acceptance & integration testing
  61. Community

  62. The Chef Community • Apache License, Version 2.0 • 1200+

    Individual contributors • 200+ Corporate contributors • Google, HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 800+ cookbooks • http://community.opscode.com
  63. None
  64. Learning Chef •docs.opscode.com •learnchef.com •opscode.eventbrite.com •lists.opscode.com

  65. Obligatory Self-promotion

  66. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
  67. Food Fight Show • The Phoenix Project • elastic search

    • LWRPs, Libraries, and Definitions • Zookeeper • Adaptive Fault Detection • What’s New in Chef 11 • ...and many more!
  68. #ChefConf 2013 Tex

  69. Yep, we’re hiring!

  70. Thanks! What Questions Do You Have? Nathen Harvey nharvey@opscode.com @nathenharvey