$30 off During Our Annual Pro Sale. View Details »

Intro to Chef - Charm City Linux

Intro to Chef - Charm City Linux

A brief introduction to Chef presented at the Charm City Linux group.

Nathen Harvey

May 13, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. None
  2. Introduction to Chef Charm City Linux Nathen Harvey nharvey@opscode.com @nathenharvey

  3. Nathen • Technical Community Manager, Opscode • Co-host of Food

    Fight Show podcast • Co-organizer • DevOpsDC • MongoDC • Co-moderator - Ruby Hangout (usually) • Formerly - Web Operations at CustomInk
  4. Chef is an automation platform for developers & systems engineers

    to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
  5. http://www.flickr.com/photos/sbh/462754460/ Infrastructure

  6. Evolving towards an automation framework • Just build it •

    Keep notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Build from template / Golden Images • Automation framework
  7. See Node Application Server

  8. See Nodes Application Server Application Database

  9. See Nodes Grow Application Server Application Databases

  10. Application Servers Application Databases See Nodes Grow

  11. Application Servers Application Databases Load Balancer See Nodes Grow

  12. See Nodes Grow Application Servers Application Databases Load Balancers

  13. See Nodes Grow Application Servers Application Database Cache Load Balancers

    Application Databases
  14. Tied together with Config Application Servers Application Database Cache Load

    Balancers Application Databases
  15. Infrastructure is a Snowflake Application Servers Application Database Cache Load

    Balancers Floating IP? Application Databases
  16. Evolving Complexity Load Balancers Application Servers NoSQL Database Slaves ApplicationCache

    Database Cache Database
  17. Complexity Grows Quickly DC1 DC3 DC2

  18. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/

  19. Golden Images are not the answer • Gold is heavy

    • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
  20. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical

    Infrastructure
  21. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
  22. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
  23. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
  24. • Don’t break anything! • Bob just got fired =(

    5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
  25. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Invalid configs! Different IP Addresses?
  26. Configuration Desperation http://www.flickr.com/photos/francoforeshock/5716969942/

  27. • But you already guessed that, didn’t you? Chef Solves

    this Problem
  28. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and

    configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
  29. • Chef-Client generates configurations directly on nodes from their run

    list • Reduce management complexity through abstraction • Store the configuration of your programs in version control http://www.flickr.com/photos/ssoosay/5126146763/ Nodes
  30. Collections of Resources • Networking • Files • Directories •

    Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Other Stuff http://www.flickr.com/photos/stevekeys/3123167585/
  31. Declarative Interface to Resources • Define policy • Say what,

    not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
  32. Ruby! extra_packages = case node['platform'] when "ubuntu","debian" %w{ ruby1.8 ruby1.8-dev

    rdoc1.8 ri1.8 libopenssl-ruby } end extra_packages.each do |pkg| package pkg do action :install end end
  33. Recipes and Cookbooks • Recipes are collections of Resources •

    Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/
  34. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search

    for nodes with Roles • Find configuration data Search
  35. pool_members = search("node","role:webserver”) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root"

    group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results to Templates
  36. # Set up application listeners here. listen application 0.0.0.0:80 balance

    roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results to Templates
  37. Jboss App Memcache Postgres Slaves Postgres Master So when this

    Nagios Graphite
  38. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes

    this
  39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Updates

    can be automatic
  40. Nagios Graphite Count the resources Jboss App Memcache Postgres Slaves

    • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
  41. Build anything • Simple internal applications • Complex external applications

    • Workstations • Hadoop clusters • IaaS infrastructure • PaaS infrastructure • SaaS applications • Storage systems • You name it http://www.flickr.com/photos/hyku/245010680/
  42. And manage it simply http://www.flickr.com/photos/helico/404640681/ • Automatically reconfigure everything •

    Linux, Windows, Unixes, BSDs • Load balancers • Metrics collection systems • Monitoring systems • Cloud migrations become trivial
  43. Code Sample Landscape of Chef-managed Infrastructure

  44. knife

  45. Knife is the command-line tool used by Chefs

  46. knife with the Chef Server • knife node • create/delete/edit

    • list • knife cookbook ... • knife role ... • knife environment ...
  47. knife search • What operating systems are running? • What

    version of ruby is running? • How much memory do you have on each machine?
  48. Ohai!

  49. "memory": { "swap": { "cached": "0kB", "total": "4128760kB", "free": "4128760kB"

    }, "total": "2055676kB", "free": "1646524kB", "buffers": "35032kB", "cached": "210276kB", "active": "125336kB", "inactive": "142884kB", "dirty": "8kB", "writeback": "0kB", "anon_pages": "22976kB", "mapped": "8416kB", "slab": "121512kB", "slab_reclaimable": "41148kB", "slab_unreclaim": "80364kB", "page_tables": "1784kB", "nfs_unstable": "0kB", "bounce": "0kB", "commit_limit": "5156596kB", "committed_as": "74980kB", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "vmalloc_chunk": "34359449936kB" }, Ohai! "block_device": { "ram0": { "size": "32768", "removable": "0" }, "ram1": { "size": "32768", "removable": "0" }, "ram2": { "size": "32768", "removable": "0" }, "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" }, "fe80::20c:29ff:fe43:26c5": { "family": "inet6", "prefixlen": "64", "scope": "Link" } },
  50. knife search

  51. knife search • knife search “*:*” -a platform • knife

    search “*:*” -a languages.ruby.version • knife search “*:*” -a memory.total
  52. knife ssh $ knife ssh "roles:rails-web" "sudo chef-client"

  53. knife bootstrap • knife bootstrap SERVER -r 'role[webserver]' -i ~/.ssh/

    id_rsa • SSH to the machine given existing credentials • Install the Chef Client • Register with the Chef Server • Run the initial Run List • Now managed with Chef!
  54. knife ec2 • $ knife ec2 • Available ec2 subcommands:

    (for details, knife SUB-COMMAND --help) • ** EC2 COMMANDS ** • knife ec2 flavor list (options) • knife ec2 instance data (options) • knife ec2 server create (options) • knife ec2 server delete SERVER [SERVER] (options) • knife ec2 server list (options) • $ knife ec2 server create -S keypair -i ~/.ssh/id_rsa -x ubuntu -I ami-4721882e -f m1.small -r 'role[webserver]'
  55. knife openstack • $ knife openstack • Available openstack subcommands:

    (for details, knife SUB- COMMAND --help) • ** OPENSTACK COMMANDS ** • knife openstack flavor list (options) • knife openstack image list (options) • knife openstack server create (options) • knife openstack server delete SERVER [SERVER] (options) • knife openstack server list (options) • $ knife openstack server create -S keypair -i ~/.ssh/id_rsa • -x ubuntu -I 1231 -f standard.small -r 'role[webserver]'
  56. Chef for Infrastructure Portability • knife ec2 • knife rackspace

    • knife hp • knife google • knife azure • knife cloudstack • knife openstack • knife vsphere • ... and many others
  57. Test Driven Development!

  58. Testing Tools • knife cookbook test - Verify ruby syntax

    • Foodcritic - Cookbook linter • Chefspec - Unit testing recipes • Fauxhai - Mock all the things • Minitest Chef Handler - post-converge tests • Why-run - Best guess
  59. Moar Testing Tools • Vagrant - Local development and testing

    • Test Kitchen - Cross-platform testing • Cucumber Chef - acceptance & integration testing
  60. Community

  61. The Chef Community • Apache License, Version 2.0 • 1200+

    Individual contributors • 200+ Corporate contributors • Google, HP, Dell, Rackspace, VMware, Joyent, Calxeda, Heroku, SUSE and many more • 800+ cookbooks • http://community.opscode.com
  62. None
  63. Learning Chef • docs.opscode.com • learnchef.com • opscode.eventbrite.com • lists.opscode.com

  64. Obligatory Self-promotion

  65. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife- plugins, and more • Best Practices for working with Chef
  66. Food Fight Show • The Phoenix Project • elastic search

    • LWRPs, Libraries, and Definitions • Zookeeper • Adaptive Fault Detection • What’s New in Chef 11 • ...and many more!
  67. Yep, we’re hiring!

  68. Thanks! What Questions Do You Have? Nathen Harvey nharvey@opscode.com @nathenharvey