$30 off During Our Annual Pro Sale. View Details »

What's New in Chef 11 - Philly DevOps

Nathen Harvey
February 27, 2013

What's New in Chef 11 - Philly DevOps

Introduction to Chef and new features of version 11. Given at the Philly DevOps meetup group.

Nathen Harvey

February 27, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. What’s New with Chef 11 Nathen Harvey @nathenharvey Philly DevOps

  2. Nathen Harvey • Technical Community Manager • Co-host Food Fight

    Show • http://foodfightshow.org • @nathenharvey
  3. Agenda • What is Chef? • What’s new in Chef

    11?
  4. Overview of Chef

  5. Chef is an automation platform for developers & systems engineers

    to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
  6. http://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/ Applications

  7. Infrastructure http://www.flickr.com/photos/sbh/462754460/

  8. Collection of Resources http://www.flickr.com/photos/philliecasablanca/3354734116/ • Networking • Files • Directories

    • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configuration • Other Stuff
  9. Code Sample Acting in Concert http://www.flickr.com/photos/glowjangles/4081048126/

  10. Code Sample To Provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/

  11. And it Evolves http://www.flickr.com/photos/16339684@N00/2681435235/

  12. See Node Application Server

  13. Application Server Application Database See Nodes

  14. Application Server Application Databases See Nodes Grow

  15. Application Servers Application Databases See Nodes Grow

  16. Application Servers Application Databases Load Balancer See Nodes Grow

  17. Application Servers Application Databases Load Balancers See Nodes Grow

  18. Application Servers Application Database Cache Load Balancers Application Databases See

    Nodes Grow
  19. Application Servers Application Database Cache Load Balancers Application Databases Tied

    Together with Configuration
  20. Application Servers Application Database Cache Load Balancers Floating IP? Application

    Databases Infrastructure is a Snowflake
  21. Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache

    Database Evolving Complexity
  22. DC1 DC3 DC2 Complexity Grows Quickly

  23. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/

  24. Golden Images are not the answer • Gold is heavy

    • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
  25. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical

    Infrastructure
  26. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
  27. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
  28. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
  29. • Don’t break anything! • Bob just got fired =(

    5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
  30. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Invalid configs! Different IP Addresses?
  31. Configuration Desperation Code Sample http://www.flickr.com/photos/francoforeshock/5716969942/

  32. • But you already guessed that, didn’t you? Chef Solves

    this Problem
  33. http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any

    other code base • Reconstruct business from code repository, data backup, and bare metal resources. Chef is Infrastructure as Code
  34. http://www.flickr.com/photos/ssoosay/5126146763/ • Chef generates configurations directly on nodes from their

    run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control Programs
  35. • Define Policy • Say what, not how • Pull

    not Push Code Sample http://www.flickr.com/photos/bixentro/2591838509/ Declarative Interface to Resources
  36. package "ntp" do action :install end service "ntpd" do action

    [:enable,:start] end template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “time.example.com”) notifies :restart, “service[ntpd]” end That looks like this
  37. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So

    when this
  38. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes

    this
  39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite This

    can happen automatically
  40. Nagios Graphite Jboss App Memcache Postgres Slaves • Load balancer

    config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the resources
  41. Blueprint Your Infrastructure • Compute • Application • Storage •

    Security • Network • Configuration Standards How Can Chef Help? Build Anything… • Provision compute resources in the Data Center and the Cloud • Infrastructure • Application Stacks • Big Data • HPC And Manage It Simply • Introduce continuous incremental change or total change. • Automatically reconfigure everything • Re-provision for disaster recovery • Fail-over to bare metal • Monitor for compliance • Cloud migrations become trivial Using 1,000’s of man-days of prior art! Discoverable and Searchable Infrastructure
  42. © Opscode, 2011 – Confidential – DO NOT DISTRIBUTE Chef

    Provides a Model for Reuse That Works 42 800+ Cookbooks “Yesterday we started open sourcing some of our Opscode Chef work created at bestbuy.com; bit.ly/ yDV9Hl #Splunk #opschef #expectmor”
  43. Components of Chef

  44. What’s New in Chef 11

  45. What's New Since Chef 0.10? • Windows support • Librarian

    • Spiceweasel • Foodcritic • Cookbooks in separate repositories • Private Chef • Food Fight podcast • knife plugins for openstack, hp, azure, gce • 0.10 renumbered to Chef 10 • full-stack client • why-run • output formatters • Test Kitchen • Solaris support • Chef Developer Summit(2!) • Berkshelf • ChefConf • Fauxhai • docs.opscode.com • Hangouts for reviews • nyan-cat output formatter
  46. What is NOT in Chef 11 • CouchDB • Ruby-based

    chef-server-api • depsolver, gecode, treetop • Merb • OpenID support in Web UI • Migration tooling (coming soon) http://trainweb.org/carl/Pullman150/IMG_3380.jpg
  47. What’s NEW in Chef 11 • PostgreSQL • erchef •

    nginx • Rails • bookshelf • omnibus-chef server
  48. What’s NEW in Chef 11 • chef-apply • partial search

    • users with key pairs • partials in templates • knife-essentials
  49. Erchef + SQL RDBMS

  50. nginx erchef solr (lucene) bookshelf PostgreSQL filesystem cookbook store filesystem

    search index RabbitMQ chef-expander chef-webui Chef Server
  51. erchef chef_wm chef_index chef_db chef_authn chef_objects solr bookshelf PostgreSQL RabbitMQ

    Erchef
  52. Erchef • Erlang • Concurrent, fault tolerant • Distributed systems

    • Complete Chef Server API rewrite
  53. Let's see some graphs!

  54. CPU Usage on Chef Server • Erchef + Ruby Chef

    • 12 mo old • 3 CPU aggregate
  55. CouchDB Uptime - cron'd restarts

  56. Database system CPU Usage CouchDB MySQL

  57. Database system Memory usage CouchDB MySQL

  58. Database system Load Average CouchDB MySQL Of course, we all

    know load average is not indicative of performance ;)
  59. API Average Latency

  60. Some other Erchef Features... • Nginx reverse proxy, https •

    WebUI is Ruby on Rails • Bookshelf stores cookbooks, like S3
  61. "Omnibus" Package • full-stack for the Chef server • RPMs,

    DEBs, Build your own? • opscode.com/chef/install • chef-server-ctl • reconfigure • test • start/stop http://apod.nasa.gov/apod/astropix.html
  62. chef-apply • run a single recipe file • without modifying

    the node's run_list • included with Chef, /usr/bin/chef-apply • chef-apply /path/to/recipe_file • chef-apply "content of a recipe file" • or from STDIN http://www.flickr.com/photos/albill/sets/72157628046395000/
  63. Example % cat apache.rb package "apache2" file "/var/www/index.html" do content

    "<h1>Hello, Philly DevOps!</h1>” end service "apache2" do action [:start, :enable] end % sudo chef-apply apache.rb
  64. partial search partial_search(:node,  'role:web',      :keys  =>  {  'name'

     =>  [  'name'  ],                            'ip'      =>  [  'ipaddress'  ],                            'kernel_version'  =>  [  'kernel',  'version'  ]                        } ).each  do  |result|    puts  result['name']    puts  result['ip']    puts  result['kernel_version'] end • instead of entire node, just the pieces you want • massive reduction in bandwidth and memory • (See also: http://ckbk.it/whitelist-node-attrs)
  65. users with keypairs • users can have key pairs, just

    like clients • knife actions as a user instead of a client • post a public key when you create a user/client • "knife user" (11.2.0) http://www.flickr.com/photos/albill/sets/72157628046395000/
  66. <?xml version='1.0' encoding='UTF-8'?> <server xmlns="urn:jboss:domain:1.2"> <extensions> <% if @infinispan -%>

    <extension module="org.jboss.as.clustering.infinispan"/> <% end -%> <extension module="org.jboss.as.web"/> <% if @webservices -%> <extension module="org.jboss.as.webservices"/> <% end -%> <extension module="org.jboss.as.weld"/> </extensions> <% if @infinispan include_template "infinispan.xml.erb" end -%> <% if @webservices include_template "webservices.xml.erb" end -%> <socket-binding-group name="standard-sockets" > ...... </socket-binding-group> </server> • Thanks to Andrea Campi! partials in templates http://www.flickr.com/photos/modern_fred/2095565021/
  67. knife-essentials • Treat local chef-repo and chef-server like a filesystem

    • knife download • knife diff • knife show • knife upload cookbooks/apache2 • knife list
  68. Chef 11 Breaking Changes • http://docs.opscode.com/ breaking_changes_chef_11. html • chef-shell

    (formerly shef) • no implicit node attributes (no more node['blah'] = 'foo') http://www.flickr.com/photos/modern_fred/2096352938/
  69. Chef 11 Breaking Changes • attribute files may access role

    and environment attributes • delayed notifications run after failed converge • encrypted data bag item format change • chef-client lock so safe from simultaneous runs
  70. Mini-FAQ!

  71. Can I get support for Open Source Chef Server? (from

    Opscode)
  72. Yes.

  73. Do I write recipes in Erlang now??

  74. Weird Syntax... -module(ranks). -export([fetch_title_and_rank/1]). -include_lib("xmerl/include/xmerl.hrl"). -define(BASE_URL, "http://webservices.amazon.com/onca/xml?" ++ "&ItemId="). fetch_title_and_rank(ISBN)

    -> URL = amazon_url_for(ISBN), { ok, {_Status, _Headers, Body }} = http:request(URL), { Xml, _Rest } = xmerl_scan:string(Body), [ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml), [ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml), { Title, Rank }. amazon_url_for(ISBN) -> ?BASE_URL ++ ISBN. No :) http://pragdave.pragprog.com/pragdave/2007/04/a_first_erlang_.html
  75. No migration tools? How do I upgrade?

  76. Migration tools • Coming soon • But, it's an API...

    • knife download, knife upload works for many use cases
  77. And now a word from our sponsors...

  78. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
  79. Chef Introductory Workshop • Save 10% with promo code “MEETUP”

    • http://opscode.eventbrite.com • Get your tickets soon!
  80. #ChefConf 2013 Tex OPSCODE-MEETUP - Save 15%

  81. Thanks ! • What questions do you have? • Nathen

    Harvey • nharvey@opscode.com • @nathenharvey