What's New in Chef 11 - Philly DevOps

Introduction to Chef and new features of version 11. Given at the Philly DevOps meetup group.

Transcript

1. What’s New with Chef 11 Nathen Harvey @nathenharvey Philly DevOps
2. Nathen Harvey • Technical Community Manager • Co-host Food Fight Show • http://foodfightshow.org • @nathenharvey
3. Agenda • What is Chef? • What’s new in Chef 11?
4. Overview of Chef
5. Chef is an automation platform for developers & systems engineers to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
6. http://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/ Applications
7. Infrastructure http://www.flickr.com/photos/sbh/462754460/
8. Collection of Resources http://www.flickr.com/photos/philliecasablanca/3354734116/ • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configuration • Other Stuff
9. Code Sample Acting in Concert http://www.flickr.com/photos/glowjangles/4081048126/
10. Code Sample To Provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/
11. And it Evolves http://www.flickr.com/photos/16339684@N00/2681435235/
12. See Node Application Server
13. Application Server Application Database See Nodes
14. Application Server Application Databases See Nodes Grow
15. Application Servers Application Databases See Nodes Grow
16. Application Servers Application Databases Load Balancer See Nodes Grow
17. Application Servers Application Databases Load Balancers See Nodes Grow
18. Application Servers Application Database Cache Load Balancers Application Databases See Nodes Grow
19. Application Servers Application Database Cache Load Balancers Application Databases Tied Together with Configuration
20. Application Servers Application Database Cache Load Balancers Floating IP? Application Databases Infrastructure is a Snowflake
21. Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache Database Evolving Complexity
22. DC1 DC3 DC2 Complexity Grows Quickly
23. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
24. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
25. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical Infrastructure
26. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
27. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
28. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
29. • Don’t break anything! • Bob just got fired =( 5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
30. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite • Invalid configs! Different IP Addresses?
31. Configuration Desperation Code Sample http://www.flickr.com/photos/francoforeshock/5716969942/
32. • But you already guessed that, didn’t you? Chef Solves this Problem
33. http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. Chef is Infrastructure as Code
34. http://www.flickr.com/photos/ssoosay/5126146763/ • Chef generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control Programs
35. • Define Policy • Say what, not how • Pull not Push Code Sample http://www.flickr.com/photos/bixentro/2591838509/ Declarative Interface to Resources
36. package "ntp" do action :install end service "ntpd" do action [:enable,:start] end template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “time.example.com”) notifies :restart, “service[ntpd]” end That looks like this
37. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So when this
38. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
39. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite This can happen automatically
40. Nagios Graphite Jboss App Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the resources
41. Blueprint Your Infrastructure • Compute • Application • Storage • Security • Network • Configuration Standards How Can Chef Help? Build Anything… • Provision compute resources in the Data Center and the Cloud • Infrastructure • Application Stacks • Big Data • HPC And Manage It Simply • Introduce continuous incremental change or total change. • Automatically reconfigure everything • Re-provision for disaster recovery • Fail-over to bare metal • Monitor for compliance • Cloud migrations become trivial Using 1,000’s of man-days of prior art! Discoverable and Searchable Infrastructure
42. © Opscode, 2011 – Confidential – DO NOT DISTRIBUTE Chef Provides a Model for Reuse That Works 42 800+ Cookbooks “Yesterday we started open sourcing some of our Opscode Chef work created at bestbuy.com; bit.ly/ yDV9Hl #Splunk #opschef #expectmor”
43. Components of Chef
44. What’s New in Chef 11
45. What's New Since Chef 0.10? • Windows support • Librarian • Spiceweasel • Foodcritic • Cookbooks in separate repositories • Private Chef • Food Fight podcast • knife plugins for openstack, hp, azure, gce • 0.10 renumbered to Chef 10 • full-stack client • why-run • output formatters • Test Kitchen • Solaris support • Chef Developer Summit(2!) • Berkshelf • ChefConf • Fauxhai • docs.opscode.com • Hangouts for reviews • nyan-cat output formatter
46. What is NOT in Chef 11 • CouchDB • Ruby-based chef-server-api • depsolver, gecode, treetop • Merb • OpenID support in Web UI • Migration tooling (coming soon) http://trainweb.org/carl/Pullman150/IMG_3380.jpg
47. What’s NEW in Chef 11 • PostgreSQL • erchef • nginx • Rails • bookshelf • omnibus-chef server
48. What’s NEW in Chef 11 • chef-apply • partial search • users with key pairs • partials in templates • knife-essentials
49. Erchef + SQL RDBMS
50. nginx erchef solr (lucene) bookshelf PostgreSQL filesystem cookbook store filesystem search index RabbitMQ chef-expander chef-webui Chef Server
51. erchef chef_wm chef_index chef_db chef_authn chef_objects solr bookshelf PostgreSQL RabbitMQ Erchef
52. Erchef • Erlang • Concurrent, fault tolerant • Distributed systems • Complete Chef Server API rewrite
53. Let's see some graphs!
54. CPU Usage on Chef Server • Erchef + Ruby Chef • 12 mo old • 3 CPU aggregate
55. CouchDB Uptime - cron'd restarts
56. Database system CPU Usage CouchDB MySQL
57. Database system Memory usage CouchDB MySQL
58. Database system Load Average CouchDB MySQL Of course, we all know load average is not indicative of performance ;)
59. API Average Latency
60. Some other Erchef Features... • Nginx reverse proxy, https • WebUI is Ruby on Rails • Bookshelf stores cookbooks, like S3
61. "Omnibus" Package • full-stack for the Chef server • RPMs, DEBs, Build your own? • opscode.com/chef/install • chef-server-ctl • reconfigure • test • start/stop http://apod.nasa.gov/apod/astropix.html
62. chef-apply • run a single recipe file • without modifying the node's run_list • included with Chef, /usr/bin/chef-apply • chef-apply /path/to/recipe_file • chef-apply "content of a recipe file" • or from STDIN http://www.flickr.com/photos/albill/sets/72157628046395000/
63. Example % cat apache.rb package "apache2" file "/var/www/index.html" do content "<h1>Hello, Philly DevOps!</h1>” end service "apache2" do action [:start, :enable] end % sudo chef-apply apache.rb
64. partial search partial_search(:node,  'role:web',      :keys  =>  {  'name'  =>  [  'name'  ],                            'ip'      =>  [  'ipaddress'  ],                            'kernel_version'  =>  [  'kernel',  'version'  ]                        } ).each  do  |result|    puts  result['name']    puts  result['ip']    puts  result['kernel_version'] end • instead of entire node, just the pieces you want • massive reduction in bandwidth and memory • (See also: http://ckbk.it/whitelist-node-attrs)
65. users with keypairs • users can have key pairs, just like clients • knife actions as a user instead of a client • post a public key when you create a user/client • "knife user" (11.2.0) http://www.flickr.com/photos/albill/sets/72157628046395000/
66. <?xml version='1.0' encoding='UTF-8'?> <server xmlns="urn:jboss:domain:1.2"> <extensions> <% if @infinispan -%> <extension module="org.jboss.as.clustering.infinispan"/> <% end -%> <extension module="org.jboss.as.web"/> <% if @webservices -%> <extension module="org.jboss.as.webservices"/> <% end -%> <extension module="org.jboss.as.weld"/> </extensions> <% if @infinispan include_template "infinispan.xml.erb" end -%> <% if @webservices include_template "webservices.xml.erb" end -%> <socket-binding-group name="standard-sockets" > ...... </socket-binding-group> </server> • Thanks to Andrea Campi! partials in templates http://www.flickr.com/photos/modern_fred/2095565021/
67. knife-essentials • Treat local chef-repo and chef-server like a filesystem • knife download • knife diff • knife show • knife upload cookbooks/apache2 • knife list
68. Chef 11 Breaking Changes • http://docs.opscode.com/ breaking_changes_chef_11. html • chef-shell (formerly shef) • no implicit node attributes (no more node['blah'] = 'foo') http://www.flickr.com/photos/modern_fred/2096352938/
69. Chef 11 Breaking Changes • attribute files may access role and environment attributes • delayed notifications run after failed converge • encrypted data bag item format change • chef-client lock so safe from simultaneous runs
70. Mini-FAQ!
71. Can I get support for Open Source Chef Server? (from Opscode)
72. Yes.
73. Do I write recipes in Erlang now??
74. Weird Syntax... -module(ranks). -export([fetch_title_and_rank/1]). -include_lib("xmerl/include/xmerl.hrl"). -define(BASE_URL, "http://webservices.amazon.com/onca/xml?" ++ "&ItemId="). fetch_title_and_rank(ISBN) -> URL = amazon_url_for(ISBN), { ok, {_Status, _Headers, Body }} = http:request(URL), { Xml, _Rest } = xmerl_scan:string(Body), [ #xmlText{value=Rank} ] = xmerl_xpath:string("//SalesRank/text()", Xml), [ #xmlText{value=Title} ] = xmerl_xpath:string("//Title/text()", Xml), { Title, Rank }. amazon_url_for(ISBN) -> ?BASE_URL ++ ISBN. No :) http://pragdave.pragprog.com/pragdave/2007/04/a_first_erlang_.html
75. No migration tools? How do I upgrade?
76. Migration tools • Coming soon • But, it's an API... • knife download, knife upload works for many use cases
77. And now a word from our sponsors...
78. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
79. Chef Introductory Workshop • Save 10% with promo code “MEETUP” • http://opscode.eventbrite.com • Get your tickets soon!
80. #ChefConf 2013 Tex OPSCODE-MEETUP - Save 15%
81. Thanks ! • What questions do you have? • Nathen Harvey • nharvey@opscode.com • @nathenharvey