$30 off During Our Annual Pro Sale. View Details »

Intro to Chef - Rocky Mountain Ruby

Nathen Harvey
September 25, 2013

Intro to Chef - Rocky Mountain Ruby

Nathen Harvey

September 25, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. An Introduction to Chef Nathen Harvey @nathenharvey Joshua Timberman @jtimberman

    nharvey@opscode.com github.com/nathenharvey joshua@opscode.com github.com/jtimberman
  2. Introductions

  3. Who Are We?

  4. Who Are You? • System administrator? • Software developer/engineer? •

    Used Chef before?
  5. Tutorial Objectives • Describe the types of problems Chef solves

    • List the components of Chef • Create a new cookbook • Use some primitives of Chef recipes • Follow some common workflows used by experienced Chef users/developers
  6. Expectations • This is a half-day workshop, not a comprehensive

    course. • We will do some hands on exercises. • You should get a taste for automating with Chef. • You should have received instructions for prerequisites prior to this tutorial • Chef works and runs on Windows, but the exercises for this tutorial are Linux-based
  7. We have a problem...

  8. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity

  9. Items of Manipulation (Resources) • Nodes • Networking • Files

    • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems
  10. Application A tale of growth...

  11. Application Application Database Add a database

  12. Application App Databases Make database redundant

  13. App Servers App Databases Application server redundancy

  14. App LB App Servers App Databases Add a load balancer

  15. App LBs App Servers App Databases Webscale!

  16. App LBs App Servers App DB Cache App DBs Now

    we need a caching layer
  17. App LBs App Servers App DB Cache App DBs Infrastructure

    Has a Topology
  18. Round Robin DNS App Servers App DB Cache App DBs

    Floating IP? Your's Is a Snowflake
  19. App LBs App Servers < Shiny! DB slaves Cache DB

    Cache DBs Complexity Increases Quickly Are we monitoring??
  20. None
  21. The Chef Framework • Reasonability • Flexibility • Library &

    Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/
  22. The Chef Tool(s) • ohai • chef-client • chef-shell •

    knife • The Ruby language Omnibus - Full Stack Native Packages
  23. The Chef API • HTTPS, RESTful API w/ JSON, RSA

    key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
  24. The Chef Community • Chef is Open Source! • Apache

    License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com
  25. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg

  26. Chef is Infrastructure as Code • Programmatically provision and configure

    • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
  27. Resources and Recipes

  28. package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source

    "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Declarative abstraction to system resources
  29. package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source

    "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end
  30. Resources describe what Not how.

  31. Resources take action through Providers

  32. Providers perform the how

  33. def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if

    @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example
  34. Package Resource package "git" { yum install git apt-get install

    git pacman sync git pkg_add -r git Providers are determined by node's platform
  35. Recipes are collections of resources package "apache2" do action :install

    end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end
  36. Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain

    related components • Files, Templates, Libraries • A cookbook is responsible for configuring a single thing, e.g. • apache2 • postgresql • A recipe is responsible for a component, e.g. • api • server • client
  37. Roles • Roles describe nodes • Roles indicate functionality •

    Roles have a first class API primitive
  38. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef

    nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search
  39. Search • Search for nodes with Roles • Find Topology

    Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625
  40. Hands On Exercises

  41. Hands on Exercises • The majority of the hands on

    exercises will be related to creating an "apache" cookbook. • The goals are to learn elements of Chef, not to learn Apache. We're going to do things the hard way, by typing in a lot of code. We have some gists with large sections of code already available to reduce what you need to type in certain places. • Errors and typos are good, as they will help students learn how to resolve errors.
  42. Legend

  43. Legend: Do I run that command on my workstation? $

    whoami i-am-a-workstation This is an example of a command to run on your workstation user@hostname:~$ whoami i-am-a-chef-node This is an example of a command to run on your target node via SSH.
  44. $ ifconfig Legend: Example Terminal Command and Output lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST>

    mtu 16384 ! options=3<RXCSUM,TXCSUM> ! inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 ! inet 127.0.0.1 netmask 0xff000000 ! inet6 ::1 prefixlen 128 gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ! ether 28:cf:e9:1f:79:a3 ! inet6 fe80::2acf:e9ff:fe1f:79a3%en0 prefixlen 64 scopeid 0x4 ! inet 10.100.0.84 netmask 0xffffff00 broadcast 10.100.0.255 ! media: autoselect ! status: active p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304 ! ether 0a:cf:e9:1f:79:a3 ! media: autoselect ! status: inactive
  45. OPEN IN EDITOR: SAVE FILE! ~/hello_world Hi! I am a

    friendly file. Legend: Example of editing a file on your workstation
  46. Workstation Setup Getting started

  47. Objectives • Have Chef installed on your Workstation • Have

    an empty chef-repo • Have a unique User created • Have an Organization created for use during training • Understand what Knife is • Have a working Knife configuration
  48. Landscape of Chef-managed Infrastructure

  49. Workstation Setup • Install Chef (if not already installed) •

    https://www.opscode.com/ chef/install/
  50. $ curl -L http://www.opscode.com/chef/install.sh | sudo bash Workstation Setup -

    Mac OS X / Linux % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6515 100 6515 0 0 20600 0 --:--:-- --:--:-- --:--:-- 31172 Downloading Chef for ubuntu... Installing Chef Selecting previously unselected package chef. (Reading database ... 47446 files and directories currently installed.) Unpacking chef (from .../tmp.MqRJP6lz/chef__amd64.deb) ... Setting up chef (11.4.4-2.ubuntu.11.04) ... Thank you for installing Chef! Processing triggers for initramfs-tools ... update-initramfs: Generating /boot/initrd.img-3.2.0-48-virtual
  51. Workstation Setup - Windows • Windows • 2008 (Windows 7)

    or 2012 (Windows 8) • i686 (32-bit) or x86_64 (64-bit) • 11.6.0 Download and install this file
  52. Your Chef Server for this class... • Set up Enterprise

    Chef account • https://getchef.opscode.com/signup
  53. Create new account • Sign up for a new account

  54. Create new account • Fill in the form with your

    own information
  55. Download “Starter Kit” • You get a .zip file from

    clicking this • Unzip the zipfile - you’ll get a “chef- repo” • Put the “chef-repo” somewhere, e.g.: • C:\Users\you\chef-repo (Win) • /Users/you/chef-repo (Mac) • /home/you/chef-repo (Linux)
  56. A quick tour of the chef-repo • Every infrastructure managed

    with Chef has a Chef Repository (“chef-repo”) • Let’s see what’s inside the chef-repo...
  57. $ cd chef-repo Verify that knife is working [~/chef-repo]$

  58. $ ls -al A quick tour of the chef-repo total

    40 drwxr-xr-x 11 nathenharvey staff 374 Sep 23 09:44 . drwxr-xr-x+ 86 nathenharvey staff 2924 Sep 23 09:45 .. drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 .berkshelf drwxr-xr-x 5 nathenharvey staff 170 Sep 23 2013 .chef -rw-r--r--@ 1 nathenharvey staff 495 Sep 23 2013 .gitignore -rw-r--r--@ 1 nathenharvey staff 1433 Sep 23 2013 Berksfile -rw-r--r--@ 1 nathenharvey staff 2292 Sep 23 2013 README.md -rw-r--r--@ 1 nathenharvey staff 3562 Sep 23 2013 Vagrantfile -rw-r--r--@ 1 nathenharvey staff 588 Sep 23 2013 chefignore drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 cookbooks drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 roles
  59. $ ls .chef What’s inside the .chef directory? ORGNAME-validator.pem USERNAME.pem

    knife.rb
  60. What’s inside the .chef directory? • knife.rb is the configuration

    file for Knife. • The other two files are certificates for authentication with the Chef Server • We’ll talk more about that later.
  61. • Your version may be different, that’s ok! Verify Knife

    $ knife --version Chef: 11.6.0 $ knife client list ORGNAME-validator
  62. Knife is the command-line tool for Chef • Knife provides

    an interface between a local Chef repository and the Chef Server. Knife lets you manage: • Nodes • Cookbooks and recipes • Roles • Stores of JSON data (data bags), including encrypted data • Environments • Cloud resources, including provisioning • The installation of Chef on management workstations • Searching of indexed data on the Chef Server
  63. Test Node Setup Setting up a target

  64. Objectives • Have Chef installed on your test node via

    “knife bootstrap” • Understand how knife bootstrap configures a node to use the Organization created in the previous section • Understand the basic configuration needed to run chef-client
  65. Target Instances • ec2-based Instance • Ubuntu 12.04.2 LTS •

    SSH • Username: opscode • Password: opscode
  66. $ knife bootstrap IPADDRESS --sudo -x opscode -P opscode -N

    “target1” "Bootstrap" the Target Instance Bootstrapping Chef on target1 target1 Starting Chef Client, version 11.6.0
  67. local workstation managed node (VM) chef-client $ knife bootstrap IPADDRESS

    --sudo -x USERNAME -P PASSWORD Opscode Enterprise Chef chef_server_url validation_client_name validation_key SSH! bash -c ' install chef configure client run chef'
  68. Verify Your Target Instance’s Chef-Client is Configured Properly $ ssh

    opscode@IPADDRESS opscode@target1:~$ ls /etc/chef client.pem client.rb first-boot.json validation.pem opscode@target1:~$ which chef-client /usr/bin/chef-client
  69. opscode@target1:~$ cat /etc/chef/client.rb Examine /etc/chef/client.rb log_level :auto log_location STDOUT chef_server_url

    "https://api.opscode.com/organizations/ ORGNAME" validation_client_name "ORGNAME-validator" node_name "target1"
  70. Overview of our environment

  71. None
  72. Chef Server • We're using hosted Enterprise Chef

  73. Managed Nodes • Many nodes can be managed by a

    Chef Server • We're going to manage a single node
  74. Workstation • We write Chef code on a local workstation

    like a laptop • We upload that code to the Chef Server • Using the API
  75. Create an apache cookbook

  76. Objectives • Understand what a cookbook is • Know how

    to create a new cookbook • Understand what a recipe is • Understand how to use the package, service, and template resources • Know how to upload a cookbook to the Chef Server • Understand what a run list is, and how to set it • How to read the output of the chef-client run
  77. What is a cookbook? • A cookbook is like a

    "package" for Chef recipes • It contains all the recipes, files, templates, libraries, etc. required to configure a portion of your infrastructure • Typically cookbooks map 1:1 to a piece of software or functionality. • "our tomcat cookbook" • "our zlib cookbook" • "our security_policy cookbook"
  78. $ knife cookbook create apache Create a new Cookbook **

    Creating cookbook apache ** Creating README for cookbook: apache ** Creating CHANGELOG for cookbook: apache ** Creating metadata for cookbook: apache
  79. $ ls -la cookbooks/apache Explore the cookbook total 24 drwxr-xr-x

    13 opscode opscode 442 Mar 24 21:25 . drwxr-xr-x 5 opscode opscode 170 Mar 24 21:25 .. -rw-r--r-- 1 opscode opscode 412 Mar 24 21:25 CHANGELOG.md -rw-r--r-- 1 opscode opscode 1447 Mar 24 21:25 README.md drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 attributes drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 definitions drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 files drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 libraries -rw-r--r-- 1 opscode opscode 276 Mar 24 21:25 metadata.rb drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 providers drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 recipes drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 resources drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 templates
  80. $ cat cookbooks/apache/metadata.rb Read the metadata name 'apache' maintainer 'YOUR_COMPANY_NAME'

    maintainer_email 'YOUR_EMAIL' license 'All rights reserved' description 'Installs/Configures apache' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0'
  81. Cookbook Metadata • Cookbooks are like packages of configuration for

    the infrastructure • They're artifacts that have a name and a version • Metadata can also have information about the cookbook like its license and maintainer • Cookbooks can depend on other cookbooks, too
  82. Other relevant components • We'll use other components of the

    cookbook throughout the exercises •recipes/default.rb •templates/default/ •attributes/[default.rb]
  83. What is a Recipe? • Ordered list of Resources •

    Ruby Domain-Specific Language (DSL)
  84. OPEN IN EDITOR: cookbooks/apache/recipes/default.rb # # Cookbook Name:: apache #

    Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # Edit the default recipe
  85. SAVE FILE! # # Cookbook Name:: apache # Recipe:: default

    # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "apache2" do action :install end Exercise: Add a package resource to install Apache to the default recipe OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
  86. package "apache2" do action :install end So the resource we

    just wrote...
  87. package "apache2" do action :install end So the resource we

    just wrote... • Is a package resource
  88. package "apache2" do action :install end So the resource we

    just wrote... • Is a package resource • Whose name is apache2
  89. package "apache2" do action :install end So the resource we

    just wrote... • Is a package resource • Whose name is apache2 • With an install action
  90. How does the package install? • Resources are declarative -

    we say what we want to have happen, rather than how • Chef uses what platform the node is running to determine the correct provider for a resource
  91. SAVE FILE! ... # All rights reserved - Do Not

    Redistribute # package "apache2" do action :install end service "apache2" do action [ :enable, :start ] end Exercise: Add a service resource to ensure the service is started and enabled at boot OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
  92. service "apache2" do action [ :enable, :start ] end So

    the resource we just wrote...
  93. service "apache2" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource
  94. service "apache2" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource • Whose name is apache2
  95. service "apache2" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource • Whose name is apache2 • With two actions: start and enable
  96. Order Matters • The order you write resources in a

    recipe is the order they will be executed in
  97. SAVE FILE! ... service "apache2" do action [ :enable, :start

    ] end template "/var/www/index.html" do source "index.html.erb" mode "0644" end Add a template resource to write the home page OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
  98. So the resource we just wrote... template "/var/www/index.html" do source

    "index.html.erb" mode "0644" end
  99. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the

    resource we just wrote... • Is a template resource
  100. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the

    resource we just wrote... • Is a template resource • Whose name is /var/www/index.html
  101. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the

    resource we just wrote... • Is a template resource • Whose name is /var/www/index.html • With two parameters: • source of index.html.erb • mode of “0644”
  102. Best Practice: Omit the action if it is the default

    • Has no action! • If you omit the action in Chef, we default to the most common positive action. In this case, it is the :create action. template "/var/www/index.html" do source "index.html.erb" mode "0644" end
  103. Full contents of the apache recipe # # Cookbook Name::

    apache # Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "apache2" do action :install end service "apache2" do action [ :enable, :start ] end template "/var/www/index.html" do source "index.html.erb" mode "0644" end
  104. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, world!</p> Add an

    index template
  105. $ knife cookbook upload apache Exercise: Upload the cookbook Uploading

    apache [0.1.0] Uploaded 1 cookbook.
  106. What is "knife cookbook upload"? • Several knife commands correspond

    to API endpoints • The verb indicates what we're doing with the cookbook • In this case we upload the cookbook to the server through the API. It's stored by the server... • We can show information about it too...
  107. Checkpoint • We have a cookbook named apache • Our

    apache cookbook has a default recipe • It has a template source file, index.html.erb • We have uploaded the cookbook to the Chef server.
  108. Anatomy of a Chef Run

  109. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    [sudo] password for opscode: Starting Chef Client, version 11.6.0 resolving cookbooks for run list: [] Synchronizing Cookbooks: Compiling Cookbooks... [2013-09-23T14:31:38+00:00] WARN: Node target1 has an empty run list. Converging 0 resources Chef Client finished, 0 resources updated
  110. Wait, why didn't Chef apply the recipe? • The node

    has an empty run list • Let us look at what we have now...
  111. $ knife node show target1 View details about the node

    Node Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags:
  112. $ knife node run list add target1 “recipe[apache]” Add apache

    to the run list target1: run_list: recipe[apache]
  113. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install - install version 2.2.22-1ubuntu1 of package apache2 * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from 94850c to 463a6b --- /var/www/index.html 2013-09-23 14:34:24.378187238 +0000 +++ /tmp/chef-rendered-template20130923-3442-1gji5jy 2013-09-23 14:34:28.174187236 +0000 @@ -1,4 +1 @@ -<html><body><h1>It works!</h1> -<p>This is the default web page for this server.</p> -<p>The web server software is running but no content has been added, yet.</p> -</body></html> +<p>Hello, world</p> Chef Client finished, 2 resources updated
  114. * service[apache2] action enable (up to date) * service[apache2] action

    start (up to date) Why wasn't the service updated? • Chef takes idempotent actions on resources to ensure they are converged to the desired state • The apache2 package on Ubuntu automatically enables and starts the apache2 service in its post- install script • Thus, Chef didn't need to do anything
  115. If we stopped apache2... $ sudo /etc/init.d/apache2 stop * Stopping

    web server apache2 apache2: [ OK ] $ sudo chef-client .... * service[apache2] action start - start service service[apache2]
  116. build node authenticate sync cookbooks load cookbooks converge node.save notification

    handlers exception Yes No chef-client success? expanded run list (recipes) Ohai! node_name platform platform_version
  117. Processing Recipes is Two-Phase • Chef processes recipes in two

    phases during "convergence" • Evaluate all the Ruby code in the recipe, looking for "Chef Resources" • Execute the providers' actions to put each resource in the declared state
  118. Node Attributes

  119. The Chef Node Object • Nodes are the objects that

    you manage with Chef • They have a few different properties • attributes • run_list • chef_environment
  120. The Chef Node Object • In client/server Chef, the Chef

    Server stores node object data • It becomes searchable through the API with knife and with recipes • Some of the data comes from ohai, which takes an inventory of the system and emits JSON data • You can add data to the node through attributes in cookbooks, roles, directly on a node, etc
  121. Node Attributes Syntax • Node attributes are hashes (of hashes)

    • Reference hash keys with square brackets and keys as strings •node["hostname"] •node["kernel"]["machine"]
  122. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, world!</p> <p>My name

    is <%= node['hostname'] %></p> Update the home page template
  123. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook.
  124. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from 463a6b to b50d3b --- /var/www/index.html 2013-09-23 14:34:28.174187236 +0000 +++ /tmp/chef-rendered-template20130923-4337-1csr3dp 2013-09-23 14:39:56.426187239 +0000 @@ -1 +1,2 @@ -<p>Hello, world</p> +<p>Hello, world!</p> +<p>My name is ip-10-164-9-210</p> Chef Client finished, 1 resources updated
  125. opscode@target1:~$ ohai Run Ohai on the node { "languages": {

    "ruby": { }, "perl": { "version": "5.14.2", "archname": "x86_64-linux-gnu-thread-multi" }, "python": { "version": "2.7.3", "builddate": "Aug 1 2012, 05:14:39" } }, <SNIP>
  126. opscode@target1:~$ ohai hostname Run Ohai on the node [ "ip-10-164-9-210"

    ]
  127. $ knife node show target1 Show the node object Node

    Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: recipe[apache] Roles: Recipes: apache Platform: ubuntu 12.04 Tags:
  128. $ knife node show target1 -a hostname Show specific node

    attribute target1: hostname: ip-10-164-9-210
  129. Cookbook Attributes

  130. Objectives • Set node attributes from a cookbook • Understand

    node attribute precedence
  131. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/attributes/default.rb default["apache"]["greeting"] = "World" Create

    an attributes file
  132. Cookbook Attributes & Precedence • Always set default node attributes

    in your cookbooks' attributes files • Use "sane" defaults - no surprises • You can use attributes in roles to set new values • Roles take precedence over cookbook settings • When a value must be set to a certain value, use override, but use this sparingly • You can’t override ohai's (automatic) attributes!
  133. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p>

    <p>My name is <%= node['hostname'] %></p> Update the home page template
  134. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook.
  135. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from b50d3b to 764f54 --- /var/www/index.html 2013-09-23 14:39:56.426187239 +0000 +++ /tmp/chef-rendered-template20130923-4826-1me21yb 2013-09-23 14:54:37.974187238 +0000 @@ -1,2 +1,2 @@ -<p>Hello, world!</p> +<p>Hello, World!</p> <p>My name is ip-10-164-9-210</p> Chef Client finished, 1 resources updated
  136. Checkpoint • We have a node attribute • node["apache"]["greeting"] •

    We've updated the index.html template to use this attribute. • This will be used again soon!
  137. Data Bags

  138. Objectives • Use Data Bags for data-driven recipes • Use

    multiple recipes for a node's run list • Control execution of arbitrary commands with Chef's resource conditionals
  139. Data Bags are generic stores of information • Data bags

    are generic, arbitrary stores of information about the infrastructure. • Data Bag Items are JSON data • Our apache cookbook provides a good baseline • We'll drive site-specific virtual hosts with data bags
  140. $ mkdir -p data_bags/vhosts Create a directory for Data Bags

  141. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/bears.json { "id" : "bears",

    "port" : 80 } Add a Data Bag Item
  142. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/clowns.json { "id" : "clowns",

    "port" : 81 } Add a Data Bag Item
  143. $ knife upload data_bags/vhosts Upload the data bags Created data_bags/vhosts

    Created data_bags/vhosts/bears.json Created data_bags/vhosts/clowns.json
  144. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/default.rb service "apache2" do action

    [:start, :enable] end execute "a2dissite default" do only_if do File.symlink?("/etc/apache2/sites-enabled/000-default") end notifies :restart, "service[apache2]" end template "/var/www/index.html" do Disable the default apache site
  145. template "/var/www/index.html" do source "index.html.erb" mode "0644" end OPEN IN

    EDITOR: /cookbooks/apache/recipes/default.rb SAVE FILE! Delete the template resource • Remove this resource from the recipe
  146. $ knife diff cookbooks/apache Diff the cookbook --- cookbooks/apache/recipes/default.rb 2013-09-23

    11:46:22.000000000 -0400 +++ cookbooks/apache/recipes/default.rb 2013-09-23 11:46:22.000000000 -0400 @@ -14,7 +14,9 @@ action [:start, :enable] end -template "/var/www/index.html" do - source "index.html.erb" - mode "0644" -end +execute "a2dissite default" do + only_if do + File.symlink?("/etc/apache2/sites-enabled/000-default") + end + notifies :restart, "service[apache2]" +end
  147. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook.
  148. $ knife diff cookbooks/apache Diff the cookbook

  149. A new recipe for virtual hosts • We'll create an

    apache::vhosts recipe to manage the virtual hosts we created in data bag items • There's a number of new things to talk about in this recipe • We'll take this nice and slow :)
  150. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb data_bag("vhosts").each do |site| site_data

    = data_bag_item("vhosts", site) site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" end Create a vhosts recipe
  151. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb document_root = "/srv/apache/#{site_name}" template

    "/etc/apache2/sites-available/#{site_name}" do source "custom-vhost.erb" mode "0644" variables( :document_root => document_root, :port => site_data["port"] ) end end Add a Virtual Hosts Configuration Template
  152. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end execute "a2ensite #{site_name}"

    do not_if do ::File.symlink?("/etc/apache2/sites-enabled/#{site_name}") end notifies :restart, "service[apache2]" end end Enable the Virtual Hosts
  153. not_if and only_if • The not_if parameter causes the resource’s

    actions to be taken only if its argument returns false • The only_if parameter is the opposite of not_if - the actions are taken only if the arguments return true • Both not_if and only_if are part of Chef (resources), not part of Ruby
  154. not_if and only_if • not_if and only_if parameters take either

    a string, or a Ruby block argument (do..end or {..}) • When the argument is a string, Chef evaluates it as a shell command to run. • When the argument is a Ruby block, Chef evaluates it as Ruby code to execute. • This is the equivalent to the code we wrote: not_if "test -L /etc/apache2/sites-enabled/#{site_name}"
  155. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end directory document_root do

    mode "0755" recursive true end end Add a directory resource
  156. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end template "#{document_root}/index.html" do

    source "index.html.erb" mode "0644" variables( :site_name => site_name, :port => site_data["port"] ) end end Index for each vhost https://gist.github.com/6673845
  157. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/custom-vhosts.erb <% if @port !=

    80 -%> Listen <%= @port %> <% end -%> <VirtualHost *:<%= @port %>> ServerAdmin webmaster@localhost DocumentRoot <%= @document_root %> <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory <%= @document_root %>> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> </VirtualHost> Index for each vhost https://gist.github.com/2866454
  158. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p>

    <p>My name is <%= node['hostname'] %></p> <p>We love <%= @site_name %></p> <p>Served from <%= node['ipaddress'] %>:<%= @port %></p> Update the index.html template
  159. $ knife diff cookbooks/apache Diff the cookbook diff --knife cookbooks/apache/recipes/default.rb

    cookbooks/apache/recipes/default.rb --- cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400 +++ cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400 @@ -19,4 +19,4 @@ File.symlink?("/etc/apache2/sites-enabled/000-default") end notifies :restart, "service[apache2]" -end \ No newline at end of file +end diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/recipes/vhosts.rb new file --- /dev/null 2013-09-23 12:04:18.000000000 -0400 +++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:04:18.000000000 -0400 @@ -0,0 +1,35 @@ +data_bag("vhosts").each do |site|
  160. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook.
  161. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run - execute a2dissite default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 2 resources updated
  162. Oops!

  163. $ knife node run list add target1 "recipe[apache::vhosts]" Update the

    run list target1: run_list: recipe[apache] recipe[apache::vhosts]
  164. opscode@target1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 11 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears] action create - create new file /etc/apache2/sites-available/bears - update content in file /etc/apache2/sites-available/bears from none to bf6a12 --- /etc/apache2/sites-available/bears 2013-09-23 16:11:07.130187239 +0000 +++ /tmp/chef-rendered-template20130923-5699-1w4prah 2013-09-23 16:11:07.130187239 +0000 @@ -0,0 +1 @@
  165. Think about what we just did... • We had two

    virtual hosts... • But we could arbitrarily add more... • Tigers on port 82, Lions on port 83, oh my!
  166. Checkpoint • Our cookbook has two recipes, default and vhosts

    • Additional data bags can be added, expanding our Virtual Hosting empire!
  167. Roles

  168. What is a role? • So far, we’ve been just

    adding recipes directly to our single node's run list • But that’s not how infrastructure works - think about how you refer to servers • "It’s a web server" • "It’s a database server" • or, "It's a database-master server"... • "It’s a monitoring server"
  169. How you use roles • Roles allow you to conveniently

    encapsulate the run lists and attributes required for a server to "be" what you already think it is • In practice, roles make it easy to configure many nodes identically without repeating yourself each time • Roles are a first class API primitive on the Chef Server
  170. Best Practice • Like data bags, you have options for

    creating roles • The best practice is that all of your roles live in the roles directory of your chef-repo • They can be created via the API and knife, but having them in source control gives you the history of changes
  171. OPEN IN EDITOR: SAVE FILE! roles/webserver.json { "name" : "webserver",

    "default_attributes" : { "apache" : { "greeting" : "Rocky Mountain Ruby" } }, "run_list" : [ "recipe[apache]", "recipe[apache::vhosts]" ] } Create a webserver role
  172. Components of a role • Roles must have a name

    • Roles may have a description • Roles may have a run_list, just like a node • Roles may set node attributes • default_attributes • override_attributes http://docs.opscode.com/essentials_roles.html
  173. $ knife upload roles/webserver.json Upload the role Created roles/webserver.json

  174. $ knife node run list remove target1 "recipe[apache::vhosts]" Remove the

    vhosts recipe target1: run_list: recipe[apache]
  175. $ knife node run list remove target1 "recipe[apache]" Remove the

    apache recipe target1: run_list:
  176. $ knife node run list add target1 "role[webserver]" Add the

    webserver role target1: run_list: role[webserver]
  177. opscode@target1:~$ sudo chef-client Run chef-client * template[/srv/apache/bears/index.html] action create -

    update content in file /srv/apache/bears/index.html from 509865 to bee814 --- /srv/apache/bears/index.html 2013-09-23 16:11:07.398187242 +0000 +++ /tmp/chef-rendered-template20130923-6418-om7gju 2013-09-23 16:29:27.778187238 +0000 @@ -1,4 +1,4 @@ -<p>Hello, World!</p> +<p>Hello, Rocky Mountain Ruby!</p> <p>My name is ip-10-164-9-210</p> <p>We love bears</p> <p>Served from 10.164.9.210:80</p> * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create - update content in file /srv/apache/clowns/index.html from 2055c3 to 2290a0 --- /srv/apache/clowns/index.html 2013-09-23 16:11:07.654187238 +0000 +++ /tmp/chef-rendered-template20130923-6418-j7lsw6 2013-09-23 16:29:27.862187239 +0000 @@ -1,4 +1,4 @@
  178. Attributes can be set multiple places • This is for

    flexibility • Set a "sane default" that will be used in a cookbook • Override it easily in a role (higher priority) • In all, there are 15 places where attributes come from (!!) • In practice, you'll use 2-3 most of the time. • The others are there when you need them. • http://docs.opscode.com/chef_overview_attributes.html
  179. Data Bags Are Not Attributes • Important: data bag items

    are not attributes • Data bags are a separate API end point • Data bags are not tied to a specific node or role in the infrastructure • Not even necessarily tied to anything, just data you want to store
  180. Checkpoint • We now have a webserver role. • We

    could apply just this role on more nodes to scale out our Virtual Hosting service. • Roles are a great way to assign attributes for specific purposes
  181. Search

  182. Chef's Search Feature • Search ties together the infrastructure topology

    • We can now search the Chef Server for all the "webserver" nodes • This is relevant for a variety of reasons • Load balancing several front ends (search from nginx, or haproxy for example) • Monitoring HTTP (search from nagios cookbook) • Graphing traffic (search from munin cookbook)
  183. However, we only have one node... • We don't have

    an environment that is really setup for doing search • But we will talk about what kind of search we do with the command-line, or in a recipe
  184. $ knife search node "role:webserver" Search for webservers with knife

    1 items found Node Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: role[webserver] Roles: webserver Recipes: apache, apache::vhosts Platform: ubuntu 12.04 Tags:
  185. Search from a recipe webservers = search(:node, "role:webserver") # or

    search(:node, "role:webserver").each do |web| #things do do on all web servers... end
  186. Available search indexes • The Chef Server indexes JSON data

    for the major API endpoints: • node • client • environment • role • All data bags are also indexed for search... • vhosts (our example)
  187. $ knife search vhosts "port:81" Search for webservers with knife

    1 items found chef_type: data_bag_item data_bag: vhosts id: clowns port: 81
  188. Replace Data Bag Item Lookup with Search

  189. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb search(:vhosts).each do |site_data| site_name

    = site_data["id"] document_root = "/srv/apache/#{site_name}" Refactor the vhosts recipe
  190. $ knife diff cookbooks/apache View the diff diff --knife cookbooks/apache/recipes/vhosts.rb

    cookbooks/apache/ recipes/vhosts.rb --- cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000 -0400 +++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000 -0400 @@ -1,5 +1,4 @@ -data_bag("vhosts").each do |site| - site_data = data_bag_item("vhosts", site) +search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}"
  191. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook.
  192. opscode@target1:~$ sudo chef-client Run chef-client Starting Chef Client, version 11.6.0

    resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 11 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears] action create (up to date) * execute[a2ensite bears] action run (skipped due to not_if) * directory[/srv/apache/bears] action create (up to date) * template[/srv/apache/bears/index.html] action create (up to date) * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create (up to date) Chef Client finished, 0 resources updated
  193. Hands on Recap

  194. Our cookbook... • Chef cookbook "apache" with two recipes: •

    default (manages apache package and service) • vhosts (iterates over data bags and renders vhost configuration) • However, Opscode publishes an "apache2" cookbook that manages much much more, including all apache2 configuration, modules, sites Debian style w/ a2ensite/a2enmod, etc
  195. Our role • webserver role in JSON • Sets a

    default attribute • Sets a run list • Uploaded with knife upload
  196. Our data bag • The vhosts data bag serves as

    an example • Directory structure follows the data bag API end point and "knife upload" makes it easy to upload everything • Create additional vhosts to see how this is dynamically expanded easily
  197. Chef Server • Hosted Enterprise Chef • Free to manage

    up to five nodes
  198. Using A Different Chef Server • Configure knife by modifying

    the chef_server_url and the node_name values in ./chef/knife.rb • Get the validation client key from the Chef Server (this differs by implementation): • Open Source: •/etc/chef-server/chef-validator.pem • Hosted Chef / Private Chef • Download after creating an organization
  199. $ cat .chef/knife.rb The knife.rb configuration # See http://docs.opscode.com/config_rb_knife.html for

    more information on knife configuration options current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name "nathenharveyrmr1" client_key "#{current_dir}/nathenharveyrmr1.pem" validation_client_name "nhrmr1-validator" validation_key "#{current_dir}/nhrmr1-validator.pem" chef_server_url "https://api.opscode.com/organizations/nhrmr1" cache_type 'BasicFile' cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) cookbook_path ["#{current_dir}/../cookbooks"]
  200. Chef Resources

  201. Chef Resources (Core Chef) • Chef client comes with 24+

    different resources • Packages, files, services, users, symlinks, registry keys, and more • Each resource has one or more providers • Some resources have platform-specific providers (e.g., package, service, user, group) http://docs.opscode.com/resource.html
  202. Managing files and directories • file • cookbook_file • remote_file

    • template • directory • remote_directory • link http://docs.opscode.com/resource.html
  203. Managing packages • package • apt_package • chef_gem • dpkg_package

    • easy_install_package • freebsd_package • gem_package • ips_package • macports_package • pacman_package • portage_package • rpm_package • smartos_package • solaris_package • yum_package http://docs.opscode.com/resource.html
  204. RubyGem packages • chef_gem - install a RubyGem into Chef's

    Ruby environment to be used in a Chef recipe • gem_package - install a RubyGem to be used by the system or an application http://docs.opscode.com/resource.html
  205. Services • service is used to manage services using the

    common init systems available • Each platform has it's own provider • arch, debian, freebsd, gentoo, "init", insserv • invokercd, macosx (launchd), redhat, "simple" • solaris (SMF), upstart, windows http://docs.opscode.com/resource.html
  206. Other Chef Resources • cron • deploy (revision, timestamped) •

    env (windows) • group • ifconfig (RHEL) • log • mdadm • mount • ohai • registry_key (windows) • route • scm (git, subversion) • user http://docs.opscode.com/resource.html
  207. There when you need them... • execute (we used this)

    • script (bash, perl, python, csh, ruby interpreters) • windows_script (batch, powershell) - Chef 11.6.0! • ruby_block http://docs.opscode.com/resource.html
  208. Cookbooks to Know About

  209. community.opscode.com • apache2, nginx • ark • build-essential • chef-client

    • chruby • cron • line • java • jenkins • minitest-handler • mysql, postgresql • openssh • omnibus_updater • partial_search • runit • whitelist-node-attrs
  210. Opscode Cookbooks with Chef Resources • Opscode has several cookbooks

    that include new custom Chef Resources • apt (apt_repository) • aws (aws_ebs_volume, aws_elastic_ip, and more) • yum (yum_repository) • windows (windows_package, windows_feature, and more) • homebrew (homebrew_package) • runit (runit_service) • many more! http://docs.opscode.com/chef/lwrps_custom.html
  211. Getting Cookbooks from the Community Site • Knife commands: •knife

    cookbook site download • knife cookbook site install (integrates with git) • Dependency resolvers (a la Ruby's bundler): •librarian-chef •berkshelf
  212. Tools to be aware of...

  213. Knife Plugins • http://docs.opscode.com/community_plugin_knife.html • Cloud plugins (ec2, openstack, rackspace,

    google azure, hpcloud, cloudstack, eucalyptus, and more) • knife-server • knife-solo • knife-preflight • knife-essentials • https://github.com/jkeiser/knife-essentials
  214. Chef Handlers • http://docs.opscode.com/community_plugin_report_handler.html • IRC • Campfire • HipChat

    • DataDog • Splunk Storm • Graylog2 • Graphite
  215. Workflow Helpers • Berkshelf: berkshelf.com • Librarian-chef: github.com/applicationsonline/ librarian-chef •

    Knife Spork: github.com/jonlives/knife-spork • Vagrant: vagrantup.com (also a testing tool)
  216. Cookbook Testing • Vagrant: vagrantup.com • Test Kitchen: github.com/opscode/test-kitchen •

    Foodcritic: acrmp.github.io/foodcritic • ChefSpec: acrmp.github.io/chefspec/ • RSpec: rspec.info • minitest-chef: github.com/calavera/minitest-chef- handler • cookbook: minitest-handler
  217. Thank you! Questions & Answers nharvey@opscode.com joshua@opscode.com