Intro to Chef - Rocky Mountain Ruby

An Introduction to Chef
Nathen Harvey
@nathenharvey
Joshua Timberman
@jtimberman
[email protected]
github.com/nathenharvey
[email protected]
github.com/jtimberman

View Slide

Introductions

View Slide

Who Are We?

View Slide

Who Are You?
• System administrator?
• Software developer/engineer?
• Used Chef before?

View Slide

Tutorial Objectives
• Describe the types of problems Chef solves
• List the components of Chef
• Create a new cookbook
• Use some primitives of Chef recipes
• Follow some common workflows used by
experienced Chef users/developers

View Slide

Expectations
• This is a half-day workshop, not a comprehensive
course.
• We will do some hands on exercises.
• You should get a taste for automating with Chef.
• You should have received instructions for
prerequisites prior to this tutorial
• Chef works and runs on Windows, but the exercises
for this tutorial are Linux-based

View Slide

We have a problem...

View Slide

http://www.ﬂickr.com/photos/michaelheiss/3090102907/
Complexity

View Slide

Items of Manipulation (Resources)
• Nodes
• Networking
• Files
• Directories
• Symlinks
• Mounts
• Routes
• Users
• Groups
• Packages
• Services
• Filesystems

View Slide

Application
A tale of growth...

View Slide

Application
Application Database
Add a database

View Slide

Application
App Databases
Make database redundant

View Slide

App Servers
App Databases
Application server redundancy

View Slide

App LB
App Servers
App Databases
Add a load balancer

View Slide

App LBs
App Servers
App Databases
Webscale!

View Slide

App LBs
App Servers
App DB Cache
App DBs
Now we need a caching layer

View Slide

App LBs
App Servers
App DB Cache
App DBs
Infrastructure Has a Topology

View Slide

Round Robin DNS
App Servers
App DB Cache
App DBs
Floating IP?
Your's Is a Snowflake

View Slide

App LBs
App Servers
< Shiny!
DB slaves
Cache
DB Cache
DBs
Complexity Increases Quickly
Are we monitoring??

View Slide

View Slide

The Chef Framework
• Reasonability
• Flexibility
• Library & Primitives
• TIMTOWTDI
• Sane defaults
http://www.ﬂickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/

View Slide

The Chef Tool(s)
• ohai
• chef-client
• chef-shell
• knife
• The Ruby language
Omnibus - Full Stack
Native Packages

View Slide

The Chef API
• HTTPS, RESTful API w/ JSON, RSA key auth
• Infrastructure data store such as node data
• Search Service
• Derivative Services?
http://www.ﬂickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/

View Slide

The Chef Community
• Chef is Open Source!
• Apache License, Version 2.0
• Hundreds of Individual and Corporate contributors.
• Hundreds of cookbooks available from the
community
• http://community.opscode.com

View Slide

How does it work?
http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg

View Slide

Chef is Infrastructure as Code
• Programmatically
provision and
configure
• Treat like any other
code base
• Reconstruct business
from code repository,
data backup, and bare
metal resources.
http://www.ﬂickr.com/photos/louisb/4555295187/

View Slide

Resources and
Recipes

View Slide

package "apache2" do
action :install
end
template "/etc/apache2/apache2.conf" do
source "apache2.conf.erb"
owner "www-data"
group "www-data"
mode 00644
notifies :restart, "service[apache2]"
end
service "apache2" do
supports :status => true, :restart => true
action [:enable, :start]
end
Declarative abstraction to
system resources

View Slide

action :install
end
owner "www-data"
group "www-data"
mode 00644
end
end

View Slide

Resources describe what
Not how.

View Slide

Resources take action
through Providers

View Slide

Providers perform the
how

View Slide

def install_package(name, version)
package_name = "#{name}=#{version}"
package_name = name if @is_virtual_package
run_command_with_systems_locale(
:command => "apt-get -q -y
#{expand_options(@new_resource.options)}
install #{package_name}",
:environment => {
"DEBIAN_FRONTEND" => "noninteractive"
}
)
end
Provider Example

View Slide

Package Resource
package "git"
{
yum install git
apt-get install git
pacman sync git
pkg_add -r git
Providers are
determined
by node's platform

View Slide

Recipes are collections of resources
action :install
end
owner "www-data"
group "www-data"
mode 00644
end
end

View Slide

Cookbooks
• Cookbooks are collections of Recipes
• Cookbooks contain related components
• Files, Templates, Libraries
• A cookbook is responsible for configuring a single thing, e.g.
• apache2
• postgresql
• A recipe is responsible for a component, e.g.
• api
• server
• client

View Slide

Roles
• Roles describe nodes
• Roles indicate functionality
• Roles have a first class API primitive

View Slide

http://www.ﬂickr.com/photos/peterrosbjerg/3913766224/
Chef Nodes
• Chef runs on nodes
• Chef nodes do the heavy lifting
• Authority about themselves
• Stored on the server when using Chef Server
• Indexed for search

View Slide

Search
• Search for nodes with Roles
• Find Topology Data
• IP addresses
• Hostnames
• FQDNs
http://www.ﬂickr.com/photos/kathycsus/2686772625

View Slide

Hands On Exercises

View Slide

Hands on Exercises
• The majority of the hands on exercises will be
related to creating an "apache" cookbook.
• The goals are to learn elements of Chef, not to learn
Apache. We're going to do things the hard way, by
typing in a lot of code. We have some gists with
large sections of code already available to reduce
what you need to type in certain places.
• Errors and typos are good, as they will help students
learn how to resolve errors.

View Slide

Legend

View Slide

Legend: Do I run that command on my workstation?
$ whoami
i-am-a-workstation
This is an example of a command to run on your workstation
user@hostname:~$ whoami
i-am-a-chef-node
This is an example of a command to run on your target node via SSH.

View Slide

$ ifconfig
Legend: Example Terminal Command and Output
lo0: flags=8049 mtu 16384
! options=3
! inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
! inet 127.0.0.1 netmask 0xff000000
! inet6 ::1 prefixlen 128
gif0: flags=8010 mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863 mtu 1500
! ether 28:cf:e9:1f:79:a3
! inet6 fe80::2acf:e9ff:fe1f:79a3%en0 prefixlen 64 scopeid 0x4
! inet 10.100.0.84 netmask 0xffffff00 broadcast 10.100.0.255
! media: autoselect
! status: active
p2p0: flags=8843 mtu 2304
! ether 0a:cf:e9:1f:79:a3
! media: autoselect
! status: inactive

View Slide

OPEN IN EDITOR:
SAVE FILE!
~/hello_world
Hi!
I am a friendly file.
Legend: Example of editing a file on your workstation

View Slide

Workstation Setup
Getting started

View Slide

Objectives
• Have Chef installed on your Workstation
• Have an empty chef-repo
• Have a unique User created
• Have an Organization created for use during training
• Understand what Knife is
• Have a working Knife configuration

View Slide

Landscape of Chef-managed Infrastructure

View Slide

Workstation Setup
• Install Chef (if not already
installed)
• https://www.opscode.com/
chef/install/

View Slide

$ curl -L http://www.opscode.com/chef/install.sh | sudo bash
Workstation Setup - Mac OS X / Linux
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6515 100 6515 0 0 20600 0 --:--:-- --:--:-- --:--:-- 31172
Downloading Chef for ubuntu...
Installing Chef
Selecting previously unselected package chef.
(Reading database ... 47446 files and directories currently installed.)
Unpacking chef (from .../tmp.MqRJP6lz/chef__amd64.deb) ...
Setting up chef (11.4.4-2.ubuntu.11.04) ...
Thank you for installing Chef!
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-3.2.0-48-virtual

View Slide

Workstation Setup - Windows
• Windows
• 2008 (Windows 7) or
2012 (Windows 8)
• i686 (32-bit) or x86_64
(64-bit)
• 11.6.0 Download and install this file

View Slide

Your Chef Server for this class...
• Set up Enterprise Chef account
• https://getchef.opscode.com/signup

View Slide

Create new account
• Sign up for a new
account

View Slide

Create new account
• Fill in the form with
your own information

View Slide

Download “Starter Kit”
• You get a .zip file from clicking this
• Unzip the zipfile - you’ll get a “chef-
repo”
• Put the “chef-repo” somewhere,
e.g.:
• C:\Users\you\chef-repo (Win)
• /Users/you/chef-repo (Mac)
• /home/you/chef-repo (Linux)

View Slide

A quick tour of the chef-repo
• Every infrastructure managed with Chef has a Chef
Repository (“chef-repo”)
• Let’s see what’s inside the chef-repo...

View Slide

$ cd chef-repo
Verify that knife is working
[~/chef-repo]$

View Slide

$ ls -al
A quick tour of the chef-repo
total 40
drwxr-xr-x 11 nathenharvey staff 374 Sep 23 09:44 .
drwxr-xr-x+ 86 nathenharvey staff 2924 Sep 23 09:45 ..
drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 .berkshelf
drwxr-xr-x 5 nathenharvey staff 170 Sep 23 2013 .chef
-rw-r--r--@ 1 nathenharvey staff 495 Sep 23 2013 .gitignore
-rw-r--r--@ 1 nathenharvey staff 1433 Sep 23 2013 Berksfile
-rw-r--r--@ 1 nathenharvey staff 2292 Sep 23 2013 README.md
-rw-r--r--@ 1 nathenharvey staff 3562 Sep 23 2013 Vagrantfile
-rw-r--r--@ 1 nathenharvey staff 588 Sep 23 2013 chefignore
drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 cookbooks
drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 roles

View Slide

$ ls .chef
What’s inside the .chef directory?
ORGNAME-validator.pem
USERNAME.pem
knife.rb

View Slide

What’s inside the .chef directory?
• knife.rb is the configuration file for Knife.
• The other two files are certificates for authentication
with the Chef Server
• We’ll talk more about that later.

View Slide

• Your version may
be different, that’s
ok!
Verify Knife
$ knife --version
Chef: 11.6.0
$ knife client list
ORGNAME-validator

View Slide

Knife is the command-line tool for Chef
• Knife provides an interface between a local Chef repository and
the Chef Server. Knife lets you manage:
• Nodes
• Cookbooks and recipes
• Roles
• Stores of JSON data (data bags), including encrypted data
• Environments
• Cloud resources, including provisioning
• The installation of Chef on management workstations
• Searching of indexed data on the Chef Server

View Slide

Test Node Setup
Setting up a target

View Slide

Objectives
• Have Chef installed on your test node via “knife
bootstrap”
• Understand how knife bootstrap configures a node
to use the Organization created in the previous
section
• Understand the basic configuration needed to run
chef-client

View Slide

Target Instances
• ec2-based Instance
• Ubuntu 12.04.2 LTS
• SSH
• Username: opscode
• Password: opscode

View Slide

$ knife bootstrap IPADDRESS --sudo -x opscode -P opscode -N “target1”
"Bootstrap" the Target Instance
Bootstrapping Chef on target1
target1 Starting Chef Client, version 11.6.0

View Slide

local workstation
managed node
(VM)
chef-client
$ knife bootstrap IPADDRESS --sudo -x USERNAME -P PASSWORD
Opscode Enterprise Chef
chef_server_url
validation_client_name
validation_key
SSH!
bash -c '
install chef
conﬁgure client
run chef'

View Slide

Verify Your Target Instance’s Chef-Client is Configured Properly
$ ssh opscode@IPADDRESS
opscode@target1:~$ ls /etc/chef
client.pem client.rb first-boot.json validation.pem
opscode@target1:~$ which chef-client
/usr/bin/chef-client

View Slide

opscode@target1:~$ cat /etc/chef/client.rb
Examine /etc/chef/client.rb
log_level :auto
log_location STDOUT
chef_server_url "https://api.opscode.com/organizations/
ORGNAME"
validation_client_name "ORGNAME-validator"
node_name "target1"

View Slide

Overview of our
environment

View Slide

View Slide

Chef Server
• We're using hosted
Enterprise Chef

View Slide

Managed Nodes
• Many nodes can be
managed by a Chef
Server
• We're going to
manage a single node

View Slide

Workstation
• We write Chef code
on a local workstation
like a laptop
• We upload that code
to the Chef Server
• Using the API

View Slide

Create an apache
cookbook

View Slide

Objectives
• Understand what a cookbook is
• Know how to create a new cookbook
• Understand what a recipe is
• Understand how to use the package, service, and
template resources
• Know how to upload a cookbook to the Chef Server
• Understand what a run list is, and how to set it
• How to read the output of the chef-client run

View Slide

What is a cookbook?
• A cookbook is like a "package" for Chef recipes
• It contains all the recipes, files, templates, libraries,
etc. required to configure a portion of your
infrastructure
• Typically cookbooks map 1:1 to a piece of software or
functionality.
• "our tomcat cookbook"
• "our zlib cookbook"
• "our security_policy cookbook"

View Slide

$ knife cookbook create apache
Create a new Cookbook
** Creating cookbook apache
** Creating README for cookbook: apache
** Creating CHANGELOG for cookbook: apache
** Creating metadata for cookbook: apache

View Slide

$ ls -la cookbooks/apache
Explore the cookbook
total 24
drwxr-xr-x 13 opscode opscode 442 Mar 24 21:25 .
drwxr-xr-x 5 opscode opscode 170 Mar 24 21:25 ..
-rw-r--r-- 1 opscode opscode 412 Mar 24 21:25 CHANGELOG.md
-rw-r--r-- 1 opscode opscode 1447 Mar 24 21:25 README.md
drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 attributes
drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 definitions
drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 files
drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 libraries
-rw-r--r-- 1 opscode opscode 276 Mar 24 21:25 metadata.rb
drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 providers
drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 recipes
drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 resources
drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 templates

View Slide

$ cat cookbooks/apache/metadata.rb
Read the metadata
name 'apache'
maintainer 'YOUR_COMPANY_NAME'
maintainer_email 'YOUR_EMAIL'
license 'All rights reserved'
description 'Installs/Configures apache'
long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
version '0.1.0'

View Slide

Cookbook Metadata
• Cookbooks are like packages of configuration for the
infrastructure
• They're artifacts that have a name and a version
• Metadata can also have information about the
cookbook like its license and maintainer
• Cookbooks can depend on other cookbooks, too

View Slide

Other relevant components
• We'll use other components of the cookbook
throughout the exercises
•recipes/default.rb
•templates/default/
•attributes/[default.rb]

View Slide

What is a Recipe?
• Ordered list of Resources
• Ruby Domain-Specific Language (DSL)

View Slide

OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
#
# Cookbook Name:: apache
# Recipe:: default
#
# Copyright 2013, YOUR_COMPANY_NAME
#
# All rights reserved - Do Not Redistribute
#
Edit the default recipe

View Slide

SAVE FILE!
#
# Recipe:: default
#
#
#
action :install
end
Exercise: Add a package resource to install Apache to the default
recipe

View Slide

action :install
end
So the resource we just wrote...

View Slide

action :install
end
• Is a package resource

View Slide

action :install
end
• Whose name is
apache2

View Slide

action :install
end
• Whose name is
apache2
• With an install action

View Slide

How does the package install?
• Resources are declarative - we say what we want
to have happen, rather than how
• Chef uses what platform the node is running to
determine the correct provider for a resource

View Slide

SAVE FILE!
...
#
action :install
end
action [ :enable, :start ]
end
Exercise: Add a service resource to ensure the service is started
and enabled at boot

View Slide

end

View Slide

end
• Is a service resource

View Slide

end
• Whose name is
apache2

View Slide

end
• Whose name is
apache2
• With two actions:
start and enable

View Slide

Order Matters
• The order you write resources in a recipe is the
order they will be executed in

View Slide

SAVE FILE!
...
end
template "/var/www/index.html" do
source "index.html.erb"
mode "0644"
end
Add a template resource to write the home page

View Slide

mode "0644"
end

View Slide

mode "0644"
end
• Is a template
resource

View Slide

mode "0644"
end
• Is a template
resource
• Whose name is
/var/www/index.html

View Slide

mode "0644"
end
• Is a template
resource
• Whose name is
/var/www/index.html
• With two parameters:
• source of
index.html.erb
• mode of “0644”

View Slide

Best Practice: Omit the action if it is the default
• Has no action!
• If you omit the action
in Chef, we default to
the most common
positive action. In this
case, it is the :create
action.
mode "0644"
end

View Slide

Full contents of the apache recipe
#
# Recipe:: default
#
#
#
action :install
end
end
mode "0644"
end

View Slide

OPEN IN EDITOR:
SAVE FILE!
cookbooks/apache/templates/default/index.html.erb
Hello, world!
Add an index template

View Slide

$ knife cookbook upload apache
Exercise: Upload the cookbook
Uploading apache [0.1.0]
Uploaded 1 cookbook.

View Slide

What is "knife cookbook upload"?
• Several knife commands correspond to API
endpoints
• The verb indicates what we're doing with the
cookbook
• In this case we upload the cookbook to the server
through the API. It's stored by the server...
• We can show information about it too...

View Slide

Checkpoint
• We have a cookbook named apache
• Our apache cookbook has a default recipe
• It has a template source file, index.html.erb
• We have uploaded the cookbook to the Chef server.

View Slide

Anatomy of a Chef Run

View Slide

opscode@target1:~$ sudo chef-client
Run the chef-client on your test node
[sudo] password for opscode:
Starting Chef Client, version 11.6.0
resolving cookbooks for run list: []
Synchronizing Cookbooks:
Compiling Cookbooks...
[2013-09-23T14:31:38+00:00] WARN: Node target1 has an empty run list.
Converging 0 resources
Chef Client finished, 0 resources updated

View Slide

Wait, why didn't Chef apply the recipe?
• The node has an empty run list
• Let us look at what we have now...

View Slide

$ knife node show target1
View details about the node
Node Name: target1
Environment: _default
FQDN: ip-10-164-9-210.ec2.internal
IP: 54.226.122.99
Run List:
Roles:
Recipes:
Platform: ubuntu 12.04
Tags:

View Slide

$ knife node run list add target1 “recipe[apache]”
Add apache to the run list
target1:
run_list: recipe[apache]

View Slide

resolving cookbooks for run list: ["apache"]
- apache
Recipe: apache::default
* package[apache2] action install
- install version 2.2.22-1ubuntu1 of package apache2
* service[apache2] action start (up to date)
* service[apache2] action enable (up to date)
* template[/var/www/index.html] action create
- update content in file /var/www/index.html from 94850c to 463a6b
--- /var/www/index.html 2013-09-23 14:34:24.378187238 +0000
+++ /tmp/chef-rendered-template20130923-3442-1gji5jy 2013-09-23 14:34:28.174187236 +0000
@@ -1,4 +1 @@
-It works!
-This is the default web page for this server.
-The web server software is running but no content has been added, yet.
-
+Hello, world

View Slide

Why wasn't the service updated?
• Chef takes idempotent actions on resources to
ensure they are converged to the desired state
• The apache2 package on Ubuntu automatically
enables and starts the apache2 service in its post-
install script
• Thus, Chef didn't need to do anything

View Slide

If we stopped apache2...
$ sudo /etc/init.d/apache2 stop
* Stopping web server apache2
apache2: [ OK ]
$ sudo chef-client
....
* service[apache2] action start
- start service service[apache2]

View Slide

build node authenticate
sync
cookbooks
load
cookbooks
converge
node.save
notiﬁcation
handlers
exception
Yes
No
chef-client
success?
expanded run list
(recipes)
Ohai!
node_name
platform
platform_version

View Slide

Processing Recipes is Two-Phase
• Chef processes recipes in two phases during
"convergence"
• Evaluate all the Ruby code in the recipe, looking
for "Chef Resources"
• Execute the providers' actions to put each resource
in the declared state

View Slide

Node Attributes

View Slide

The Chef Node Object
• Nodes are the objects that you manage with Chef
• They have a few different properties
• attributes
• run_list
• chef_environment

View Slide

The Chef Node Object
• In client/server Chef, the Chef Server stores node
object data
• It becomes searchable through the API with knife
and with recipes
• Some of the data comes from ohai, which takes an
inventory of the system and emits JSON data
• You can add data to the node through attributes in
cookbooks, roles, directly on a node, etc

View Slide

Node Attributes Syntax
• Node attributes are hashes (of hashes)
• Reference hash keys with square brackets and keys
as strings
•node["hostname"]
•node["kernel"]["machine"]

View Slide

OPEN IN EDITOR:
SAVE FILE!
Hello, world!
My name is <%= node['hostname'] %>
Update the home page template

View Slide

Upload the cookbook

View Slide

- apache
* package[apache2] action install (up to date)
- update content in file /var/www/index.html from 463a6b to b50d3b
+++ /tmp/chef-rendered-template20130923-4337-1csr3dp 2013-09-23 14:39:56.426187239 +0000
@@ -1 +1,2 @@
-Hello, world
+Hello, world!
+My name is ip-10-164-9-210

View Slide

opscode@target1:~$ ohai
Run Ohai on the node
{
"languages": {
"ruby": {
},
"perl": {
"version": "5.14.2",
"archname": "x86_64-linux-gnu-thread-multi"
},
"python": {
"version": "2.7.3",
"builddate": "Aug 1 2012, 05:14:39"
}
},

View Slide

opscode@target1:~$ ohai hostname
Run Ohai on the node
[
"ip-10-164-9-210"
]

View Slide

$ knife node show target1
Show the node object
Node Name: target1
IP: 54.226.122.99
Run List: recipe[apache]
Roles:
Recipes: apache
Tags:

View Slide

$ knife node show target1 -a hostname
Show specific node attribute
target1:
hostname: ip-10-164-9-210

View Slide

Cookbook Attributes

View Slide

Objectives
• Set node attributes from a cookbook
• Understand node attribute precedence

View Slide

OPEN IN EDITOR:
SAVE FILE!
cookbooks/apache/attributes/default.rb
default["apache"]["greeting"] = "World"
Create an attributes file

View Slide

Cookbook Attributes & Precedence
• Always set default node attributes in your
cookbooks' attributes files
• Use "sane" defaults - no surprises
• You can use attributes in roles to set new values
• Roles take precedence over cookbook settings
• When a value must be set to a certain value, use
override, but use this sparingly
• You can’t override ohai's (automatic) attributes!

View Slide

OPEN IN EDITOR:
SAVE FILE!
Hello, <%= node['apache']['greeting'] %>!
Update the home page template

View Slide

Upload the cookbook

View Slide

- apache
- update content in file /var/www/index.html from b50d3b to 764f54
+++ /tmp/chef-rendered-template20130923-4826-1me21yb 2013-09-23 14:54:37.974187238 +0000
@@ -1,2 +1,2 @@
-Hello, world!
+Hello, World!
My name is ip-10-164-9-210

View Slide

Checkpoint
• We have a node attribute
• node["apache"]["greeting"]
• We've updated the index.html template to use this
attribute.
• This will be used again soon!

View Slide

Data Bags

View Slide

Objectives
• Use Data Bags for data-driven recipes
• Use multiple recipes for a node's run list
• Control execution of arbitrary commands with Chef's
resource conditionals

View Slide

Data Bags are generic stores of information
• Data bags are generic, arbitrary stores of information
about the infrastructure.
• Data Bag Items are JSON data
• Our apache cookbook provides a good baseline
• We'll drive site-specific virtual hosts with data bags

View Slide

$ mkdir -p data_bags/vhosts
Create a directory for Data Bags

View Slide

OPEN IN EDITOR:
SAVE FILE!
data_bags/vhosts/bears.json
{
"id" : "bears",
"port" : 80
}
Add a Data Bag Item

View Slide

OPEN IN EDITOR:
SAVE FILE!
data_bags/vhosts/clowns.json
{
"id" : "clowns",
"port" : 81
}
Add a Data Bag Item

View Slide

$ knife upload data_bags/vhosts
Upload the data bags
Created data_bags/vhosts
Created data_bags/vhosts/bears.json
Created data_bags/vhosts/clowns.json

View Slide

OPEN IN EDITOR:
SAVE FILE!
cookbooks/apache/recipes/default.rb
action [:start, :enable]
end
execute "a2dissite default" do
only_if do
File.symlink?("/etc/apache2/sites-enabled/000-default")
end
end
Disable the default apache site

View Slide

mode "0644"
end
OPEN IN EDITOR: /cookbooks/apache/recipes/default.rb
SAVE FILE!
Delete the template resource
• Remove this resource from the recipe

View Slide

$ knife diff cookbooks/apache
Diff the cookbook
--- cookbooks/apache/recipes/default.rb 2013-09-23 11:46:22.000000000 -0400
+++ cookbooks/apache/recipes/default.rb 2013-09-23 11:46:22.000000000 -0400
@@ -14,7 +14,9 @@
action [:start, :enable]
end
-template "/var/www/index.html" do
- source "index.html.erb"
- mode "0644"
-end
+execute "a2dissite default" do
+ only_if do
+ File.symlink?("/etc/apache2/sites-enabled/000-default")
+ end
+ notifies :restart, "service[apache2]"
+end

View Slide

Upload the cookbook

View Slide

Diff the cookbook

View Slide

A new recipe for virtual hosts
• We'll create an apache::vhosts recipe to manage
the virtual hosts we created in data bag items
• There's a number of new things to talk about in this
recipe
• We'll take this nice and slow :)

View Slide

OPEN IN EDITOR:
SAVE FILE!
cookbooks/apache/recipes/vhosts.rb
data_bag("vhosts").each do |site|
site_data = data_bag_item("vhosts", site)
site_name = site_data["id"]
document_root = "/srv/apache/#{site_name}"
end
Create a vhosts recipe

View Slide

OPEN IN EDITOR:
SAVE FILE!
template "/etc/apache2/sites-available/#{site_name}" do
source "custom-vhost.erb"
mode "0644"
variables(
:document_root => document_root,
:port => site_data["port"]
)
end
end
Add a Virtual Hosts Configuration Template

View Slide

OPEN IN EDITOR:
SAVE FILE!
end
execute "a2ensite #{site_name}" do
not_if do
::File.symlink?("/etc/apache2/sites-enabled/#{site_name}")
end
end
end
Enable the Virtual Hosts

View Slide

not_if and only_if
• The not_if parameter causes the resource’s actions
to be taken only if its argument returns false
• The only_if parameter is the opposite of not_if - the
actions are taken only if the arguments return true
• Both not_if and only_if are part of Chef (resources),
not part of Ruby

View Slide

not_if and only_if
• not_if and only_if parameters take either a string, or
a Ruby block argument (do..end or {..})
• When the argument is a string, Chef evaluates it as
a shell command to run.
• When the argument is a Ruby block, Chef
evaluates it as Ruby code to execute.
• This is the equivalent to the code we wrote:
not_if "test -L /etc/apache2/sites-enabled/#{site_name}"

View Slide

OPEN IN EDITOR:
SAVE FILE!
end
directory document_root do
mode "0755"
recursive true
end
end
Add a directory resource

View Slide

OPEN IN EDITOR:
SAVE FILE!
end
template "#{document_root}/index.html" do
mode "0644"
variables(
:site_name => site_name,
:port => site_data["port"]
)
end
end
Index for each vhost
https://gist.github.com/6673845

View Slide

OPEN IN EDITOR:
SAVE FILE!
cookbooks/apache/templates/default/custom-vhosts.erb
<% if @port != 80 -%>
Listen <%= @port %>
<% end -%>
>
ServerAdmin webmaster@localhost
DocumentRoot <%= @document_root %>

Options FollowSymLinks
AllowOverride None

>
Options Indexes FollowSymLinks MultiViews
AllowOverride None
Order allow,deny
allow from all

Index for each vhost
https://gist.github.com/2866454

View Slide

OPEN IN EDITOR:
SAVE FILE!
Hello, <%= node['apache']['greeting'] %>!
We love <%= @site_name %>
Served from <%= node['ipaddress'] %>:<%= @port %>
Update the index.html template

View Slide

Diff the cookbook
diff --knife cookbooks/apache/recipes/default.rb cookbooks/apache/recipes/default.rb
--- cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400
+++ cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400
@@ -19,4 +19,4 @@
File.symlink?("/etc/apache2/sites-enabled/000-default")
end
-end
\ No newline at end of file
+end
diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/recipes/vhosts.rb
new file
--- /dev/null 2013-09-23 12:04:18.000000000 -0400
+++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:04:18.000000000 -0400
@@ -0,0 +1,35 @@
+data_bag("vhosts").each do |site|

View Slide

Upload the cookbook

View Slide

- apache
* execute[a2dissite default] action run
- execute a2dissite default
* service[apache2] action restart
- restart service service[apache2]

View Slide

Oops!

View Slide

$ knife node run list add target1 "recipe[apache::vhosts]"
Update the run list
target1:
run_list:
recipe[apache]
recipe[apache::vhosts]

View Slide

resolving cookbooks for run list: ["apache", "apache::vhosts"]
- apache
* execute[a2dissite default] action run (skipped due to only_if)
Recipe: apache::vhosts
* template[/etc/apache2/sites-available/bears] action create
- create new file /etc/apache2/sites-available/bears
- update content in file /etc/apache2/sites-available/bears from none to bf6a12
--- /etc/apache2/sites-available/bears 2013-09-23 16:11:07.130187239 +0000
+++ /tmp/chef-rendered-template20130923-5699-1w4prah 2013-09-23 16:11:07.130187239 +0000
@@ -0,0 +1 @@

View Slide

Think about what we just did...
• We had two virtual hosts...
• But we could arbitrarily add more...
• Tigers on port 82, Lions on port 83, oh my!

View Slide

Checkpoint
• Our cookbook has two recipes, default and vhosts
• Additional data bags can be added, expanding our
Virtual Hosting empire!

View Slide

Roles

View Slide

What is a role?
• So far, we’ve been just adding recipes directly to our
single node's run list
• But that’s not how infrastructure works - think about
how you refer to servers
• "It’s a web server"
• "It’s a database server"
• or, "It's a database-master server"...
• "It’s a monitoring server"

View Slide

How you use roles
• Roles allow you to conveniently encapsulate the run
lists and attributes required for a server to "be" what
you already think it is
• In practice, roles make it easy to configure many
nodes identically without repeating yourself each
time
• Roles are a first class API primitive on the Chef
Server

View Slide

Best Practice
• Like data bags, you have options for creating roles
• The best practice is that all of your roles live in the
roles directory of your chef-repo
• They can be created via the API and knife, but
having them in source control gives you the history
of changes

View Slide

OPEN IN EDITOR:
SAVE FILE!
roles/webserver.json
{
"name" : "webserver",
"default_attributes" : {
"apache" : {
"greeting" : "Rocky Mountain Ruby"
}
},
"run_list" : [
"recipe[apache]",
"recipe[apache::vhosts]"
]
}
Create a webserver role

View Slide

Components of a role
• Roles must have a name
• Roles may have a description
• Roles may have a run_list, just like a node
• Roles may set node attributes
• default_attributes
• override_attributes
http://docs.opscode.com/essentials_roles.html

View Slide

$ knife upload roles/webserver.json
Upload the role
Created roles/webserver.json

View Slide

$ knife node run list remove target1 "recipe[apache::vhosts]"
Remove the vhosts recipe
target1:
run_list: recipe[apache]

View Slide

$ knife node run list remove target1 "recipe[apache]"
Remove the apache recipe
target1:
run_list:

View Slide

$ knife node run list add target1 "role[webserver]"
Add the webserver role
target1:
run_list: role[webserver]

View Slide

Run chef-client
* template[/srv/apache/bears/index.html] action create
- update content in file /srv/apache/bears/index.html from 509865 to bee814
--- /srv/apache/bears/index.html 2013-09-23 16:11:07.398187242 +0000
+++ /tmp/chef-rendered-template20130923-6418-om7gju 2013-09-23 16:29:27.778187238 +0000
@@ -1,4 +1,4 @@
-Hello, World!
+Hello, Rocky Mountain Ruby!
My name is ip-10-164-9-210
We love bears
Served from 10.164.9.210:80
* template[/etc/apache2/sites-available/clowns] action create (up to date)
* execute[a2ensite clowns] action run (skipped due to not_if)
* directory[/srv/apache/clowns] action create (up to date)
* template[/srv/apache/clowns/index.html] action create
- update content in file /srv/apache/clowns/index.html from 2055c3 to 2290a0
--- /srv/apache/clowns/index.html 2013-09-23 16:11:07.654187238 +0000
+++ /tmp/chef-rendered-template20130923-6418-j7lsw6 2013-09-23 16:29:27.862187239 +0000
@@ -1,4 +1,4 @@

View Slide

Attributes can be set multiple places
• This is for flexibility
• Set a "sane default" that will be used in a cookbook
• Override it easily in a role (higher priority)
• In all, there are 15 places where attributes come from
(!!)
• In practice, you'll use 2-3 most of the time.
• The others are there when you need them.
• http://docs.opscode.com/chef_overview_attributes.html

View Slide

Data Bags Are Not Attributes
• Important: data bag items are not attributes
• Data bags are a separate API end point
• Data bags are not tied to a specific node or role in
the infrastructure
• Not even necessarily tied to anything, just data you
want to store

View Slide

Checkpoint
• We now have a webserver role.
• We could apply just this role on more nodes to scale
out our Virtual Hosting service.
• Roles are a great way to assign attributes for
specific purposes

View Slide

Search

View Slide

Chef's Search Feature
• Search ties together the infrastructure topology
• We can now search the Chef Server for all the
"webserver" nodes
• This is relevant for a variety of reasons
• Load balancing several front ends (search from
nginx, or haproxy for example)
• Monitoring HTTP (search from nagios cookbook)
• Graphing traffic (search from munin cookbook)

View Slide

However, we only have one node...
• We don't have an environment that is really setup for
doing search
• But we will talk about what kind of search we do with
the command-line, or in a recipe

View Slide

$ knife search node "role:webserver"
Search for webservers with knife
1 items found
Node Name: target1
IP: 54.226.122.99
Run List: role[webserver]
Roles: webserver
Recipes: apache, apache::vhosts
Tags:

View Slide

Search from a recipe
webservers = search(:node, "role:webserver")
# or
search(:node, "role:webserver").each do |web|
#things do do on all web servers...
end

View Slide

Available search indexes
• The Chef Server indexes JSON data for the major
API endpoints:
• node
• client
• environment
• role
• All data bags are also indexed for search...
• vhosts (our example)

View Slide

$ knife search vhosts "port:81"
Search for webservers with knife
1 items found
chef_type: data_bag_item
data_bag: vhosts
id: clowns
port: 81

View Slide

Replace Data Bag Item
Lookup with Search

View Slide

OPEN IN EDITOR:
SAVE FILE!
search(:vhosts).each do |site_data|
Refactor the vhosts recipe

View Slide

View the diff
diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/
recipes/vhosts.rb
--- cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000
-0400
+++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000
-0400
@@ -1,5 +1,4 @@
-data_bag("vhosts").each do |site|
- site_data = data_bag_item("vhosts", site)
+search(:vhosts).each do |site_data|

View Slide

Upload the cookbook

View Slide

Run chef-client
resolving cookbooks for run list: ["apache", "apache::vhosts"]
- apache
* execute[a2dissite default] action run (skipped due to only_if)
Recipe: apache::vhosts
* template[/etc/apache2/sites-available/bears] action create (up to date)
* execute[a2ensite bears] action run (skipped due to not_if)
* directory[/srv/apache/bears] action create (up to date)
* template[/srv/apache/bears/index.html] action create (up to date)
* template[/etc/apache2/sites-available/clowns] action create (up to date)
* execute[a2ensite clowns] action run (skipped due to not_if)
* directory[/srv/apache/clowns] action create (up to date)
* template[/srv/apache/clowns/index.html] action create (up to date)

View Slide

Hands on Recap

View Slide

Our cookbook...
• Chef cookbook "apache" with two recipes:
• default (manages apache package and service)
• vhosts (iterates over data bags and renders vhost
configuration)
• However, Opscode publishes an "apache2"
cookbook that manages much much more, including
all apache2 configuration, modules, sites Debian
style w/ a2ensite/a2enmod, etc

View Slide

Our role
• webserver role in JSON
• Sets a default attribute
• Sets a run list
• Uploaded with knife upload

View Slide

Our data bag
• The vhosts data bag serves as an example
• Directory structure follows the data bag API end
point and "knife upload" makes it easy to upload
everything
• Create additional vhosts to see how this is
dynamically expanded easily

View Slide

Chef Server
• Hosted Enterprise Chef
• Free to manage up to five nodes

View Slide

Using A Different Chef Server
• Configure knife by modifying the chef_server_url
and the node_name values in ./chef/knife.rb
• Get the validation client key from the Chef Server
(this differs by implementation):
• Open Source:
•/etc/chef-server/chef-validator.pem
• Hosted Chef / Private Chef
• Download after creating an organization

View Slide

$ cat .chef/knife.rb
The knife.rb configuration
# See http://docs.opscode.com/config_rb_knife.html for more information on knife
configuration options
current_dir = File.dirname(__FILE__)
log_level :info
log_location STDOUT
node_name "nathenharveyrmr1"
client_key "#{current_dir}/nathenharveyrmr1.pem"
validation_client_name "nhrmr1-validator"
validation_key "#{current_dir}/nhrmr1-validator.pem"
chef_server_url "https://api.opscode.com/organizations/nhrmr1"
cache_type 'BasicFile'
cache_options( :path => "#{ENV['HOME']}/.chef/checksums" )
cookbook_path ["#{current_dir}/../cookbooks"]

View Slide

Chef Resources

View Slide

Chef Resources (Core Chef)
• Chef client comes with 24+ different resources
• Packages, files, services, users, symlinks, registry
keys, and more
• Each resource has one or more providers
• Some resources have platform-specific providers
(e.g., package, service, user, group)
http://docs.opscode.com/resource.html

View Slide

Managing files and directories
• file
• cookbook_file
• remote_file
• template
• directory
• remote_directory
• link

View Slide

Managing packages
• package
• apt_package
• chef_gem
• dpkg_package
• easy_install_package
• freebsd_package
• gem_package
• ips_package
• macports_package
• pacman_package
• portage_package
• rpm_package
• smartos_package
• solaris_package
• yum_package

View Slide

RubyGem packages
• chef_gem - install a RubyGem into Chef's Ruby
environment to be used in a Chef recipe
• gem_package - install a RubyGem to be used by
the system or an application

View Slide

Services
• service is used to manage services using the
common init systems available
• Each platform has it's own provider
• arch, debian, freebsd, gentoo, "init", insserv
• invokercd, macosx (launchd), redhat, "simple"
• solaris (SMF), upstart, windows

View Slide

Other Chef Resources
• cron
• deploy (revision,
timestamped)
• env (windows)
• group
• ifconfig (RHEL)
• log
• mdadm
• mount
• ohai
• registry_key (windows)
• route
• scm (git, subversion)
• user

View Slide

There when you need them...
• execute (we used this)
• script (bash, perl, python, csh, ruby interpreters)
• windows_script (batch, powershell) - Chef 11.6.0!
• ruby_block

View Slide

Cookbooks to Know
About

View Slide

community.opscode.com
• apache2, nginx
• ark
• build-essential
• chef-client
• chruby
• cron
• line
• java
• jenkins
• minitest-handler
• mysql, postgresql
• openssh
• omnibus_updater
• partial_search
• runit
• whitelist-node-attrs

View Slide

Opscode Cookbooks with Chef Resources
• Opscode has several cookbooks that include new custom Chef
Resources
• apt (apt_repository)
• aws (aws_ebs_volume, aws_elastic_ip, and more)
• yum (yum_repository)
• windows (windows_package, windows_feature, and more)
• homebrew (homebrew_package)
• runit (runit_service)
• many more!
http://docs.opscode.com/chef/lwrps_custom.html

View Slide

Getting Cookbooks from the Community Site
• Knife commands:
•knife cookbook site download
• knife cookbook site install (integrates
with git)
• Dependency resolvers (a la Ruby's bundler):
•librarian-chef
•berkshelf

View Slide

Tools to be aware of...

View Slide

Knife Plugins
• http://docs.opscode.com/community_plugin_knife.html
• Cloud plugins (ec2, openstack, rackspace, google
azure, hpcloud, cloudstack, eucalyptus, and more)
• knife-server
• knife-solo
• knife-preflight
• knife-essentials
• https://github.com/jkeiser/knife-essentials

View Slide

Chef Handlers
• http://docs.opscode.com/community_plugin_report_handler.html
• IRC
• Campfire
• HipChat
• DataDog
• Splunk Storm
• Graylog2
• Graphite

View Slide

Workflow Helpers
• Berkshelf: berkshelf.com
• Librarian-chef: github.com/applicationsonline/
librarian-chef
• Knife Spork: github.com/jonlives/knife-spork
• Vagrant: vagrantup.com (also a testing tool)

View Slide

Cookbook Testing
• Vagrant: vagrantup.com
• Test Kitchen: github.com/opscode/test-kitchen
• Foodcritic: acrmp.github.io/foodcritic
• ChefSpec: acrmp.github.io/chefspec/
• RSpec: rspec.info
• minitest-chef: github.com/calavera/minitest-chef-
handler
• cookbook: minitest-handler

View Slide

Thank you!
Questions & Answers
[email protected]
[email protected]

View Slide

Intro to Chef - Rocky Mountain Ruby

Intro to Chef - Rocky Mountain Ruby

Nathen Harvey

More Decks by Nathen Harvey

Other Decks in Technology

Featured

Transcript