Slide 1

Slide 1 text

System Management with Chef Tutorial: OSCON 2013 Joshua Timberman @jtimberman James Casey @jamesc_000 [email protected] github.com/jtimberman [email protected] github.com/jamesc Tuesday, July 23, 13

Slide 2

Slide 2 text

Introductions Tuesday, July 23, 13

Slide 3

Slide 3 text

Who Are We? Tuesday, July 23, 13

Slide 4

Slide 4 text

Who Are You? • System administrator? • Software developer/engineer? • Used Chef before? Tuesday, July 23, 13

Slide 5

Slide 5 text

Tutorial Objectives • Understand Chef and the problem it solves • Understand the components of Chef and how they fit together • Know just enough Ruby to get started • Be able to create a new cookbook • Understand the core primitives of Chef recipes • Get accustomed to the common workflows used by experienced Chef users/developers Tuesday, July 23, 13

Slide 6

Slide 6 text

Expectations • This is (barely) a half-day workshop, not a comprehensive course. • We will do some hands on exercises. • You should get a taste for automating with Chef. • You should have received instructions for prerequisites prior to this tutorial • Chef works and runs on Windows, but the exercises for this tutorial are Linux-based Tuesday, July 23, 13

Slide 7

Slide 7 text

We have a problem... Tuesday, July 23, 13

Slide 8

Slide 8 text

http://www.ﬂickr.com/photos/michaelheiss/3090102907/ Complexity Tuesday, July 23, 13

Slide 9

Slide 9 text

Items of Manipulation (Resources) • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems Tuesday, July 23, 13

Slide 10

Slide 10 text

Application A tale of growth... Tuesday, July 23, 13

Slide 11

Slide 11 text

Application Application Database Add a database Tuesday, July 23, 13

Slide 12

Slide 12 text

Application App Databases Make database redundant Tuesday, July 23, 13

Slide 13

Slide 13 text

App Servers App Databases Application server redundancy Tuesday, July 23, 13

Slide 14

Slide 14 text

App LB App Servers App Databases Add a load balancer Tuesday, July 23, 13

Slide 15

Slide 15 text

App LBs App Servers App Databases Webscale! Tuesday, July 23, 13

Slide 16

Slide 16 text

App LBs App Servers App DB Cache App DBs Now we need a caching layer Tuesday, July 23, 13

Slide 17

Slide 17 text

App LBs App Servers App DB Cache App DBs Infrastructure Has a Topology Tuesday, July 23, 13

Slide 18

Slide 18 text

Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your's Is a Snowflake Tuesday, July 23, 13

Slide 19

Slide 19 text

App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring?? Tuesday, July 23, 13

Slide 20

Slide 20 text

Tuesday, July 23, 13

Slide 21

Slide 21 text

The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults http://www.ﬂickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/ Tuesday, July 23, 13

Slide 22

Slide 22 text

The Chef Tool(s) • ohai • chef-client • chef-shell • knife • The Ruby language Omnibus - Full Stack Native Packages Tuesday, July 23, 13

Slide 23

Slide 23 text

The Chef API • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.ﬂickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/ Tuesday, July 23, 13

Slide 24

Slide 24 text

The Chef Community • Chef is Open Source! • Apache License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com Tuesday, July 23, 13

Slide 25

Slide 25 text

How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg Tuesday, July 23, 13

Slide 26

Slide 26 text

Chef Enables Infrastructure as Code • Resources • Recipes • Cookbooks and Roles • Source Code http://www.ﬂickr.com/photos/louisb/4555295187/ Tuesday, July 23, 13

Slide 27

Slide 27 text

Resources and Recipes Tuesday, July 23, 13

Slide 28

Slide 28 text

package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Declarative abstraction to system resources Tuesday, July 23, 13

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Resources describe what Not how. Tuesday, July 23, 13

Slide 31

Slide 31 text

Resources take action through Providers Tuesday, July 23, 13

Slide 32

Slide 32 text

Providers perform the how Tuesday, July 23, 13

Slide 33

Slide 33 text

def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example Tuesday, July 23, 13

Slide 34

Slide 34 text

Package Resource package "git" { yum install git apt-get install git pacman sync git pkg_add -r git Providers are determined by node's platform Tuesday, July 23, 13

Slide 35

Slide 35 text

Recipes are collections of resources package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Tuesday, July 23, 13

Slide 36

Slide 36 text

Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain related components • Files, Templates, Libraries • A cookbook is responsible for configuring a single thing, e.g. • apache2 • postgresql • A recipe is responsible for a component, e.g. • api • server • client Tuesday, July 23, 13

Slide 37

Slide 37 text

Roles • Roles describe nodes • Roles indicate functionality • Roles have a first class API primitive Tuesday, July 23, 13

Slide 38

Slide 38 text

http://www.ﬂickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search Tuesday, July 23, 13

Slide 39

Slide 39 text

Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.ﬂickr.com/photos/kathycsus/2686772625 Tuesday, July 23, 13

Slide 40

Slide 40 text

Hands On Exercises Tuesday, July 23, 13

Slide 41

Slide 41 text

$ a command you type or file you open Exercise Slides Look Like this: Output from the command, or content that goes into the file Tuesday, July 23, 13

Slide 42

Slide 42 text

$ It will be revealed here Hidden Commands Test Your Memory After you have read the contents and had a chance to remember the command to type... Tuesday, July 23, 13

Slide 43

Slide 43 text

Hands on Exercises • The majority of the hands on exercises will be related to creating an "apache" cookbook. • The goals are to learn elements of Chef, not to learn Apache. We're going to do things the hard way, by typing in a lot of code. We have some gists with large sections of code already available to reduce what you need to type in certain places. • Errors and typos are good, as they will help students learn how to resolve errors. Tuesday, July 23, 13

Slide 44

Slide 44 text

Our Environment • Virtual Machine image (virtualbox, vmware) • Ubuntu 12.04 (32 bit) • Login: • user - vagrant • password - vagrant Tuesday, July 23, 13

Slide 45

Slide 45 text

Prerequisite Verification Tuesday, July 23, 13

Slide 46

Slide 46 text

Objectives • Verify all the prerequisites are met to complete all the exercises. • Introduce the students to the Chef Repository • Start up the Chef Server • Introduce the knife command-line tool Tuesday, July 23, 13

Slide 47

Slide 47 text

Verify Requirements • Virtual Machine should be imported into VirtualBox OR VMware (Fusion/Workstation/Player). • Virtual Machine should be able to reach the internet • Chef, Git, Vim, and Emacs are installed • chef-client and knife commands • git, vim, emacs commands • chef-zero gem is installed Tuesday, July 23, 13

Slide 48

Slide 48 text

$ ssh [email protected] Log into the VM [email protected]'s password: vagrant Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23- generic i686) * Documentation: https://help.ubuntu.com/ Last login: Thu Jul 18 19:29:26 2013 vagrant@packer-virtualbox:~$ Tuesday, July 23, 13

Slide 49

Slide 49 text

Verification commands (ssh to VM) $ ping -c 3 github.com $ which knife $ which chef-client $ which git $ which vim $ which emacs $ /opt/chef/embedded/bin/chef-zero -h Tuesday, July 23, 13

Slide 50

Slide 50 text

$ git clone git://github.com/jtimberman/oscon2013-chef-repo Clone the Chef Repository Cloning into 'oscon2013-chef-repo'... remote: Counting objects: 13, done. remote: Compressing objects: 100% (9/9), done. remote: Total 13 (delta 2), reused 13 (delta 2) Receiving objects: 100% (13/13), done. Resolving deltas: 100% (2/2), done. Tuesday, July 23, 13

Slide 51

Slide 51 text

$ ls oscon2013-chef-repo Tour of the Chef Repository cookbooks/ data_bags/ README.md roles/ Tuesday, July 23, 13

Slide 52

Slide 52 text

$ /opt/chef/embedded/bin/chef-zero -l debug Start up the Chef Server >> Starting Chef Zero (v1.5.3)... >> Puma (v1.6.3) is listening at http://127.0.0.1:8889 >> Press CTRL+C to stop $ /opt/chef/embedded/bin/chef-zero -d Tuesday, July 23, 13

Slide 53

Slide 53 text

The Flavors of Chef • Open Source Chef Server • Opscode Hosted Chef • Opscode Private Chef • Chef Solo • Chef Zero (our flavor today) • Alternate implementations of the server API Tuesday, July 23, 13

Slide 54

Slide 54 text

Chef Server Components • Runs on HTTPS (port 443) • API Service • WebUI Service • Search Index (SOLR) • Message Queue (RabbitMQ) • Database (PostgreSQL) • Cookbook Service (Bookshelf/ filesystem) Tuesday, July 23, 13

Slide 55

Slide 55 text

Why not Open Source Chef Server? • Open Source Chef Server package is > 200M • Numerous moving parts • We have limited time in a 3 hour tutorial, and would rather focus on the steeper part of the learning curve • You wouldn't learn sendmail just to send email... Tuesday, July 23, 13

Slide 56

Slide 56 text

After tutorial... • Download and install Open Source Chef Server • Use the VM we provided here today • opscode.com/chef/install • (Please don't do this during the tutorial!) Tuesday, July 23, 13

Slide 57

Slide 57 text

What is Chef Zero? • Simple reference implementation of the Chef Server API • Everything runs in memory (fast!) • Everything runs in memory (no persistence!) • No authentication (easy/simple for testing use) • Simple installation (gem install chef-zero) • Uses "puma" web server (fast, cross-platform) • Runs on port 8889 (chef-server default is 443) Tuesday, July 23, 13

Slide 58

Slide 58 text

The chef-zero server is ephemeral • Everything is stored in memory • If the process crashes or is terminated, it all must be uploaded again • We'll cover how to do this in later exercises Tuesday, July 23, 13

Slide 59

Slide 59 text

$ cd oscon2013-chef-repo Work from the Chef Repository • We're going to work from the Chef Repository • All knife commands are run here • All paths are relative to this directory Tuesday, July 23, 13

Slide 60

Slide 60 text

$ knife client list Verify Access to Chef Server chef-validator chef-webui Tuesday, July 23, 13

Slide 61

Slide 61 text

Overview of our environment Tuesday, July 23, 13

Slide 62

Slide 62 text

Tuesday, July 23, 13

Slide 63

Slide 63 text

Chef Server • We're using Chef Zero running on "localhost" • A Chef Server could be Opscode Hosted or Private, Open Source • The point is we're accessing it over network accessible API service Tuesday, July 23, 13

Slide 64

Slide 64 text

Managed Nodes • Many nodes can be managed by a Chef Server • We're going to manage a single node • It happens to be on the same system as the Chef Zero server • Again, we're accessing the server via the API Tuesday, July 23, 13

Slide 65

Slide 65 text

Workstation • We write Chef code on a local workstation like a laptop • We upload that code to the Chef Server • Again, using the API Tuesday, July 23, 13

Slide 66

Slide 66 text

A Little Ruby Tuesday, July 23, 13

Slide 67

Slide 67 text

Chef uses a Ruby DSL • You don’t need a lot of Ruby • When you do need something more advanced, it’s already there • Allows us to easily borrow intermediate to advanced features like testing when we want it, too • Most of the syntax is familiar if you’ve done scripting in other languages, though Tuesday, July 23, 13

Slide 68

Slide 68 text

Ruby is Object Oriented • Everything in Ruby is an Object • Call a method on an object with '.': •File.open("/etc/hosts") •[1, 3, 5].each • Ruby is dynamically typed, and it is "duck typed" • "If it walks like a duck, and it talks like a duck..." • Newlines end a statement, but multiple statements can be written on a single line separated by ; Tuesday, July 23, 13

Slide 69

Slide 69 text

Ruby Basics: IRB • IRB is "interactive Ruby" • It is a "REPL" or "Read, Eval, Print, Loop" • Installed w/ Ruby package • Installed w/ Chef Omnibus Package • /opt/chef/embedded/bin/irb Tuesday, July 23, 13

Slide 70

Slide 70 text

Ruby Syntax Basics • Assignment: •local_variable = "Some Value" • Strings: •"This is a string" •'So is this' • Interpolation: • Only with double-quotes •"this is #{local_variable}" Tuesday, July 23, 13

Slide 71

Slide 71 text

Ruby Basics: Numbers • Numbers are unquoted literals • Integers •24 • Floats •4.2 • Other types •0x12F • Math(s)! •3 + 4; 8 * 73; (2+7)/3 Tuesday, July 23, 13

Slide 72

Slide 72 text

Ruby Basics: symbols • Represent names and strings • Only one copy exists in memory • Preceding colon •:my_symbol • Often used as hash keys •{:vim => "awesome"} • Can be used in attributes, just be consistent Tuesday, July 23, 13

Slide 73

Slide 73 text

Symbols and Strings irb> "string".object_id => 70221927110640 irb> "string".object_id => 70221927077000 irb> "string".object_id => 70221927058480 irb> :string.object_id => 156968 irb> :string.object_id => 156968 Tuesday, July 23, 13

Slide 74

Slide 74 text

Ruby Basics: Flow Control • Conditional Logic •if, case, unless • Structures close with end keyword • Equality operators •<, >, <=, >= • Iterators •.each • True / False •true, false, nil OMG 0 is true!! Tuesday, July 23, 13

Slide 75

Slide 75 text

Ruby Basics: Data Structures • Arrays • ["vim", "emacs", "nano", "notepad++"] • Hashes • {"vim" => "awesome, installed by default", "emacs" => "awesome but not installed", "notepad++" => "wat? this isn’t Windows"} • symbols and strings are different keys • {"vim" => "awesome", :vim => "different"} Tuesday, July 23, 13

Slide 76

Slide 76 text

Create an apache cookbook Tuesday, July 23, 13

Slide 77

Slide 77 text

Objectives • Understand what a cookbook is • Know how to create a new cookbook • Understand what a recipe is • Understand how to use the package, service, and template resources • Know how to upload a cookbook to the Chef Server • Understand what a run list is, and how to set it • How to read the output of the chef-client run Tuesday, July 23, 13

Slide 78

Slide 78 text

What is a cookbook? • A cookbook is like a "package" for Chef recipes • It contains all the recipes, files, templates, libraries, etc. required to configure a portion of your infrastructure • Typically cookbooks map 1:1 to a piece of software or functionality. • "our tomcat cookbook" • "our zlib cookbook" • "our security_policy cookbook" Tuesday, July 23, 13

Slide 79

Slide 79 text

$ knife cookbook create apache Create the cookbook with knife ** Creating cookbook apache ** Creating README for cookbook: apache ** Creating CHANGELOG for cookbook: apache ** Creating metadata for cookbook: apache Tuesday, July 23, 13

Slide 80

Slide 80 text

$ ls cookbooks/apache What did this create? attributes/ CHANGELOG.md definitions/ files/ libraries/ metadata.rb providers/ README.md recipes/ resources/ templates/ Tuesday, July 23, 13

Slide 81

Slide 81 text

$ cat cookbooks/apache/metadata.rb What is metadata.rb? name 'apache' maintainer 'YOUR_COMPANY_NAME' maintainer_email 'YOUR_EMAIL' license 'All rights reserved' description 'Installs/Configures apache' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0' Tuesday, July 23, 13

Slide 82

Slide 82 text

Cookbook Metadata • Cookbooks are like packages of configuration for the infrastructure • They're artifacts that have a name and a version • Metadata can also have information about the cookbook like its license and maintainer • Cookbooks can depend on other cookbooks, too Tuesday, July 23, 13

Slide 83

Slide 83 text

Other relevant components • We'll use other components of the cookbook throughout the exercises •recipes/default.rb •templates/default/ •attributes/[default.rb] Tuesday, July 23, 13

Slide 84

Slide 84 text

What is a Recipe? • Ordered list of Resources • Ruby Domain-Specific Language (DSL) • Named by the cookbook and the recipe filename • The "default.rb" recipe is referred to by the name of the cookbook (apache) • If we added a recipe to the cookbook named "vhosts.rb" we would refer to it as apache::vhosts Tuesday, July 23, 13

Slide 85

Slide 85 text

cookbooks/apache/recipes/default.rb Add a package resource in the default recipe Tuesday, July 23, 13

Slide 86

Slide 86 text

So the resource we just wrote... • Is a package resource • Whose name is apache2 • With an install action Tuesday, July 23, 13

Slide 87

Slide 87 text

How does the package install? • Resources are declarative - we say what we want to have happen, rather than how • Chef uses what platform the node is running to determine the correct provider for a resource Tuesday, July 23, 13

Slide 88

Slide 88 text

cookbooks/apache/recipes/default.rb Add a service resource Tuesday, July 23, 13

Slide 89

Slide 89 text

The resource we just wrote... • Is a service resource • Whose name is apache2 • With two actions: enable and start • It has a parameter called supports that takes a hash of options as an argument Tuesday, July 23, 13

Slide 90

Slide 90 text

cookbooks/apache/recipes/default.rb Add a template to manage the home page Tuesday, July 23, 13

Slide 91

Slide 91 text

The resource we just wrote... • Is a template resource • Its name is also the destination of the file on the filesystem • Its source is the location of the template file in the cookbook • Its mode sets the permissions on the file • No action! Default is create Tuesday, July 23, 13

Slide 92

Slide 92 text

cookbooks/apache/templates/default/index.html.erb Create the index.html.erb template

Hello, World

Tuesday, July 23, 13

Slide 93

Slide 93 text

$ knife cookbook upload apache Upload the cookbook to the Chef Server Uploading apache [0.1.0] Uploaded 1 cookbook. Tuesday, July 23, 13

Slide 94

Slide 94 text

What is "knife cookbook upload"? • Several knife commands correspond to API endpoints • The verb indicates what we're doing with the cookbook • In this case we upload the cookbook to the server through the API. It's stored by the server... • We can show information about it too... Tuesday, July 23, 13

Slide 95

Slide 95 text

$ knife cookbook show apache Show the cookbook on the Chef Server apache 0.1.0 Tuesday, July 23, 13

Slide 96

Slide 96 text

$ knife cookbook show apache 0.1.0 Show a specific cookbook version Tuesday, July 23, 13

Slide 97

Slide 97 text

Cookbook content on the Chef Server Tuesday, July 23, 13

Slide 98

Slide 98 text

Checkpoint • We have a cookbook named apache • Our apache cookbook has a default recipe • It has a template source file, index.html.erb • We have uploaded the cookbook to the Chef Zero Server. Tuesday, July 23, 13

Slide 99

Slide 99 text

Anatomy of a Chef Run Tuesday, July 23, 13

Slide 100

Slide 100 text

$ knife node list Before we run Chef, list the nodes (no output) Tuesday, July 23, 13

Slide 101

Slide 101 text

Create /etc/chef and validation.pem $ sudo mkdir /etc/chef $ sudo cp .chef/zero.pem /etc/chef/validation.pem Tuesday, July 23, 13

Slide 102

Slide 102 text

$ sudo vi /etc/chef/client.rb Create /etc/chef/client.rb chef_server_url "http://localhost:8889" Tuesday, July 23, 13

Slide 103

Slide 103 text

/etc/chef/ client.pem? /etc/chef/ validation.pem? 401! Request API Client Sign Requests client.pem Yes No No Yes Tuesday, July 23, 13

Slide 104

Slide 104 text

Chef Zero doesn't authenticate • Chef Zero doesn't enforce authentication (other Chef Servers do) • It also doesn't run over HTTPS (other Chef Servers do) • The authentication cycle is still valid, though • The client.pem will still be written out, let's take a look... Tuesday, July 23, 13

Slide 105

Slide 105 text

$ sudo chef-client Now, run Chef Client Starting Chef Client, version 11.4.4 Creating a new client identity for packer-vmware using the validator key. resolving cookbooks for run list: [] Synchronizing Cookbooks: Compiling Cookbooks... [2013-07-06T14:42:35+00:00] WARN: Node packer-vmware has an empty run list. Converging 0 resources Chef Client finished, 0 resources updated Tuesday, July 23, 13

Slide 106

Slide 106 text

Wait, why didn't Chef apply the recipe? • We didn't have a node object on the server • We didn't tell Chef about the node when we started chef-client • Let us look at what we have now... Tuesday, July 23, 13

Slide 107

Slide 107 text

$ ls /etc/chef Look in /etc/chef client.pem client.rb validation.pem Tuesday, July 23, 13

Slide 108

Slide 108 text

$ knife client list Show the client list chef-validator chef-webui packer-vmware packer-virtualbox (Your client should be named -vmware or - virtualbox, depending on the VM image) Tuesday, July 23, 13

Slide 109

Slide 109 text

$ knife client show packer-vmware Show the new client admin: false chef_type: client json_class: Chef::ApiClient name: packer-vmware public_key: -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvW1NgunHiKJqvXNk5kLO 7AkqFY6O0pZL46y9/OiqrCUYdDeaIeskYWKaKIVFdUcqFf/nBao97y35cfsSOkJI ZTQUdk0lEUF/+mdGOAjur/tUIllltWG4xJHI5EBEfvSxV8DO3gUuizIIs591SNsp XCWlLsBCYRRzhGhA60JMF4Y1EEXhVINhMsr2jleNOPCInGxz3sqWY+1KfwyxNjHL woH9Vi6uGCZ1FT0A1i19Kg7j1EQSCEMhcMKZ8vpUpAnmwwqshFHx+UubrDCd1v8H VSdJAHtUqqc3iLTfSRZIjak7J8ZdYXF8chg1wfgkjnpVcTr2lOy3iuLzxReI/9rs iQIDAQAB -----END PUBLIC KEY----- important to note that the server stores the PUBLIC key - think SSH Tuesday, July 23, 13

Slide 110

Slide 110 text

$ knife node list Show the node list again packer-vmware packer-virtualbox (we'll use -vmware throughout the remainder of the slides) Tuesday, July 23, 13

Slide 111

Slide 111 text

$ knife node show packer-vmware Show the new node Node Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags: Tuesday, July 23, 13

Slide 112

Slide 112 text

$ knife node run list add packer-vmware 'recipe[apache]' How do we modify the run list? packer-vmware: run_list: recipe[apache] Tuesday, July 23, 13

Slide 113

Slide 113 text

$ sudo chef-client Run Chef Client again Starting Chef Client, version 11.4.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install - install version 2.2.22-1ubuntu1.3 of package apache2 * service[apache2] action enable (up to date) * service[apache2] action start (up to date) * template[/var/www/index.html] action create - update template[/var/www/index.html] from 94850c to 3668a4 --- /var/www/index.html 2013-07-06 14:51:17.206134567 +0000 +++ /tmp/chef-rendered-template20130706-17703-en6g83 2013-07-06 14:51:17.702126265 +0000 @@ -1,4 +1,2 @@ -

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

- +

Hello, World

+ Chef Client finished, 2 resources updated Tuesday, July 23, 13

Slide 114

Slide 114 text

$ sudo apt-get update Did you get this error? [2013-07-18T21:14:56+00:00] ERROR: Running exception handlers [2013-07-18T21:14:56+00:00] FATAL: Saving node information to / var/chef/cache/failed-run-data.json [2013-07-18T21:14:56+00:00] ERROR: Exception handlers complete Chef Client failed. 0 resources updated [2013-07-18T21:14:56+00:00] FATAL: Stacktrace dumped to /var/chef/ cache/chef-stacktrace.out [2013-07-18T21:14:56+00:00] FATAL: Chef::Exceptions::Exec: package[apache2] (apache::default line 9) had an error: Chef::Exceptions::Exec: apt-get -q -y install apache2=2.2.22-1ubuntu1.3 returned 100, expected 0 Tuesday, July 23, 13

Slide 115

Slide 115 text

The problem... • This is [an unintended] illustration of the problem with "golden images" • The images we're using were created a couple of weeks ago • Ubuntu has released new versions of Apache 2 packages (security update) • The apt cache is stale, so we need to update it • The recommended practice is to put "apt-get update" in a recipe, though we won't do that today Tuesday, July 23, 13

Slide 116

Slide 116 text

It works!

This is the default web page for this server.

The web server software is running but no content has been added, yet.

- +

Hello, World

+ Chef Client finished, 2 resources updated Tuesday, July 23, 13

Slide 117

Slide 117 text

* service[apache2] action enable (up to date) * service[apache2] action start (up to date) Why wasn't the service updated? • Chef takes idempotent actions on resources to ensure they are converged to the desired state • The apache2 package on Ubuntu automatically enables and starts the apache2 service in its post- install script • Thus, Chef didn't need to do anything Tuesday, July 23, 13

Slide 118

Slide 118 text

If we stopped apache2... $ sudo /etc/init.d/apache2 stop * Stopping web server apache2 apache2: [ OK ] $ sudo chef-client .... * service[apache2] action start - start service service[apache2] Tuesday, July 23, 13

Slide 119

Slide 119 text

build node authenticate sync cookbooks load cookbooks converge node.save notiﬁcation handlers exception Yes No chef-client success? expanded run list (recipes) Ohai! node_name platform platform_version Tuesday, July 23, 13

Slide 120

Slide 120 text

Processing Recipes is Two-Phase • Chef processes recipes in two phases during "convergence" • Evaluate all the Ruby code in the recipe, looking for "Chef Resources" • Execute the providers' actions to put each resource in the declared state Tuesday, July 23, 13

Slide 121

Slide 121 text

Node Attributes Tuesday, July 23, 13

Slide 122

Slide 122 text

The Chef Node Object • Nodes are the objects that you manage with Chef • They have a few different properties • attributes • run_list • chef_environment Tuesday, July 23, 13

Slide 123

Slide 123 text

The Chef Node Object • In client/server Chef, the Chef Server stores node object data • It becomes searchable through the API with knife and with recipes • Some of the data comes from ohai, which takes an inventory of the system and emits JSON data • You can add data to the node through attributes in cookbooks, roles, directly on a node, etc Tuesday, July 23, 13

Slide 124

Slide 124 text

Node Attributes Syntax • Node attributes are hashes (of hashes) • Reference hash keys with square brackets and keys as strings •node["hostname"] •node["kernel"]["machine"] Tuesday, July 23, 13

Slide 125

Slide 125 text

cookbooks/apache/templates/default/index.html.erb Add a line to the index.html.erb •"erb" stands for "embedded ruby" Tuesday, July 23, 13

Slide 126

Slide 126 text

erb syntax • Built into the Ruby Standard Library, but there is an separate library (erubis) that performs a bit better • To embed a value within an ERB template: • Start with <%= • Write your Ruby expression - most commonly a node attribute • End with %> • A lot like working in PHP, Mason, etc Tuesday, July 23, 13

Slide 127

Slide 127 text

$ knife cookbook upload apache Upload the new cookbook Uploading apache [0.1.0] Uploaded 1 cookbook. Tuesday, July 23, 13

Slide 128

Slide 128 text

$ sudo chef-client Run Chef Client * template[/var/www/index.html] action create - update template[/var/www/index.html] from 3668a4 to 418b05 --- /var/www/index.html 2013-07-06 14:51:17.758125327 +0000 +++ /tmp/chef-rendered-template20130706-18568-cd45vw 2013-07-06 15:09:56.631395484 +0000 @@ -1,2 +1,2 @@

Hello, World

- +

My name is packer-vmware

Chef Client finished, 1 resources updated Explain how the checksum things work for the content change in the template. Note that we don't get the ERB content, we get the rendered content of the template which comes from the attribute Tuesday, July 23, 13

Slide 129

Slide 129 text

$ ohai Run ohai on the node { "languages": { "ruby": { }, "python": { "version": "2.7.3", "builddate": "Apr 10 2013, 05:46:21" }, "perl": { "version": "5.14.2", "archname": "i686-linux-gnu-thread-multi-64int" } }, "kernel": { "name": "Linux", "release": "3.5.0-23-generic", "version": "#35~precise1-Ubuntu SMP Fri Jan 25 17:15:33 UTC 2013", "machine": "i686" Tuesday, July 23, 13

Slide 130

Slide 130 text

Run ohai with an attribute $ ohai hostname [ "packer-vmware" ] $ ohai ipaddress [ "192.168.21.134" ] Explain that ohai just mainly runs commands on the underlying system for Linux/Unix systems. hostname -f default route from netstat Tuesday, July 23, 13

Slide 131

Slide 131 text

$ knife node show packer-vmware Show the Node object Node Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: recipe[apache] Roles: Recipes: apache Platform: ubuntu 12.04 Tags: Tuesday, July 23, 13

Slide 132

Slide 132 text

Show specific attributes of the node $ knife node show packer-vmware -a hostname packer-vmware: hostname: packer-vmware $ knife node show packer-vmware -a ipaddress packer-vmware: ipaddress: 192.168.21.134 $ knife node show packer-vmware -a platform packer-vmware: platform: ubuntu Tuesday, July 23, 13

Slide 133

Slide 133 text

Cookbook Attributes Tuesday, July 23, 13

Slide 134

Slide 134 text

Objectives • Set node attributes from a cookbook • Understand node attribute precedence Tuesday, July 23, 13

Slide 135

Slide 135 text

cookbooks/apache/attributes/default.rb Create a cookbook attributes file what is default? what are these brackets? what is an equals zomg what are “? Tuesday, July 23, 13

Slide 136

Slide 136 text

Cookbook Attributes & Precedence • Always set default node attributes in your cookbooks' attributes files • Use "sane" defaults - no surprises • You can use attributes in roles to set new values • Roles take precedence over cookbook settings • When a value must be set to a certain value, use override, but use this sparingly • You can’t override ohai's (automatic) attributes! Tuesday, July 23, 13

Slide 137

Slide 137 text

cookbooks/apache/templates/default/index.html.erb Update the template to use this attribute Tuesday, July 23, 13

Slide 138

Slide 138 text

$ knife cookbook upload $ sudo chef-client Upload the cookbook, run Chef Recipe: apache::default * package[apache2] action install (up to date) * template[/var/www/index.html] action create (up to date) * service[apache2] action enable (up to date) * service[apache2] action start (up to date) Chef Client finished, 0 resources updated Why didn't the index.html file change? The attribute is the same value as the previous content Tuesday, July 23, 13

Slide 139

Slide 139 text

Checkpoint • We have a node attribute • node["apache"]["greeting"] • We've updated the index.html template to use this attribute. • This will be used again soon! Tuesday, July 23, 13

Slide 140

Slide 140 text

Data Bags Tuesday, July 23, 13

Slide 141

Slide 141 text

Objectives • Learn how to use Data Bags for data-driven recipes • Use the "knife essentials" plugin commands • Understand using multiple recipes for a node's run list • Control execution of arbitrary commands with Chef's resource conditionals Tuesday, July 23, 13

Slide 142

Slide 142 text

Data Bags are generic stores of information • Data bags are generic, arbitrary stores of information about the infrastructure. • Data Bag Items are JSON data • Our apache cookbook provides a good baseline • We'll drive site-specific virtual hosts with data bags Tuesday, July 23, 13

Slide 143

Slide 143 text

Create the data bag and items $ mkdir data_bags/vhosts $ vi data_bags/vhosts/bears.json $ vi data_bags/vhosts/clowns.json Tuesday, July 23, 13

Slide 144

Slide 144 text

data_bags/vhosts/bears.json Create the bears.json item { "id": "bears", "port": 80 } Numeric Literal! Tuesday, July 23, 13

Slide 145

Slide 145 text

data_bags/vhosts/clowns.json Create the clowns.json item { "id": "clowns", "port": 81 } Numeric Literal! Tuesday, July 23, 13

Slide 146

Slide 146 text

$ knife upload data_bags/vhosts Upload all the data bag items Created remote/data_bags/vhosts/ Created remote/data_bags/vhosts/bears.json Created remote/data_bags/vhosts/clowns.json Tuesday, July 23, 13

Slide 147

Slide 147 text

knife upload • Comes from the "knife essentials" plugin, now built into Chef (11+) • Treat the local chef-repo and the chef server like filesystems • Useful commands like upload, download, show, diff • Easier to remember than all the different object actions • Tab completion! Tuesday, July 23, 13

Slide 148

Slide 148 text

cookbooks/apache/recipes/default.rb Disable the default apache site Remove template resource Tuesday, July 23, 13

Slide 149

Slide 149 text

$ knife diff cookbooks/apache Introducing knife diff diff --knife cookbooks/apache/recipes/default.rb cookbooks/apache/recipes/default.rb --- cookbooks/apache/recipes/default.rb 2013-07-06 16:28:15.136742911 +0000 +++ cookbooks/apache/recipes/default.rb 2013-07-06 16:28:15.136742911 +0000 @@ -16,8 +16,9 @@ action [:enable, :start] end -template "/var/www/index.html" do - source "index.html.erb" - mode "0644" +execute "a2dissite default" do + only_if do + ::File.symlink?("/etc/apache2/sites-enabled/000-default") + end + notifies :restart, "service[apache2]" end Tuesday, July 23, 13

Slide 150

Slide 150 text

$ knife upload cookbooks/apache Upload the cookbook Updated remote/cookbooks/apache Tuesday, July 23, 13

Slide 151

Slide 151 text

$ knife diff cookbooks/apache Run knife diff again (no output) Tuesday, July 23, 13

Slide 152

Slide 152 text

A new recipe for virtual hosts • We'll create an apache::vhosts recipe to manage the virtual hosts we created in data bag items • There's a number of new things to talk about in this recipe • We'll take this nice and slow :) Tuesday, July 23, 13

Slide 153

Slide 153 text

cookbooks/apache/recipes/vhosts.rb Iterate over all the vhosts items... Tuesday, July 23, 13

Slide 154

Slide 154 text

cookbooks/apache/recipes/vhosts.rb Load the item's data Tuesday, July 23, 13

Slide 155

Slide 155 text

cookbooks/apache/recipes/vhosts.rb Set a local variable for convenience Tuesday, July 23, 13

Slide 156

Slide 156 text

cookbooks/apache/recipes/vhosts.rb Set another local variable... Tuesday, July 23, 13

Slide 157

Slide 157 text

cookbooks/apache/recipes/vhosts.rb Add a virtual host configuration template Tuesday, July 23, 13

Slide 158

Slide 158 text

cookbooks/apache/recipes/vhosts.rb Enable the configuration for apache Tuesday, July 23, 13

Slide 159

Slide 159 text

not_if and only_if • The not_if parameter causes the resource’s actions to be taken only if its argument returns false • The only_if parameter is the opposite of not_if - the actions are taken only if the arguments return true • Both not_if and only_if are part of Chef (resources), not part of Ruby Tuesday, July 23, 13

Slide 160

Slide 160 text

not_if and only_if • not_if and only_if parameters take either a string, or a Ruby block argument (do..end or {..}) • When the argument is a string, Chef evaluates it as a shell command to run. • When the argument is a Ruby block, Chef evaluates it as Ruby code to execute. • This is the equivalent to the code we wrote: not_if "test -L /etc/apache2/sites-enabled/#{site_name}" Tuesday, July 23, 13

Slide 161

Slide 161 text

cookbooks/apache/recipes/vhosts.rb Add a directory resource Tuesday, July 23, 13

Slide 162

Slide 162 text

cookbooks/apache/recipes/vhosts.rb Render a special index.html for each vhost Tuesday, July 23, 13

Slide 163

Slide 163 text

cookbooks/apache/recipes/vhosts.rb Render a special index.html for each vhost Closes the data_bag "each" block Tuesday, July 23, 13

Slide 164

Slide 164 text

cookbooks/apache/templates/default/custom-vhost.erb Create the custom-vhost.erb template Tuesday, July 23, 13

Slide 165

Slide 165 text

https://gist.github.com/2866454 Tuesday, July 23, 13

Slide 166

Slide 166 text

cookbooks/apache/templates/default/index.html.erb Update index.html.erb template... Tuesday, July 23, 13

Slide 167

Slide 167 text

$ knife diff cookbooks/apache Show a diff from the server diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/ recipes/vhosts.rb new file ... diff --knife cookbooks/apache/templates/default/index.html.erb cookbooks/apache/ templates/default/index.html.erb ... diff --knife cookbooks/apache/templates/default/custom-vhost.erb cookbooks/apache/templates/default/custom-vhost.erb new file Tuesday, July 23, 13

Slide 168

Slide 168 text

$ knife upload cookbooks/apache Upload the cookbook Updated cookbooks/apache Tuesday, July 23, 13

Slide 169

Slide 169 text

$ sudo chef-client Run Chef Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action enable (up to date) * service[apache2] action start (up to date) * execute[a2dissite default] action run - execute a2dissite default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 2 resources updated Tuesday, July 23, 13

Slide 170

Slide 170 text

$ knife node run list add packer-vmware 'recipe[apache::vhosts]' Why didn't we get the vhost changes? packer-vmware: run_list: recipe[apache] recipe[apache::vhosts] Tuesday, July 23, 13

Slide 171

Slide 171 text

$ sudo chef-client Run Chef Recipe: apache::vhosts [69/1004] * template[/etc/apache2/sites-available/bears] action create - create template[/etc/apache2/sites-available/bears] --- /tmp/chef-tempfile20130706-20667-2eoiit 2013-07-06 17:24:41.1760 60907 +0000 +++ /tmp/chef-rendered-template20130706-20667-bfl97x 2013-07-06 17:24 :41.176060907 +0000 @@ -0,0 +1,16 @@ + + + ServerAdmin webmaster@localhost + + DocumentRoot /srv/apache/bears + + Options FollowSymLinks + AllowOverride None + + + Options Indexes FollowSymLinks MultiViews + AllowOverride None + Order allow,deny + allow from all + + Tuesday, July 23, 13

Slide 172

Slide 172 text

Chef client output... * execute[a2ensite bears] action run - execute a2ensite bears * directory[/srv/apache/bears] action create - create new directory /srv/apache/bears - change mode from '' to '0755' * template[/srv/apache/bears/index.html] action create - create template[/srv/apache/bears/index.html] --- /tmp/chef-tempfile20130706-20667-1b3ob9q 2013-07-06 17:24:41.284059101 +0000 +++ /tmp/chef-rendered-template20130706-20667-u45tmu 2013-07-06 17:24:41.284059101 +0000 @@ -0,0 +1,4 @@ +

Hello, World

My name is packer-vmware

We love bears

Served from 192.168.21.134:80

... Recipe: apache::default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 9 resources updated Tuesday, July 23, 13

Slide 173

Slide 173 text

Visit your marvelous web site! Tuesday, July 23, 13

Slide 174

Slide 174 text

Visit your marvelous website... port 81! Tuesday, July 23, 13

Slide 175

Slide 175 text

Think about what we just did... • We had two virtual hosts... • But we could arbitrarily add more... • Tigers on port 82, Lions on port 83, oh my! Tuesday, July 23, 13

Slide 176

Slide 176 text

Checkpoint • We'll now use the "knife essentials" upload command to upload to our Chef server. • Our cookbook has two recipes, default and vhosts • Additional data bags can be added, expanding our Virtual Hosting empire! Tuesday, July 23, 13

Slide 177

Slide 177 text

Roles Tuesday, July 23, 13

Slide 178

Slide 178 text

What is a role? • So far, we’ve been just adding recipes directly to our single node's run list • But that’s not how infrastructure works - think about how you refer to servers • "It’s a web server" • "It’s a database server" • or, "It's a database-master server"... • "It’s a monitoring server" Tuesday, July 23, 13

Slide 179

Slide 179 text

How you use roles • Roles allow you to conveniently encapsulate the run lists and attributes required for a server to "be" what you already think it is • In practice, roles make it easy to configure many nodes identically without repeating yourself each time • Roles are a first class API primitive on the Chef Server Tuesday, July 23, 13

Slide 180

Slide 180 text

Best Practice • Like data bags, you have options for creating roles • The best practice is that all of your roles live in the roles directory of your chef-repo • They can be created via the API and knife, but having them in source control gives you the history of changes Tuesday, July 23, 13

Slide 181

Slide 181 text

roles/webserver.json Create a webserver role Tuesday, July 23, 13

Slide 182

Slide 182 text

Components of a role • Roles must have a name • Roles may have a description • Roles may have a run_list, just like a node • Roles may set node attributes • default_attributes • override_attributes http://docs.opscode.com/essentials_roles.html Tuesday, July 23, 13

Slide 183

Slide 183 text

$ knife upload roles/webserver.json Upload the role... Created roles/webserver.json Tuesday, July 23, 13

Slide 184

Slide 184 text

$ knife show roles/webserver.json -VV Show the role DEBUG: Signing the request as zero DEBUG: Sending HTTP Request via GET to localhost:8889/roles/webserver roles/webserver.json: { "name": "webserver", "default_attributes": { "apache": { "greeting": "OSCON Attendees" } }, "run_list": [ "recipe[apache]", "recipe[apache::vhosts]" ] } Tuesday, July 23, 13

Slide 185

Slide 185 text

$ knife node edit packer-vmware Replace the node's run list with the role ERROR: You must set your EDITOR environment variable or configure your editor via knife.rb Tuesday, July 23, 13

Slide 186

Slide 186 text

Set an $EDITOR environment variable $ export EDITOR=/usr/bin/vi OR $ export EDITOR=/usr/bin/emacs OR $ export EDITOR=/usr/bin/nano $ knife node edit packer-vmware Tuesday, July 23, 13

Slide 187

Slide 187 text

Tuesday, July 23, 13

Slide 188

Slide 188 text

Tuesday, July 23, 13

Slide 189

Slide 189 text

$ sudo chef-client Run Chef * template[/srv/apache/bears/index.html] action create - update template[/srv/apache/bears/index.html] from f0a6cc to 652df4 --- /srv/apache/bears/index.html 2013-07-04 23:39:46.016042626 +0000 +++ /tmp/chef-rendered-template20130705-12730-1ndlpik 2013-07-05 01:21:15.406065168 +0000 @@ -1,4 +1,4 @@ -

Hello, World

Hello, OSCON Attendees

My name is packer-vmware

We love bears

Served from 192.168.21.134:80

* template[/srv/apache/clowns/index.html] action create - update template[/srv/apache/clowns/index.html] from 9b27ed to 71c3b5 --- /srv/apache/clowns/index.html 2013-07-04 23:39:46.076041619 +0000 +++ /tmp/chef-rendered-template20130705-12730-1a9eud5 2013-07-05 01:21:15.470064095 +0000 @@ -1,4 +1,4 @@ -

Hello, World

Hello, OSCON Attendees

My name is packer-vmware

We love clowns

Served from 192.168.21.134:81

Tuesday, July 23, 13

Slide 190

Slide 190 text

Attributes can be set multiple places • This is for flexibility • Set a "sane default" that will be used in a cookbook • Override it easily in a role (higher priority) • In all, there are 15 places where attributes come from (!!) • In practice, you'll use 2-3 most of the time. • The others are there when you need them. • http://docs.opscode.com/chef_overview_attributes.html Tuesday, July 23, 13

Slide 191

Slide 191 text

Data Bags Are Not Attributes • Important: data bag items are not attributes • Data bags are a separate API end point • Data bags are not tied to a specific node or role in the infrastructure • Not even necessarily tied to anything, just data you want to store Tuesday, July 23, 13

Slide 192

Slide 192 text

Checkpoint • We now have a webserver role. • We could apply just this role on more nodes to scale out our Virtual Hosting service. • Roles are a great way to assign attributes for specific purposes Tuesday, July 23, 13

Slide 193

Slide 193 text

Search Tuesday, July 23, 13

Slide 194

Slide 194 text

Chef's Search Feature • Search ties together the infrastructure topology • We can now search the Chef Server for all the "webserver" nodes • This is relevant for a variety of reasons • Load balancing several front ends (search from nginx, or haproxy for example) • Monitoring HTTP (search from nagios cookbook) • Graphing traffic (search from munin cookbook) Tuesday, July 23, 13

Slide 195

Slide 195 text

However, we only have one node... • We don't have an environment that is really setup for doing search • But we will talk about what kind of search we do with the command-line, or in a recipe Tuesday, July 23, 13

Slide 196

Slide 196 text

$ knife search node "role:webserver" Search for webservers with knife 1 items found Node Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: role[webserver] Roles: webserver Recipes: apache, apache::vhosts Platform: ubuntu 12.04 Tags: Break down the search command - mention the index, the query, fields, and how to know what fields can be searched Tuesday, July 23, 13

Slide 197

Slide 197 text

$ knife search node "role:webserver" -VV Run with -VV for additional verbosity DEBUG: Signing the request as zero DEBUG: Sending HTTP Request via GET to localhost:8889/search/node DEBUG: ---- HTTP Status and Header Data: ---- DEBUG: HTTP 1.1 200 OK DEBUG: content-type: application/json DEBUG: server: chef-zero DEBUG: connection: close DEBUG: content-length: 33516 DEBUG: ---- End HTTP Status/Header Data ---- 1 items found ... Tuesday, July 23, 13

Slide 198

Slide 198 text

Search from a recipe Tuesday, July 23, 13

Slide 199

Slide 199 text

Available search indexes • The Chef Server indexes JSON data for the major API endpoints: • node • client • environment • role • All data bags are also indexed for search... • vhosts (our example) Tuesday, July 23, 13

Slide 200

Slide 200 text

$ knife search vhosts "port:81" Search for webservers with knife 1 items found chef_type: data_bag_item data_bag: vhosts id: clowns port: 81 Break down the search command - mention the index, the query, fields, and how to know what fields can be searched Tuesday, July 23, 13

Slide 201

Slide 201 text

Replace Data Bag Item Lookup with Search Tuesday, July 23, 13

Slide 202

Slide 202 text

cookbooks/apache/recipes/vhosts.rb Refactor the vhosts recipe... Tuesday, July 23, 13

Slide 203

Slide 203 text

cookbooks/apache/recipes/vhosts.rb Replace with a single search line... Tuesday, July 23, 13

Slide 204

Slide 204 text

$ knife diff cookbooks/apache Show the differences on the server diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/recipes/ vhosts.rb --- cookbooks/apache/recipes/vhosts.rb 2013-07-06 17:57:13.127385411 +0000 +++ cookbooks/apache/recipes/vhosts.rb 2013-07-06 17:57:13.127385411 +0000 @@ -1,5 +1,4 @@ -data_bag("vhosts").each do |site| - site_data = data_bag_item("vhosts", site) +search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" Tuesday, July 23, 13

Slide 205

Slide 205 text

$ knife upload cookbooks/apache Upload to the Chef Server Updated remote/cookbooks/apache Tuesday, July 23, 13

Slide 206

Slide 206 text

$ sudo chef-client Run Chef Client Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears] action create (up to date) * execute[a2ensite bears] action run (skipped due to not_if) * directory[/srv/apache/bears] action create (up to date) * template[/srv/apache/bears/index.html] action create (up to date) * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create (up to date) Chef Client finished, 0 resources updated Tuesday, July 23, 13

Slide 207

Slide 207 text

Hands on Recap Tuesday, July 23, 13

Slide 208

Slide 208 text

Our cookbook... • Chef cookbook "apache" with two recipes: • default (manages apache package and service) • vhosts (iterates over data bags and renders vhost configuration) • However, Opscode publishes an "apache2" cookbook that manages much much more, including all apache2 configuration, modules, sites Debian style w/ a2ensite/a2enmod, etc Tuesday, July 23, 13

Slide 209

Slide 209 text

Our role • webserver role in JSON • Sets a default attribute • Sets a run list • Uploaded with knife upload Tuesday, July 23, 13

Slide 210

Slide 210 text

Our data bag • The vhosts data bag serves as an example • Directory structure follows the data bag API end point and "knife upload" makes it easy to upload everything • Create additional vhosts to see how this is dynamically expanded easily Tuesday, July 23, 13

Slide 211

Slide 211 text

Chef Server • Our Chef Server, Chef Zero, does not persist to disk. • If we ^C the running foreground process or kill the background daemonized process, all the Chef Server data is lost • We can easily upload our repository to a different Chef Server (Open Source, Hosted, Private, or restart Chef Zero) • knife upload . Tuesday, July 23, 13

Slide 212

Slide 212 text

Using A Different Chef Server • Configure knife by modifying the chef_server_url and the node_name values in ./chef/knife.rb • Get the validation client key from the Chef Server (this differs by implementation): • Open Source: •/etc/chef-server/chef-validator.pem • Hosted Chef / Private Chef • Download after creating an organization Tuesday, July 23, 13

Slide 213

Slide 213 text

$ cat .chef/knife.rb The knife.rb configuration Tuesday, July 23, 13

Slide 214

Slide 214 text

On Your Own: Install a Chef Server • Chef 11 Server, Up and Running • 3 Commands! (ish) • Full stack package, ~300Mb • (Please don't download on the wifi! :)) http://bit.ly/XjDGDs Tuesday, July 23, 13

Slide 215

Slide 215 text

Chef Resources Tuesday, July 23, 13

Slide 216

Slide 216 text

Chef Resources (Core Chef) • Chef client comes with 24+ different resources • Packages, files, services, users, symlinks, registry keys, and more • Each resource has one or more providers • Some resources have platform-specific providers (e.g., package, service, user, group) http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 217

Slide 217 text

Managing files and directories • file • cookbook_file • remote_file • template • directory • remote_directory • link http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 218

Slide 218 text

Managing packages • package • apt_package • chef_gem • dpkg_package • easy_install_package • freebsd_package • gem_package • ips_package • macports_package • pacman_package • portage_package • rpm_package • smartos_package • solaris_package • yum_package http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 219

Slide 219 text

RubyGem packages • chef_gem - install a RubyGem into Chef's Ruby environment to be used in a Chef recipe • gem_package - install a RubyGem to be used by the system or an application http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 220

Slide 220 text

Services • service is used to manage services using the common init systems available • Each platform has it's own provider • arch, debian, freebsd, gentoo, "init", insserv • invokercd, macosx (launchd), redhat, "simple" • solaris (SMF), upstart, windows http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 221

Slide 221 text

Other Chef Resources • cron • deploy (revision, timestamped) • env (windows) • group • ifconfig (RHEL) • log • mdadm • mount • ohai • registry_key (windows) • route • scm (git, subversion) • user http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 222

Slide 222 text

There when you need them... • execute (we used this) • script (bash, perl, python, csh, ruby interpreters) • windows_script (batch, powershell) - Chef 11.6.0! • ruby_block http://docs.opscode.com/resource.html Tuesday, July 23, 13

Slide 223

Slide 223 text

Cookbooks to Know About Tuesday, July 23, 13

Slide 224

Slide 224 text

community.opscode.com • apache2, nginx • ark • build-essential • chef-client • chruby • cron • line • java • jenkins • minitest-handler • mysql, postgresql • openssh • omnibus_updater • partial_search • runit • whitelist-node-attrs Tuesday, July 23, 13

Slide 225

Slide 225 text

Opscode Cookbooks with Chef Resources • Opscode has several cookbooks that include new custom Chef Resources • apt (apt_repository) • aws (aws_ebs_volume, aws_elastic_ip, and more) • yum (yum_repository) • windows (windows_package, windows_feature, and more) • homebrew (homebrew_package) • runit (runit_service) • many more! http://docs.opscode.com/chef/lwrps_custom.html Tuesday, July 23, 13

Slide 226

Slide 226 text

Getting Cookbooks from the Community Site • Knife commands: •knife cookbook site download • knife cookbook site install (integrates with git) • Dependency resolvers (a la Ruby's bundler): •librarian-chef •berkshelf Tuesday, July 23, 13

Slide 227

Slide 227 text

Tools to be aware of... Tuesday, July 23, 13

Slide 228

Slide 228 text

Knife Plugins • http://docs.opscode.com/community_plugin_knife.html • Cloud plugins (ec2, openstack, rackspace, google azure, hpcloud, cloudstack, eucalyptus, and more) • knife-server • knife-solo • knife-preflight • knife-essentials • https://github.com/jkeiser/knife-essentials Tuesday, July 23, 13

Slide 229

Slide 229 text

Chef Handlers • http://docs.opscode.com/community_plugin_report_handler.html • IRC • Campfire • HipChat • DataDog • Splunk Storm • Graylog2 • Graphite Tuesday, July 23, 13

Slide 230

Slide 230 text

Workflow Helpers • Berkshelf: berkshelf.com • Librarian-chef: github.com/applicationsonline/ librarian-chef • Knife Spork: github.com/jonlives/knife-spork • Vagrant: vagrantup.com (also a testing tool) Tuesday, July 23, 13

Slide 231

Slide 231 text

Cookbook Testing • Vagrant: vagrantup.com • Test Kitchen: github.com/opscode/test-kitchen • Foodcritic: acrmp.github.io/foodcritic • ChefSpec: acrmp.github.io/chefspec/ • RSpec: rspec.info • minitest-chef: github.com/calavera/minitest-chef- handler • cookbook: minitest-handler Tuesday, July 23, 13

Slide 232

Slide 232 text

Thank you! Questions & Answers [email protected] [email protected] Tuesday, July 23, 13