OSCON 2013 System Management with Chef Tutorial

OSCON 2013 System Management with Chef Tutorial

Raw slides for tutorial

4eb6098fc8de5a5f37199c3668b11590?s=128

Joshua Timberman

July 23, 2013
Tweet

Transcript

  1. 1.

    System Management with Chef Tutorial: OSCON 2013 Joshua Timberman @jtimberman

    James Casey @jamesc_000 joshua@opscode.com github.com/jtimberman james@opscode.com github.com/jamesc Tuesday, July 23, 13
  2. 5.

    Tutorial Objectives • Understand Chef and the problem it solves

    • Understand the components of Chef and how they fit together • Know just enough Ruby to get started • Be able to create a new cookbook • Understand the core primitives of Chef recipes • Get accustomed to the common workflows used by experienced Chef users/developers Tuesday, July 23, 13
  3. 6.

    Expectations • This is (barely) a half-day workshop, not a

    comprehensive course. • We will do some hands on exercises. • You should get a taste for automating with Chef. • You should have received instructions for prerequisites prior to this tutorial • Chef works and runs on Windows, but the exercises for this tutorial are Linux-based Tuesday, July 23, 13
  4. 9.

    Items of Manipulation (Resources) • Nodes • Networking • Files

    • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems Tuesday, July 23, 13
  5. 16.

    App LBs App Servers App DB Cache App DBs Now

    we need a caching layer Tuesday, July 23, 13
  6. 17.

    App LBs App Servers App DB Cache App DBs Infrastructure

    Has a Topology Tuesday, July 23, 13
  7. 18.

    Round Robin DNS App Servers App DB Cache App DBs

    Floating IP? Your's Is a Snowflake Tuesday, July 23, 13
  8. 19.

    App LBs App Servers < Shiny! DB slaves Cache DB

    Cache DBs Complexity Increases Quickly Are we monitoring?? Tuesday, July 23, 13
  9. 21.

    The Chef Framework • Reasonability • Flexibility • Library &

    Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/ Tuesday, July 23, 13
  10. 22.

    The Chef Tool(s) • ohai • chef-client • chef-shell •

    knife • The Ruby language Omnibus - Full Stack Native Packages Tuesday, July 23, 13
  11. 23.

    The Chef API • HTTPS, RESTful API w/ JSON, RSA

    key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/ Tuesday, July 23, 13
  12. 24.

    The Chef Community • Chef is Open Source! • Apache

    License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com Tuesday, July 23, 13
  13. 26.

    Chef Enables Infrastructure as Code • Resources • Recipes •

    Cookbooks and Roles • Source Code http://www.flickr.com/photos/louisb/4555295187/ Tuesday, July 23, 13
  14. 28.

    package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source

    "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Declarative abstraction to system resources Tuesday, July 23, 13
  15. 29.

    package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source

    "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Tuesday, July 23, 13
  16. 33.

    def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if

    @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example Tuesday, July 23, 13
  17. 34.

    Package Resource package "git" { yum install git apt-get install

    git pacman sync git pkg_add -r git Providers are determined by node's platform Tuesday, July 23, 13
  18. 35.

    Recipes are collections of resources package "apache2" do action :install

    end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Tuesday, July 23, 13
  19. 36.

    Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain

    related components • Files, Templates, Libraries • A cookbook is responsible for configuring a single thing, e.g. • apache2 • postgresql • A recipe is responsible for a component, e.g. • api • server • client Tuesday, July 23, 13
  20. 37.

    Roles • Roles describe nodes • Roles indicate functionality •

    Roles have a first class API primitive Tuesday, July 23, 13
  21. 38.

    http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef

    nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search Tuesday, July 23, 13
  22. 39.

    Search • Search for nodes with Roles • Find Topology

    Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625 Tuesday, July 23, 13
  23. 41.

    $ a command you type or file you open Exercise

    Slides Look Like this: Output from the command, or content that goes into the file Tuesday, July 23, 13
  24. 42.

    $ It will be revealed here Hidden Commands Test Your

    Memory After you have read the contents and had a chance to remember the command to type... Tuesday, July 23, 13
  25. 43.

    Hands on Exercises • The majority of the hands on

    exercises will be related to creating an "apache" cookbook. • The goals are to learn elements of Chef, not to learn Apache. We're going to do things the hard way, by typing in a lot of code. We have some gists with large sections of code already available to reduce what you need to type in certain places. • Errors and typos are good, as they will help students learn how to resolve errors. Tuesday, July 23, 13
  26. 44.

    Our Environment • Virtual Machine image (virtualbox, vmware) • Ubuntu

    12.04 (32 bit) • Login: • user - vagrant • password - vagrant Tuesday, July 23, 13
  27. 46.

    Objectives • Verify all the prerequisites are met to complete

    all the exercises. • Introduce the students to the Chef Repository • Start up the Chef Server • Introduce the knife command-line tool Tuesday, July 23, 13
  28. 47.

    Verify Requirements • Virtual Machine should be imported into VirtualBox

    OR VMware (Fusion/Workstation/Player). • Virtual Machine should be able to reach the internet • Chef, Git, Vim, and Emacs are installed • chef-client and knife commands • git, vim, emacs commands • chef-zero gem is installed Tuesday, July 23, 13
  29. 48.

    $ ssh vagrant@192.168.21.134 Log into the VM vagrant@192.168.21.134's password: vagrant

    Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-23- generic i686) * Documentation: https://help.ubuntu.com/ Last login: Thu Jul 18 19:29:26 2013 vagrant@packer-virtualbox:~$ Tuesday, July 23, 13
  30. 49.

    Verification commands (ssh to VM) $ ping -c 3 github.com

    $ which knife $ which chef-client $ which git $ which vim $ which emacs $ /opt/chef/embedded/bin/chef-zero -h Tuesday, July 23, 13
  31. 50.

    $ git clone git://github.com/jtimberman/oscon2013-chef-repo Clone the Chef Repository Cloning into

    'oscon2013-chef-repo'... remote: Counting objects: 13, done. remote: Compressing objects: 100% (9/9), done. remote: Total 13 (delta 2), reused 13 (delta 2) Receiving objects: 100% (13/13), done. Resolving deltas: 100% (2/2), done. Tuesday, July 23, 13
  32. 52.

    $ /opt/chef/embedded/bin/chef-zero -l debug Start up the Chef Server >>

    Starting Chef Zero (v1.5.3)... >> Puma (v1.6.3) is listening at http://127.0.0.1:8889 >> Press CTRL+C to stop $ /opt/chef/embedded/bin/chef-zero -d Tuesday, July 23, 13
  33. 53.

    The Flavors of Chef • Open Source Chef Server •

    Opscode Hosted Chef • Opscode Private Chef • Chef Solo • Chef Zero (our flavor today) • Alternate implementations of the server API Tuesday, July 23, 13
  34. 54.

    Chef Server Components • Runs on HTTPS (port 443) •

    API Service • WebUI Service • Search Index (SOLR) • Message Queue (RabbitMQ) • Database (PostgreSQL) • Cookbook Service (Bookshelf/ filesystem) Tuesday, July 23, 13
  35. 55.

    Why not Open Source Chef Server? • Open Source Chef

    Server package is > 200M • Numerous moving parts • We have limited time in a 3 hour tutorial, and would rather focus on the steeper part of the learning curve • You wouldn't learn sendmail just to send email... Tuesday, July 23, 13
  36. 56.

    After tutorial... • Download and install Open Source Chef Server

    • Use the VM we provided here today • opscode.com/chef/install • (Please don't do this during the tutorial!) Tuesday, July 23, 13
  37. 57.

    What is Chef Zero? • Simple reference implementation of the

    Chef Server API • Everything runs in memory (fast!) • Everything runs in memory (no persistence!) • No authentication (easy/simple for testing use) • Simple installation (gem install chef-zero) • Uses "puma" web server (fast, cross-platform) • Runs on port 8889 (chef-server default is 443) Tuesday, July 23, 13
  38. 58.

    The chef-zero server is ephemeral • Everything is stored in

    memory • If the process crashes or is terminated, it all must be uploaded again • We'll cover how to do this in later exercises Tuesday, July 23, 13
  39. 59.

    $ cd oscon2013-chef-repo Work from the Chef Repository • We're

    going to work from the Chef Repository • All knife commands are run here • All paths are relative to this directory Tuesday, July 23, 13
  40. 63.

    Chef Server • We're using Chef Zero running on "localhost"

    • A Chef Server could be Opscode Hosted or Private, Open Source • The point is we're accessing it over network accessible API service Tuesday, July 23, 13
  41. 64.

    Managed Nodes • Many nodes can be managed by a

    Chef Server • We're going to manage a single node • It happens to be on the same system as the Chef Zero server • Again, we're accessing the server via the API Tuesday, July 23, 13
  42. 65.

    Workstation • We write Chef code on a local workstation

    like a laptop • We upload that code to the Chef Server • Again, using the API Tuesday, July 23, 13
  43. 67.

    Chef uses a Ruby DSL • You don’t need a

    lot of Ruby • When you do need something more advanced, it’s already there • Allows us to easily borrow intermediate to advanced features like testing when we want it, too • Most of the syntax is familiar if you’ve done scripting in other languages, though Tuesday, July 23, 13
  44. 68.

    Ruby is Object Oriented • Everything in Ruby is an

    Object • Call a method on an object with '.': •File.open("/etc/hosts") •[1, 3, 5].each • Ruby is dynamically typed, and it is "duck typed" • "If it walks like a duck, and it talks like a duck..." • Newlines end a statement, but multiple statements can be written on a single line separated by ; Tuesday, July 23, 13
  45. 69.

    Ruby Basics: IRB • IRB is "interactive Ruby" • It

    is a "REPL" or "Read, Eval, Print, Loop" • Installed w/ Ruby package • Installed w/ Chef Omnibus Package • /opt/chef/embedded/bin/irb Tuesday, July 23, 13
  46. 70.

    Ruby Syntax Basics • Assignment: •local_variable = "Some Value" •

    Strings: •"This is a string" •'So is this' • Interpolation: • Only with double-quotes •"this is #{local_variable}" Tuesday, July 23, 13
  47. 71.

    Ruby Basics: Numbers • Numbers are unquoted literals • Integers

    •24 • Floats •4.2 • Other types •0x12F • Math(s)! •3 + 4; 8 * 73; (2+7)/3 Tuesday, July 23, 13
  48. 72.

    Ruby Basics: symbols • Represent names and strings • Only

    one copy exists in memory • Preceding colon •:my_symbol • Often used as hash keys •{:vim => "awesome"} • Can be used in attributes, just be consistent Tuesday, July 23, 13
  49. 73.

    Symbols and Strings irb> "string".object_id => 70221927110640 irb> "string".object_id =>

    70221927077000 irb> "string".object_id => 70221927058480 irb> :string.object_id => 156968 irb> :string.object_id => 156968 Tuesday, July 23, 13
  50. 74.

    Ruby Basics: Flow Control • Conditional Logic •if, case, unless

    • Structures close with end keyword • Equality operators •<, >, <=, >= • Iterators •.each • True / False •true, false, nil OMG 0 is true!! Tuesday, July 23, 13
  51. 75.

    Ruby Basics: Data Structures • Arrays • ["vim", "emacs", "nano",

    "notepad++"] • Hashes • {"vim" => "awesome, installed by default", "emacs" => "awesome but not installed", "notepad++" => "wat? this isn’t Windows"} • symbols and strings are different keys • {"vim" => "awesome", :vim => "different"} Tuesday, July 23, 13
  52. 77.

    Objectives • Understand what a cookbook is • Know how

    to create a new cookbook • Understand what a recipe is • Understand how to use the package, service, and template resources • Know how to upload a cookbook to the Chef Server • Understand what a run list is, and how to set it • How to read the output of the chef-client run Tuesday, July 23, 13
  53. 78.

    What is a cookbook? • A cookbook is like a

    "package" for Chef recipes • It contains all the recipes, files, templates, libraries, etc. required to configure a portion of your infrastructure • Typically cookbooks map 1:1 to a piece of software or functionality. • "our tomcat cookbook" • "our zlib cookbook" • "our security_policy cookbook" Tuesday, July 23, 13
  54. 79.

    $ knife cookbook create apache Create the cookbook with knife

    ** Creating cookbook apache ** Creating README for cookbook: apache ** Creating CHANGELOG for cookbook: apache ** Creating metadata for cookbook: apache Tuesday, July 23, 13
  55. 80.

    $ ls cookbooks/apache What did this create? attributes/ CHANGELOG.md definitions/

    files/ libraries/ metadata.rb providers/ README.md recipes/ resources/ templates/ Tuesday, July 23, 13
  56. 81.

    $ cat cookbooks/apache/metadata.rb What is metadata.rb? name 'apache' maintainer 'YOUR_COMPANY_NAME'

    maintainer_email 'YOUR_EMAIL' license 'All rights reserved' description 'Installs/Configures apache' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0' Tuesday, July 23, 13
  57. 82.

    Cookbook Metadata • Cookbooks are like packages of configuration for

    the infrastructure • They're artifacts that have a name and a version • Metadata can also have information about the cookbook like its license and maintainer • Cookbooks can depend on other cookbooks, too Tuesday, July 23, 13
  58. 83.

    Other relevant components • We'll use other components of the

    cookbook throughout the exercises •recipes/default.rb •templates/default/ •attributes/[default.rb] Tuesday, July 23, 13
  59. 84.

    What is a Recipe? • Ordered list of Resources •

    Ruby Domain-Specific Language (DSL) • Named by the cookbook and the recipe filename • The "default.rb" recipe is referred to by the name of the cookbook (apache) • If we added a recipe to the cookbook named "vhosts.rb" we would refer to it as apache::vhosts Tuesday, July 23, 13
  60. 86.

    So the resource we just wrote... • Is a package

    resource • Whose name is apache2 • With an install action Tuesday, July 23, 13
  61. 87.

    How does the package install? • Resources are declarative -

    we say what we want to have happen, rather than how • Chef uses what platform the node is running to determine the correct provider for a resource Tuesday, July 23, 13
  62. 89.

    The resource we just wrote... • Is a service resource

    • Whose name is apache2 • With two actions: enable and start • It has a parameter called supports that takes a hash of options as an argument Tuesday, July 23, 13
  63. 91.

    The resource we just wrote... • Is a template resource

    • Its name is also the destination of the file on the filesystem • Its source is the location of the template file in the cookbook • Its mode sets the permissions on the file • No action! Default is create Tuesday, July 23, 13
  64. 93.

    $ knife cookbook upload apache Upload the cookbook to the

    Chef Server Uploading apache [0.1.0] Uploaded 1 cookbook. Tuesday, July 23, 13
  65. 94.

    What is "knife cookbook upload"? • Several knife commands correspond

    to API endpoints • The verb indicates what we're doing with the cookbook • In this case we upload the cookbook to the server through the API. It's stored by the server... • We can show information about it too... Tuesday, July 23, 13
  66. 95.

    $ knife cookbook show apache Show the cookbook on the

    Chef Server apache 0.1.0 Tuesday, July 23, 13
  67. 98.

    Checkpoint • We have a cookbook named apache • Our

    apache cookbook has a default recipe • It has a template source file, index.html.erb • We have uploaded the cookbook to the Chef Zero Server. Tuesday, July 23, 13
  68. 100.

    $ knife node list Before we run Chef, list the

    nodes (no output) Tuesday, July 23, 13
  69. 101.

    Create /etc/chef and validation.pem $ sudo mkdir /etc/chef $ sudo

    cp .chef/zero.pem /etc/chef/validation.pem Tuesday, July 23, 13
  70. 104.

    Chef Zero doesn't authenticate • Chef Zero doesn't enforce authentication

    (other Chef Servers do) • It also doesn't run over HTTPS (other Chef Servers do) • The authentication cycle is still valid, though • The client.pem will still be written out, let's take a look... Tuesday, July 23, 13
  71. 105.

    $ sudo chef-client Now, run Chef Client Starting Chef Client,

    version 11.4.4 Creating a new client identity for packer-vmware using the validator key. resolving cookbooks for run list: [] Synchronizing Cookbooks: Compiling Cookbooks... [2013-07-06T14:42:35+00:00] WARN: Node packer-vmware has an empty run list. Converging 0 resources Chef Client finished, 0 resources updated Tuesday, July 23, 13
  72. 106.

    Wait, why didn't Chef apply the recipe? • We didn't

    have a node object on the server • We didn't tell Chef about the node when we started chef-client • Let us look at what we have now... Tuesday, July 23, 13
  73. 108.

    $ knife client list Show the client list chef-validator chef-webui

    packer-vmware packer-virtualbox (Your client should be named -vmware or - virtualbox, depending on the VM image) Tuesday, July 23, 13
  74. 109.

    $ knife client show packer-vmware Show the new client admin:

    false chef_type: client json_class: Chef::ApiClient name: packer-vmware public_key: -----BEGIN PUBLIC KEY----- MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvW1NgunHiKJqvXNk5kLO 7AkqFY6O0pZL46y9/OiqrCUYdDeaIeskYWKaKIVFdUcqFf/nBao97y35cfsSOkJI ZTQUdk0lEUF/+mdGOAjur/tUIllltWG4xJHI5EBEfvSxV8DO3gUuizIIs591SNsp XCWlLsBCYRRzhGhA60JMF4Y1EEXhVINhMsr2jleNOPCInGxz3sqWY+1KfwyxNjHL woH9Vi6uGCZ1FT0A1i19Kg7j1EQSCEMhcMKZ8vpUpAnmwwqshFHx+UubrDCd1v8H VSdJAHtUqqc3iLTfSRZIjak7J8ZdYXF8chg1wfgkjnpVcTr2lOy3iuLzxReI/9rs iQIDAQAB -----END PUBLIC KEY----- important to note that the server stores the PUBLIC key - think SSH Tuesday, July 23, 13
  75. 110.

    $ knife node list Show the node list again packer-vmware

    packer-virtualbox (we'll use -vmware throughout the remainder of the slides) Tuesday, July 23, 13
  76. 111.

    $ knife node show packer-vmware Show the new node Node

    Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags: Tuesday, July 23, 13
  77. 112.

    $ knife node run list add packer-vmware 'recipe[apache]' How do

    we modify the run list? packer-vmware: run_list: recipe[apache] Tuesday, July 23, 13
  78. 113.

    $ sudo chef-client Run Chef Client again Starting Chef Client,

    version 11.4.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install - install version 2.2.22-1ubuntu1.3 of package apache2 * service[apache2] action enable (up to date) * service[apache2] action start (up to date) * template[/var/www/index.html] action create - update template[/var/www/index.html] from 94850c to 3668a4 --- /var/www/index.html 2013-07-06 14:51:17.206134567 +0000 +++ /tmp/chef-rendered-template20130706-17703-en6g83 2013-07-06 14:51:17.702126265 +0000 @@ -1,4 +1,2 @@ -<html><body><h1>It works!</h1> -<p>This is the default web page for this server.</p> -<p>The web server software is running but no content has been added, yet.</p> -</body></html> +<p>Hello, World</p> + Chef Client finished, 2 resources updated Tuesday, July 23, 13
  79. 114.

    $ sudo apt-get update Did you get this error? [2013-07-18T21:14:56+00:00]

    ERROR: Running exception handlers [2013-07-18T21:14:56+00:00] FATAL: Saving node information to / var/chef/cache/failed-run-data.json [2013-07-18T21:14:56+00:00] ERROR: Exception handlers complete Chef Client failed. 0 resources updated [2013-07-18T21:14:56+00:00] FATAL: Stacktrace dumped to /var/chef/ cache/chef-stacktrace.out [2013-07-18T21:14:56+00:00] FATAL: Chef::Exceptions::Exec: package[apache2] (apache::default line 9) had an error: Chef::Exceptions::Exec: apt-get -q -y install apache2=2.2.22-1ubuntu1.3 returned 100, expected 0 Tuesday, July 23, 13
  80. 115.

    The problem... • This is [an unintended] illustration of the

    problem with "golden images" • The images we're using were created a couple of weeks ago • Ubuntu has released new versions of Apache 2 packages (security update) • The apt cache is stale, so we need to update it • The recommended practice is to put "apt-get update" in a recipe, though we won't do that today Tuesday, July 23, 13
  81. 116.

    $ sudo chef-client Run Chef Client again Starting Chef Client,

    version 11.4.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install - install version 2.2.22-1ubuntu1.3 of package apache2 * service[apache2] action enable (up to date) * service[apache2] action start (up to date) * template[/var/www/index.html] action create - update template[/var/www/index.html] from 94850c to 3668a4 --- /var/www/index.html 2013-07-06 14:51:17.206134567 +0000 +++ /tmp/chef-rendered-template20130706-17703-en6g83 2013-07-06 14:51:17.702126265 +0000 @@ -1,4 +1,2 @@ -<html><body><h1>It works!</h1> -<p>This is the default web page for this server.</p> -<p>The web server software is running but no content has been added, yet.</p> -</body></html> +<p>Hello, World</p> + Chef Client finished, 2 resources updated Tuesday, July 23, 13
  82. 117.

    * service[apache2] action enable (up to date) * service[apache2] action

    start (up to date) Why wasn't the service updated? • Chef takes idempotent actions on resources to ensure they are converged to the desired state • The apache2 package on Ubuntu automatically enables and starts the apache2 service in its post- install script • Thus, Chef didn't need to do anything Tuesday, July 23, 13
  83. 118.

    If we stopped apache2... $ sudo /etc/init.d/apache2 stop * Stopping

    web server apache2 apache2: [ OK ] $ sudo chef-client .... * service[apache2] action start - start service service[apache2] Tuesday, July 23, 13
  84. 119.

    build node authenticate sync cookbooks load cookbooks converge node.save notification

    handlers exception Yes No chef-client success? expanded run list (recipes) Ohai! node_name platform platform_version Tuesday, July 23, 13
  85. 120.

    Processing Recipes is Two-Phase • Chef processes recipes in two

    phases during "convergence" • Evaluate all the Ruby code in the recipe, looking for "Chef Resources" • Execute the providers' actions to put each resource in the declared state Tuesday, July 23, 13
  86. 122.

    The Chef Node Object • Nodes are the objects that

    you manage with Chef • They have a few different properties • attributes • run_list • chef_environment Tuesday, July 23, 13
  87. 123.

    The Chef Node Object • In client/server Chef, the Chef

    Server stores node object data • It becomes searchable through the API with knife and with recipes • Some of the data comes from ohai, which takes an inventory of the system and emits JSON data • You can add data to the node through attributes in cookbooks, roles, directly on a node, etc Tuesday, July 23, 13
  88. 124.

    Node Attributes Syntax • Node attributes are hashes (of hashes)

    • Reference hash keys with square brackets and keys as strings •node["hostname"] •node["kernel"]["machine"] Tuesday, July 23, 13
  89. 126.

    erb syntax • Built into the Ruby Standard Library, but

    there is an separate library (erubis) that performs a bit better • To embed a value within an ERB template: • Start with <%= • Write your Ruby expression - most commonly a node attribute • End with %> • A lot like working in PHP, Mason, etc Tuesday, July 23, 13
  90. 127.

    $ knife cookbook upload apache Upload the new cookbook Uploading

    apache [0.1.0] Uploaded 1 cookbook. Tuesday, July 23, 13
  91. 128.

    $ sudo chef-client Run Chef Client * template[/var/www/index.html] action create

    - update template[/var/www/index.html] from 3668a4 to 418b05 --- /var/www/index.html 2013-07-06 14:51:17.758125327 +0000 +++ /tmp/chef-rendered-template20130706-18568-cd45vw 2013-07-06 15:09:56.631395484 +0000 @@ -1,2 +1,2 @@ <p>Hello, World</p> - +<p>My name is packer-vmware</p> Chef Client finished, 1 resources updated Explain how the checksum things work for the content change in the template. Note that we don't get the ERB content, we get the rendered content of the template which comes from the attribute Tuesday, July 23, 13
  92. 129.

    $ ohai Run ohai on the node { "languages": {

    "ruby": { }, "python": { "version": "2.7.3", "builddate": "Apr 10 2013, 05:46:21" }, "perl": { "version": "5.14.2", "archname": "i686-linux-gnu-thread-multi-64int" } }, "kernel": { "name": "Linux", "release": "3.5.0-23-generic", "version": "#35~precise1-Ubuntu SMP Fri Jan 25 17:15:33 UTC 2013", "machine": "i686" <SNIP> Tuesday, July 23, 13
  93. 130.

    Run ohai with an attribute $ ohai hostname [ "packer-vmware"

    ] $ ohai ipaddress [ "192.168.21.134" ] Explain that ohai just mainly runs commands on the underlying system for Linux/Unix systems. hostname -f default route from netstat Tuesday, July 23, 13
  94. 131.

    $ knife node show packer-vmware Show the Node object Node

    Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: recipe[apache] Roles: Recipes: apache Platform: ubuntu 12.04 Tags: Tuesday, July 23, 13
  95. 132.

    Show specific attributes of the node $ knife node show

    packer-vmware -a hostname packer-vmware: hostname: packer-vmware $ knife node show packer-vmware -a ipaddress packer-vmware: ipaddress: 192.168.21.134 $ knife node show packer-vmware -a platform packer-vmware: platform: ubuntu Tuesday, July 23, 13
  96. 134.

    Objectives • Set node attributes from a cookbook • Understand

    node attribute precedence Tuesday, July 23, 13
  97. 135.

    cookbooks/apache/attributes/default.rb Create a cookbook attributes file what is default? what

    are these brackets? what is an equals zomg what are “? Tuesday, July 23, 13
  98. 136.

    Cookbook Attributes & Precedence • Always set default node attributes

    in your cookbooks' attributes files • Use "sane" defaults - no surprises • You can use attributes in roles to set new values • Roles take precedence over cookbook settings • When a value must be set to a certain value, use override, but use this sparingly • You can’t override ohai's (automatic) attributes! Tuesday, July 23, 13
  99. 138.

    $ knife cookbook upload $ sudo chef-client Upload the cookbook,

    run Chef Recipe: apache::default * package[apache2] action install (up to date) * template[/var/www/index.html] action create (up to date) * service[apache2] action enable (up to date) * service[apache2] action start (up to date) Chef Client finished, 0 resources updated Why didn't the index.html file change? The attribute is the same value as the previous content Tuesday, July 23, 13
  100. 139.

    Checkpoint • We have a node attribute • node["apache"]["greeting"] •

    We've updated the index.html template to use this attribute. • This will be used again soon! Tuesday, July 23, 13
  101. 141.

    Objectives • Learn how to use Data Bags for data-driven

    recipes • Use the "knife essentials" plugin commands • Understand using multiple recipes for a node's run list • Control execution of arbitrary commands with Chef's resource conditionals Tuesday, July 23, 13
  102. 142.

    Data Bags are generic stores of information • Data bags

    are generic, arbitrary stores of information about the infrastructure. • Data Bag Items are JSON data • Our apache cookbook provides a good baseline • We'll drive site-specific virtual hosts with data bags Tuesday, July 23, 13
  103. 143.

    Create the data bag and items $ mkdir data_bags/vhosts $

    vi data_bags/vhosts/bears.json $ vi data_bags/vhosts/clowns.json Tuesday, July 23, 13
  104. 146.

    $ knife upload data_bags/vhosts Upload all the data bag items

    Created remote/data_bags/vhosts/ Created remote/data_bags/vhosts/bears.json Created remote/data_bags/vhosts/clowns.json Tuesday, July 23, 13
  105. 147.

    knife upload • Comes from the "knife essentials" plugin, now

    built into Chef (11+) • Treat the local chef-repo and the chef server like filesystems • Useful commands like upload, download, show, diff • Easier to remember than all the different object actions • Tab completion! Tuesday, July 23, 13
  106. 149.

    $ knife diff cookbooks/apache Introducing knife diff diff --knife cookbooks/apache/recipes/default.rb

    cookbooks/apache/recipes/default.rb --- cookbooks/apache/recipes/default.rb 2013-07-06 16:28:15.136742911 +0000 +++ cookbooks/apache/recipes/default.rb 2013-07-06 16:28:15.136742911 +0000 @@ -16,8 +16,9 @@ action [:enable, :start] end -template "/var/www/index.html" do - source "index.html.erb" - mode "0644" +execute "a2dissite default" do + only_if do + ::File.symlink?("/etc/apache2/sites-enabled/000-default") + end + notifies :restart, "service[apache2]" end Tuesday, July 23, 13
  107. 152.

    A new recipe for virtual hosts • We'll create an

    apache::vhosts recipe to manage the virtual hosts we created in data bag items • There's a number of new things to talk about in this recipe • We'll take this nice and slow :) Tuesday, July 23, 13
  108. 159.

    not_if and only_if • The not_if parameter causes the resource’s

    actions to be taken only if its argument returns false • The only_if parameter is the opposite of not_if - the actions are taken only if the arguments return true • Both not_if and only_if are part of Chef (resources), not part of Ruby Tuesday, July 23, 13
  109. 160.

    not_if and only_if • not_if and only_if parameters take either

    a string, or a Ruby block argument (do..end or {..}) • When the argument is a string, Chef evaluates it as a shell command to run. • When the argument is a Ruby block, Chef evaluates it as Ruby code to execute. • This is the equivalent to the code we wrote: not_if "test -L /etc/apache2/sites-enabled/#{site_name}" Tuesday, July 23, 13
  110. 167.

    $ knife diff cookbooks/apache Show a diff from the server

    diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/ recipes/vhosts.rb new file ... diff --knife cookbooks/apache/templates/default/index.html.erb cookbooks/apache/ templates/default/index.html.erb ... diff --knife cookbooks/apache/templates/default/custom-vhost.erb cookbooks/apache/templates/default/custom-vhost.erb new file Tuesday, July 23, 13
  111. 169.

    $ sudo chef-client Run Chef Recipe: apache::default * package[apache2] action

    install (up to date) * service[apache2] action enable (up to date) * service[apache2] action start (up to date) * execute[a2dissite default] action run - execute a2dissite default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 2 resources updated Tuesday, July 23, 13
  112. 170.

    $ knife node run list add packer-vmware 'recipe[apache::vhosts]' Why didn't

    we get the vhost changes? packer-vmware: run_list: recipe[apache] recipe[apache::vhosts] Tuesday, July 23, 13
  113. 171.

    $ sudo chef-client Run Chef Recipe: apache::vhosts [69/1004] * template[/etc/apache2/sites-available/bears]

    action create - create template[/etc/apache2/sites-available/bears] --- /tmp/chef-tempfile20130706-20667-2eoiit 2013-07-06 17:24:41.1760 60907 +0000 +++ /tmp/chef-rendered-template20130706-20667-bfl97x 2013-07-06 17:24 :41.176060907 +0000 @@ -0,0 +1,16 @@ + +<VirtualHost *:80> + ServerAdmin webmaster@localhost + + DocumentRoot /srv/apache/bears + <Directory /> + Options FollowSymLinks + AllowOverride None + </Directory> + <Directory /srv/apache/bears> + Options Indexes FollowSymLinks MultiViews + AllowOverride None + Order allow,deny + allow from all + </Directory> +</VirtualHost> Tuesday, July 23, 13
  114. 172.

    Chef client output... * execute[a2ensite bears] action run - execute

    a2ensite bears * directory[/srv/apache/bears] action create - create new directory /srv/apache/bears - change mode from '' to '0755' * template[/srv/apache/bears/index.html] action create - create template[/srv/apache/bears/index.html] --- /tmp/chef-tempfile20130706-20667-1b3ob9q 2013-07-06 17:24:41.284059101 +0000 +++ /tmp/chef-rendered-template20130706-20667-u45tmu 2013-07-06 17:24:41.284059101 +0000 @@ -0,0 +1,4 @@ +<p>Hello, World</p> +<p>My name is packer-vmware</p> +<p>We love bears</p> +<p>Served from 192.168.21.134:80</p> ... Recipe: apache::default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 9 resources updated Tuesday, July 23, 13
  115. 175.

    Think about what we just did... • We had two

    virtual hosts... • But we could arbitrarily add more... • Tigers on port 82, Lions on port 83, oh my! Tuesday, July 23, 13
  116. 176.

    Checkpoint • We'll now use the "knife essentials" upload command

    to upload to our Chef server. • Our cookbook has two recipes, default and vhosts • Additional data bags can be added, expanding our Virtual Hosting empire! Tuesday, July 23, 13
  117. 178.

    What is a role? • So far, we’ve been just

    adding recipes directly to our single node's run list • But that’s not how infrastructure works - think about how you refer to servers • "It’s a web server" • "It’s a database server" • or, "It's a database-master server"... • "It’s a monitoring server" Tuesday, July 23, 13
  118. 179.

    How you use roles • Roles allow you to conveniently

    encapsulate the run lists and attributes required for a server to "be" what you already think it is • In practice, roles make it easy to configure many nodes identically without repeating yourself each time • Roles are a first class API primitive on the Chef Server Tuesday, July 23, 13
  119. 180.

    Best Practice • Like data bags, you have options for

    creating roles • The best practice is that all of your roles live in the roles directory of your chef-repo • They can be created via the API and knife, but having them in source control gives you the history of changes Tuesday, July 23, 13
  120. 182.

    Components of a role • Roles must have a name

    • Roles may have a description • Roles may have a run_list, just like a node • Roles may set node attributes • default_attributes • override_attributes http://docs.opscode.com/essentials_roles.html Tuesday, July 23, 13
  121. 184.

    $ knife show roles/webserver.json -VV Show the role DEBUG: Signing

    the request as zero DEBUG: Sending HTTP Request via GET to localhost:8889/roles/webserver roles/webserver.json: { "name": "webserver", "default_attributes": { "apache": { "greeting": "OSCON Attendees" } }, "run_list": [ "recipe[apache]", "recipe[apache::vhosts]" ] } Tuesday, July 23, 13
  122. 185.

    $ knife node edit packer-vmware Replace the node's run list

    with the role ERROR: You must set your EDITOR environment variable or configure your editor via knife.rb Tuesday, July 23, 13
  123. 186.

    Set an $EDITOR environment variable $ export EDITOR=/usr/bin/vi OR $

    export EDITOR=/usr/bin/emacs OR $ export EDITOR=/usr/bin/nano $ knife node edit packer-vmware Tuesday, July 23, 13
  124. 189.

    $ sudo chef-client Run Chef * template[/srv/apache/bears/index.html] action create -

    update template[/srv/apache/bears/index.html] from f0a6cc to 652df4 --- /srv/apache/bears/index.html 2013-07-04 23:39:46.016042626 +0000 +++ /tmp/chef-rendered-template20130705-12730-1ndlpik 2013-07-05 01:21:15.406065168 +0000 @@ -1,4 +1,4 @@ -<p>Hello, World</p> +<p>Hello, OSCON Attendees</p> <p>My name is packer-vmware</p> <p>We love bears</p> <p>Served from 192.168.21.134:80</p> * template[/srv/apache/clowns/index.html] action create - update template[/srv/apache/clowns/index.html] from 9b27ed to 71c3b5 --- /srv/apache/clowns/index.html 2013-07-04 23:39:46.076041619 +0000 +++ /tmp/chef-rendered-template20130705-12730-1a9eud5 2013-07-05 01:21:15.470064095 +0000 @@ -1,4 +1,4 @@ -<p>Hello, World</p> +<p>Hello, OSCON Attendees</p> <p>My name is packer-vmware</p> <p>We love clowns</p> <p>Served from 192.168.21.134:81</p> Tuesday, July 23, 13
  125. 190.

    Attributes can be set multiple places • This is for

    flexibility • Set a "sane default" that will be used in a cookbook • Override it easily in a role (higher priority) • In all, there are 15 places where attributes come from (!!) • In practice, you'll use 2-3 most of the time. • The others are there when you need them. • http://docs.opscode.com/chef_overview_attributes.html Tuesday, July 23, 13
  126. 191.

    Data Bags Are Not Attributes • Important: data bag items

    are not attributes • Data bags are a separate API end point • Data bags are not tied to a specific node or role in the infrastructure • Not even necessarily tied to anything, just data you want to store Tuesday, July 23, 13
  127. 192.

    Checkpoint • We now have a webserver role. • We

    could apply just this role on more nodes to scale out our Virtual Hosting service. • Roles are a great way to assign attributes for specific purposes Tuesday, July 23, 13
  128. 194.

    Chef's Search Feature • Search ties together the infrastructure topology

    • We can now search the Chef Server for all the "webserver" nodes • This is relevant for a variety of reasons • Load balancing several front ends (search from nginx, or haproxy for example) • Monitoring HTTP (search from nagios cookbook) • Graphing traffic (search from munin cookbook) Tuesday, July 23, 13
  129. 195.

    However, we only have one node... • We don't have

    an environment that is really setup for doing search • But we will talk about what kind of search we do with the command-line, or in a recipe Tuesday, July 23, 13
  130. 196.

    $ knife search node "role:webserver" Search for webservers with knife

    1 items found Node Name: packer-vmware Environment: _default FQDN: packer-vmware IP: 192.168.21.134 Run List: role[webserver] Roles: webserver Recipes: apache, apache::vhosts Platform: ubuntu 12.04 Tags: Break down the search command - mention the index, the query, fields, and how to know what fields can be searched Tuesday, July 23, 13
  131. 197.

    $ knife search node "role:webserver" -VV Run with -VV for

    additional verbosity DEBUG: Signing the request as zero DEBUG: Sending HTTP Request via GET to localhost:8889/search/node DEBUG: ---- HTTP Status and Header Data: ---- DEBUG: HTTP 1.1 200 OK DEBUG: content-type: application/json DEBUG: server: chef-zero DEBUG: connection: close DEBUG: content-length: 33516 DEBUG: ---- End HTTP Status/Header Data ---- 1 items found ... Tuesday, July 23, 13
  132. 199.

    Available search indexes • The Chef Server indexes JSON data

    for the major API endpoints: • node • client • environment • role • All data bags are also indexed for search... • vhosts (our example) Tuesday, July 23, 13
  133. 200.

    $ knife search vhosts "port:81" Search for webservers with knife

    1 items found chef_type: data_bag_item data_bag: vhosts id: clowns port: 81 Break down the search command - mention the index, the query, fields, and how to know what fields can be searched Tuesday, July 23, 13
  134. 204.

    $ knife diff cookbooks/apache Show the differences on the server

    diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/recipes/ vhosts.rb --- cookbooks/apache/recipes/vhosts.rb 2013-07-06 17:57:13.127385411 +0000 +++ cookbooks/apache/recipes/vhosts.rb 2013-07-06 17:57:13.127385411 +0000 @@ -1,5 +1,4 @@ -data_bag("vhosts").each do |site| - site_data = data_bag_item("vhosts", site) +search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" Tuesday, July 23, 13
  135. 205.

    $ knife upload cookbooks/apache Upload to the Chef Server Updated

    remote/cookbooks/apache Tuesday, July 23, 13
  136. 206.

    $ sudo chef-client Run Chef Client Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears]

    action create (up to date) * execute[a2ensite bears] action run (skipped due to not_if) * directory[/srv/apache/bears] action create (up to date) * template[/srv/apache/bears/index.html] action create (up to date) * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create (up to date) Chef Client finished, 0 resources updated Tuesday, July 23, 13
  137. 208.

    Our cookbook... • Chef cookbook "apache" with two recipes: •

    default (manages apache package and service) • vhosts (iterates over data bags and renders vhost configuration) • However, Opscode publishes an "apache2" cookbook that manages much much more, including all apache2 configuration, modules, sites Debian style w/ a2ensite/a2enmod, etc Tuesday, July 23, 13
  138. 209.

    Our role • webserver role in JSON • Sets a

    default attribute • Sets a run list • Uploaded with knife upload Tuesday, July 23, 13
  139. 210.

    Our data bag • The vhosts data bag serves as

    an example • Directory structure follows the data bag API end point and "knife upload" makes it easy to upload everything • Create additional vhosts to see how this is dynamically expanded easily Tuesday, July 23, 13
  140. 211.

    Chef Server • Our Chef Server, Chef Zero, does not

    persist to disk. • If we ^C the running foreground process or kill the background daemonized process, all the Chef Server data is lost • We can easily upload our repository to a different Chef Server (Open Source, Hosted, Private, or restart Chef Zero) • knife upload . Tuesday, July 23, 13
  141. 212.

    Using A Different Chef Server • Configure knife by modifying

    the chef_server_url and the node_name values in ./chef/knife.rb • Get the validation client key from the Chef Server (this differs by implementation): • Open Source: •/etc/chef-server/chef-validator.pem • Hosted Chef / Private Chef • Download after creating an organization Tuesday, July 23, 13
  142. 214.

    On Your Own: Install a Chef Server • Chef 11

    Server, Up and Running • 3 Commands! (ish) • Full stack package, ~300Mb • (Please don't download on the wifi! :)) http://bit.ly/XjDGDs Tuesday, July 23, 13
  143. 216.

    Chef Resources (Core Chef) • Chef client comes with 24+

    different resources • Packages, files, services, users, symlinks, registry keys, and more • Each resource has one or more providers • Some resources have platform-specific providers (e.g., package, service, user, group) http://docs.opscode.com/resource.html Tuesday, July 23, 13
  144. 217.

    Managing files and directories • file • cookbook_file • remote_file

    • template • directory • remote_directory • link http://docs.opscode.com/resource.html Tuesday, July 23, 13
  145. 218.

    Managing packages • package • apt_package • chef_gem • dpkg_package

    • easy_install_package • freebsd_package • gem_package • ips_package • macports_package • pacman_package • portage_package • rpm_package • smartos_package • solaris_package • yum_package http://docs.opscode.com/resource.html Tuesday, July 23, 13
  146. 219.

    RubyGem packages • chef_gem - install a RubyGem into Chef's

    Ruby environment to be used in a Chef recipe • gem_package - install a RubyGem to be used by the system or an application http://docs.opscode.com/resource.html Tuesday, July 23, 13
  147. 220.

    Services • service is used to manage services using the

    common init systems available • Each platform has it's own provider • arch, debian, freebsd, gentoo, "init", insserv • invokercd, macosx (launchd), redhat, "simple" • solaris (SMF), upstart, windows http://docs.opscode.com/resource.html Tuesday, July 23, 13
  148. 221.

    Other Chef Resources • cron • deploy (revision, timestamped) •

    env (windows) • group • ifconfig (RHEL) • log • mdadm • mount • ohai • registry_key (windows) • route • scm (git, subversion) • user http://docs.opscode.com/resource.html Tuesday, July 23, 13
  149. 222.

    There when you need them... • execute (we used this)

    • script (bash, perl, python, csh, ruby interpreters) • windows_script (batch, powershell) - Chef 11.6.0! • ruby_block http://docs.opscode.com/resource.html Tuesday, July 23, 13
  150. 224.

    community.opscode.com • apache2, nginx • ark • build-essential • chef-client

    • chruby • cron • line • java • jenkins • minitest-handler • mysql, postgresql • openssh • omnibus_updater • partial_search • runit • whitelist-node-attrs Tuesday, July 23, 13
  151. 225.

    Opscode Cookbooks with Chef Resources • Opscode has several cookbooks

    that include new custom Chef Resources • apt (apt_repository) • aws (aws_ebs_volume, aws_elastic_ip, and more) • yum (yum_repository) • windows (windows_package, windows_feature, and more) • homebrew (homebrew_package) • runit (runit_service) • many more! http://docs.opscode.com/chef/lwrps_custom.html Tuesday, July 23, 13
  152. 226.

    Getting Cookbooks from the Community Site • Knife commands: •knife

    cookbook site download • knife cookbook site install (integrates with git) • Dependency resolvers (a la Ruby's bundler): •librarian-chef •berkshelf Tuesday, July 23, 13
  153. 228.

    Knife Plugins • http://docs.opscode.com/community_plugin_knife.html • Cloud plugins (ec2, openstack, rackspace,

    google azure, hpcloud, cloudstack, eucalyptus, and more) • knife-server • knife-solo • knife-preflight • knife-essentials • https://github.com/jkeiser/knife-essentials Tuesday, July 23, 13
  154. 229.

    Chef Handlers • http://docs.opscode.com/community_plugin_report_handler.html • IRC • Campfire • HipChat

    • DataDog • Splunk Storm • Graylog2 • Graphite Tuesday, July 23, 13
  155. 230.

    Workflow Helpers • Berkshelf: berkshelf.com • Librarian-chef: github.com/applicationsonline/ librarian-chef •

    Knife Spork: github.com/jonlives/knife-spork • Vagrant: vagrantup.com (also a testing tool) Tuesday, July 23, 13
  156. 231.

    Cookbook Testing • Vagrant: vagrantup.com • Test Kitchen: github.com/opscode/test-kitchen •

    Foodcritic: acrmp.github.io/foodcritic • ChefSpec: acrmp.github.io/chefspec/ • RSpec: rspec.info • minitest-chef: github.com/calavera/minitest-chef- handler • cookbook: minitest-handler Tuesday, July 23, 13