Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to Chef SCaLE12x

Nathen Harvey
February 22, 2014

Intro to Chef SCaLE12x

Introduction to Chef Workshop from SCaLE12x.

Nathen Harvey

February 22, 2014
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. v2.0.0 Introduction to Chef 1 Nathen Harvey @nathenharvey Kennon Kwok

    @kennonkwok [email protected] github.com/nathenharvey [email protected] github.com/kennonkwok Friday, February 21, 14
  2. Intro to Chef - 10:00AM 2 • Pre-requisites • Install

    the Chef Client - getchef.com/chef/install • Have an ssh client • Have a good text editor $ irb irb(main):001:0> "password".reverse => "drowssap" Friday, February 21, 14
  3. Instructor Introduction • Name: Nathen Harvey • Current job role:

    Director Community & #learnchef • Previous job roles/background: Web Operations, Customer Support, Web Development • Experience with Chef/Config Management: Been using Chef for about 4 years. • Favorite Text Editor: Sublime Text 4 Friday, February 21, 14
  4. Instructor Introduction • Name: Kennon Kwok • Current job role:

    Consulting Engineer • Previous job roles/background: Ops Engineer, SRE, HPC Engineer • Experience with Chef/Config Management: Chef user for ~1 year, Puppet user for a few years • Favorite Text Editor: vim 5 Friday, February 21, 14
  5. Who are you? • System administrator? • Developer? • Ruby

    Developer? • DevOp? 6 Friday, February 21, 14
  6. Who are you? • Favorite text editor? • Which version

    control system do you use? •cp foo foo.bak •cp foo{,.`date +%Y%m%d%H%M`} 8 Friday, February 21, 14
  7. Tutorial Objectives • Describe the types of problems Chef solves

    • List the components of Chef • Create a new cookbook • Use some primitives of Chef recipes • Follow some common workflows used by experienced Chef users/developers Friday, February 21, 14
  8. Pre-requisites? • Install the Chef Client - getchef.com/chef/install • Have

    an ssh client • Have a good text editor • http://bit.ly/scale12x-chef Friday, February 21, 14
  9. Expectations • This is a one-day workshop, not a comprehensive

    course. • We will do some hands on exercises. • You should get a taste for automating with Chef. • You should have received instructions for prerequisites prior to this tutorial • Chef works and runs on Windows, but the exercises for this tutorial are Linux-based Friday, February 21, 14
  10. How to learn Chef • You bring the domain expertise

    about your business and problems • Chef provides a framework for solving those problems • Our job is to work together to teach you how to express solutions to your problems with Chef 12 Friday, February 21, 14
  11. Lesson Objectives • After completing the lesson, you will be

    able to • Describe how Chef thinks about Infrastructure Automation • Define the following terms: • Node • Resource • Recipe • Cookbook • Run List • Roles • Search 14 Friday, February 21, 14
  12. Items of Manipulation (Resources) • Networking • Files • Directories

    • Symlinks • Mounts • Registry Keys • Powershell Scripts • Users • Groups • Packages • Services • Filesystems 16 Friday, February 21, 14
  13. App LBs App Servers App DB Cache App DBs Now

    we need a caching layer 23 Friday, February 21, 14
  14. App LBs App Servers App DB Cache App DBs Infrastructure

    has a Topology 24 Friday, February 21, 14
  15. Round Robin DNS App Servers App DB Cache App DBs

    Floating IP? Your Infrastructure is a Snowflake 25 Friday, February 21, 14
  16. App LBs App Servers < Shiny! DB slaves Cache DB

    Cache DBs Complexity Increases Quickly 26 Friday, February 21, 14
  17. App LBs App Servers < Shiny! DB slaves Cache DB

    Cache DBs Complexity Increases Quickly Are we monitoring?? 26 Friday, February 21, 14
  18. App LBs App Servers DB slaves Cache DB Cache DBs

    ...and change happens! 27 Friday, February 21, 14
  19. App LBs App Servers DB slaves Cache DB Cache DBs

    ...and change happens! Add a Central Log Host Central Log Host 28 Friday, February 21, 14
  20. App LBs App Servers DB slaves Cache DB Cache DBs

    ...and change happens! Add a Central Log Host Update syslog.conf on all Nodes Central Log Host 29 Friday, February 21, 14
  21. Chef Solves This Problem • But you already guessed that,

    didn’t you? 30 Friday, February 21, 14
  22. Chef is Infrastructure as Code • Programmatically provision and configure

    components http://www.flickr.com/photos/louisb/4555295187/ 31 Friday, February 21, 14
  23. Chef is Infrastructure as Code • Treat like any other

    code base http://www.flickr.com/photos/louisb/4555295187/ 32 Friday, February 21, 14
  24. Chef is Infrastructure as Code • Reconstruct business from code

    repository, data backup, and compute resources http://www.flickr.com/photos/louisb/4555295187/ 33 Friday, February 21, 14
  25. Chef is Infrastructure as Code • Programmatically provision and configure

    components • Treat like any other code base • Reconstruct business from code repository, data backup, and compute resources http://www.flickr.com/photos/louisb/4555295187/ 34 Friday, February 21, 14
  26. Configuration Code • Chef ensures each Node complies with the

    policy • Policy is determined by the configurations in each Node’s run list • Reduce management complexity through abstraction • Store the configuration of your infrastructure in version control 35 Friday, February 21, 14
  27. Declarative Interface to Resources • You define the policy in

    your Chef configuration • Your policy states what state each resource should be in, but not how to get there • Chef-client will pull the policy from the Chef Server and enforce the policy on the Node 36 Friday, February 21, 14
  28. Managing Complexity • Organizations • Environments • Roles • Nodes

    • Recipes • Cookbooks • Search 37 Friday, February 21, 14
  29. Environments Define Policy • Environments may include data attributes necessary

    for configuring your infrastructure, e.g. • The URL of your payment service’s API • The location of your package repository • The version of the Chef configuration files that should be used 40 Friday, February 21, 14
  30. Roles Define Policy • Roles may include an ordered list

    of Chef configuration files that should be applied • This list is called a Run List • Order is always important in the Run List • Roles may include data attributes necessary for configuring your infrastructure, for example: • The port that the application server listens on • A list of applications that should be deployed 42 Friday, February 21, 14
  31. Node • Each Node will • Belong to one Organization

    • Belong to one Environment • Have zero or more Roles 44 Friday, February 21, 14
  32. Nodes Adhere to Policy • The chef-client application runs on

    each node, which • Gathers the current system configuration of the node • Downloads the desired system configuration policies from the Chef server for that node • Configures the node such that it adheres to those policies 45 Friday, February 21, 14
  33. Resources • A Resource represents a piece of the system

    and its desired state • A package that should be installed • A service that should be running • A file that should be generated • A cron job that should be configured • A user that should be managed • and more 46 Friday, February 21, 14
  34. Resources in Recipes • Resources are the fundamental building blocks

    of Chef configuration • Resources are gathered into Recipes • Recipes ensure the system is in the desired state 47 Friday, February 21, 14
  35. Recipes • Configuration files that describe resources and their desired

    state • Recipes can: • Install and configure software components • Manage files • Deploy applications • Execute other recipes • and more 48 Friday, February 21, 14
  36. Example Recipe package "apache2" template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner

    "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end service "apache2" do action [:enable,:start] supports :reload => true end 49 Friday, February 21, 14
  37. Cookbooks • Recipes are stored in Cookbooks • Cookbooks contain

    recipes, templates, files, custom resources, etc • Code re-use and modularity http://www.flickr.com/photos/shutterhacks/4474421855/ 50 Friday, February 21, 14
  38. Run List Node Enterprise Chef chef-client What policy should I

    follow? "recipe[ntp::client]" "recipe[users]" "role[webserver]" 52 Friday, February 21, 14
  39. Run List Enterprise Chef chef-client What policy should I follow?

    "recipe[ntp::client]" "recipe[users]" "role[webserver]" 53 Friday, February 21, 14
  40. Run List Specifies Policy • The Run List is an

    ordered collection of policies that the Node should follow • Chef-client obtains the Run List from the Chef Server • Chef-client ensures the Node complies with the policy in the Run List 54 Friday, February 21, 14
  41. Search • Search for nodes with Roles • Find Topology

    Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625 55 Friday, February 21, 14
  42. Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source

    "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end 56 Friday, February 21, 14
  43. Search for Nodes pool_members = search("node","role:webserver") template "/etc/haproxy/haproxy.cfg" do source

    "haproxy-app_lb.cfg.erb" owner "root" group "root" mode 0644 variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end 57 Friday, February 21, 14
  44. Pass results into Templates # Set up application listeners here.

    listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> 58 Friday, February 21, 14
  45. Pass results into Templates # Set up application listeners here.

    listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> 59 Friday, February 21, 14
  46. # Set up application listeners here. listen application 0.0.0.0:80 balance

    roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>:> weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass results into Templates 60 Friday, February 21, 14
  47. Memcache Postgres Slaves Postgres Master Nagios Graphite Jboss App Memcache

    Postgres Slaves Postgres Master Nagios Graphite ...this can happen automatically 63 Friday, February 21, 14
  48. Nagios Graphite Nagios Graphite Memcache Postgres Slaves • Load balancer

    config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the Resources Jboss App 64 Friday, February 21, 14
  49. Manage Complexity • Determine the desired state of your infrastructure

    • Identify the Resources required to meet that state • Gather the Resources into Recipes • Compose a Run List from Recipes and Roles • Apply a Run List to each Node in your Environment • Your infrastructure adheres to the policy modeled in Chef 65 Friday, February 21, 14
  50. Configuration Drift • Configuration Drift happens when: • Your infrastructure

    requirements change • The configuration of a server falls out of policy • Chef makes it easy to manage • Model the new requirements in your Chef configuration files • Run the chef-client to enforce your policies 66 Friday, February 21, 14
  51. Lesson Objectives • After completing the lesson, you will be

    able to • Login to Enterprise Chef • View your Organization in Enterprise Chef • Describe Knife, the Chef command line utility • Use Knife on your Workstation 68 Friday, February 21, 14
  52. Legend: Do I run that command on my workstation? $

    whoami i-am-a-workstation This is an example of a command to run on your workstation user@hostname:~$ whoami i-am-a-chef-node This is an example of a command to run on your target node via SSH. 70 Friday, February 21, 14
  53. $ ifconfig Legend: Example Terminal Command and Output lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST>

    mtu 16384 ! options=3<RXCSUM,TXCSUM> ! inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 ! inet 127.0.0.1 netmask 0xff000000 ! inet6 ::1 prefixlen 128 gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ! ether 28:cf:e9:1f:79:a3 ! inet6 fe80::2acf:e9ff:fe1f:79a3%en0 prefixlen 64 scopeid 0x4 ! inet 10.100.0.84 netmask 0xffffff00 broadcast 10.100.0.255 ! media: autoselect ! status: active p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304 ! ether 0a:cf:e9:1f:79:a3 ! media: autoselect ! status: inactive 71 Friday, February 21, 14
  54. OPEN IN EDITOR: SAVE FILE! ~/hello_world Hi! I am a

    friendly file. Legend: Example of editing a file on your workstation 72 Friday, February 21, 14
  55. Install Chef • Install Chef (if not already installed) •

    http://www.getchef.com/chef/install 75 Friday, February 21, 14
  56. $ curl -L http://www.opscode.com/chef/install.sh | sudo bash Workstation Setup -

    Mac OS X / Linux % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 14100 100 14100 0 0 8814 0 0:00:01 0:00:01 --:--:-- 8812 Downloading Chef for mac_os_x... downloading http://www.getchef.com/chef/metadata?v=&prerelease=false&p=mac_os_x&pv=10.7&m=x86_64 to file /tmp/install.sh.79770/metadata.txt trying curl... url! http://opscode-omnibus-packages.s3.amazonaws.com/mac_os_x/10.7/x86_64/chef-11.8.2_1.mac_os_x. 10.7.2.sh md5! af157c6ef941e52f69a9dd6a3b57f597 sha256! c003c0951d80245b1f02c4588f9157a55e7b94a00dd6ac163aed8ef2e854619a downloaded metadata file looks valid... downloading http://opscode-omnibus-packages.s3.amazonaws.com/mac_os_x/10.7/x86_64/ chef-11.8.2_1.mac_os_x.10.7.2.sh to file /tmp/install.sh.79770/chef--mac_os_x-10.7-x86_64.sh trying curl... ............................................................................ Thank you for installing Chef! 79 Friday, February 21, 14
  57. Workstation Setup - Windows • Windows • 2008 (Windows 7)

    or 2012 (Windows 8) • i686 (32-bit) or x86_64 (64-bit) • 11.10.4 Download and install this file 80 Friday, February 21, 14
  58. What just happened? • Chef and all of its dependencies

    installed via an operating system-specific package ("omnibus installer") • Installation includes • The Ruby language - used by Chef • knife - Command line tool for administrators • chef-client - Client application • ohai - System profiler • ...and more 82 Friday, February 21, 14
  59. Your Chef Server for this class... • Hosted Enterprise Chef

    http://www.getchef.com 84 Friday, February 21, 14
  60. Create new account • Sign up for a new account

    • Chef Organization • provides multi-tenancy • name must be globally unique 85 Friday, February 21, 14
  61. Download "Starter Kit" for your Org • You get a

    .zip file from clicking this • Unzip the zipfile - you’ll get a "chef- repo" • Put the "chef-repo" somewhere, e.g.: • C:\Users\you\chef-repo (Win) • /Users/you/chef-repo (Mac) • /home/you/chef-repo (Linux) 87 Friday, February 21, 14
  62. Knife is the command-line tool for Chef • Knife provides

    an API interface between a local Chef repository and the Chef Server, and lets you manage: • Nodes • Cookbooks and recipes • Roles • Stores of JSON data (data bags), including encrypted data • Environments • Cloud resources, including provisioning • The installation of Chef on management workstations • Searching of indexed data on the Chef Server 88 Friday, February 21, 14
  63. A quick tour of the chef-repo • Every infrastructure managed

    with Chef has a Chef Repository ("chef-repo") • Type all commands in this class from the chef-repo directory • Let’s see what’s inside the chef-repo... 89 Friday, February 21, 14
  64. $ ls -al Tour the chef-repo total 40 drwxr-xr-x@ 11

    opscode opscode 374 Dec 15 09:42 . drwxr-xr-x+ 92 opscode opscode 3128 Dec 15 09:43 .. drwxr-xr-x@ 3 opscode opscode 102 Dec 15 2013 .berkshelf drwxr-xr-x@ 5 opscode opscode 170 Dec 15 2013 .chef -rw-r--r--@ 1 opscode opscode 495 Dec 15 2013 .gitignore -rw-r--r--@ 1 opscode opscode 1433 Dec 15 2013 Berksfile -rw-r--r--@ 1 opscode opscode 2416 Dec 15 2013 README.md -rw-r--r--@ 1 opscode opscode 3567 Dec 15 2013 Vagrantfile -rw-r--r--@ 1 opscode opscode 588 Dec 15 2013 chefignore drwxr-xr-x@ 3 opscode opscode 102 Dec 15 2013 cookbooks drwxr-xr-x@ 3 opscode opscode 102 Dec 15 2013 roles 91 Friday, February 21, 14
  65. What’s inside the .chef directory? • knife.rb is the configuration

    file for Knife. • The other two files are certificates for authentication with the Chef Server • We’ll talk more about that later. 93 Friday, February 21, 14
  66. • Your version may be different, that’s ok! Verify Knife

    $ knife --version Chef: 11.10.4 $ knife client list ORGNAME-validator 94 Friday, February 21, 14
  67. Best Practice: Use a text editor with a project drawer,

    or equivalent • Chef is about Infrastructure as Code • People who code for a living use text editors that are designed for the task • Vim, Emacs, Sublime Text, Notepad++, etc. 95 Friday, February 21, 14
  68. Benefits of a good text editor • A good editor

    will • Show line numbers • Highlight syntax • Autocomplete commands • Allow you to open multiple files 96 Friday, February 21, 14
  69. If you do not have a preferred text editor on

    your workstation already... • Download Sublime Text • Free trial, not time bound • Works on every platform • sublimetext.com 97 Friday, February 21, 14
  70. $ git init Initialize a git repository Initialized empty Git

    repository in / Users/opscode/chef-repo/.git/ 101 Friday, February 21, 14
  71. $ git status Check git status # On branch master

    # # Initial commit # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # .berkshelf/ # .chef/ # .gitignore # Berksfile # README.md # Vagrantfile # chefignore # cookbooks/ # roles/ nothing added to commit but untracked files present (use "git add" to track) 102 Friday, February 21, 14
  72. $ git commit -m "add the starter kit from Chef"

    Add files to git [master (root-commit) 2af68fb] add the starter kit from Chef 13 files changed, 360 insertions(+) create mode 100644 .berkshelf/config.json create mode 100644 .chef/knife.rb create mode 100644 .gitignore create mode 100644 Berksfile create mode 100644 README.md create mode 100644 Vagrantfile create mode 100644 chefignore create mode 100644 cookbooks/starter/attributes/default.rb create mode 100644 cookbooks/starter/files/default/sample.txt create mode 100644 cookbooks/starter/metadata.rb create mode 100644 cookbooks/starter/recipes/default.rb create mode 100644 cookbooks/starter/templates/default/sample.erb create mode 100644 roles/starter.rb 104 Friday, February 21, 14
  73. Lesson Objectives • After completing the lesson, you will be

    able to • Install Chef nodes using "knife bootstrap" • Explain how knife bootstrap configures a node to use the Organization created in the previous section • Explain the basic configuration needed to run chef- client 107 Friday, February 21, 14
  74. Training Node • The labs require a node to be

    managed • We allow for two different options • Bring your own Node • Use the CloudShare training environment 110 Friday, February 21, 14
  75. Bring Your Own Node • Use your own Virtual Machine

    (VM) or Server • Required for the labs: • CentOS 6+ • 512 MB RAM • 15 GB Disk • sudo or root level permissions 111 Friday, February 21, 14
  76. CloudShare Node • Register and login to CloudShare • Start

    Using This Environment 113 Friday, February 21, 14
  77. $ ssh chef@<EXTERNAL_ADDRESS> Lab - Login The authenticity of host

    'uvo1qrwls0jdgs3blvt.vm.cld.sr (69.195.232.110)' can't be established. RSA key fingerprint is d9:95:a3:b9:02:27:e9:cd: 74:e4:a2:34:23:f5:a6:8b. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'uvo1qrwls0jdgs3blvt.vm.cld.sr, 69.195.232.110' (RSA) to the list of known hosts. [email protected]'s password: Last login: Mon Jan 6 16:26:24 2014 from host86-145-117-53.range86-145.btcentralplus.com [chef@CentOS63 ~]$ 116 Friday, February 21, 14
  78. Checkpoint • At this point you should have • One

    virtual machine (VM) or server that you’ll use for the lab exercises • The IP address or public hostname • An application for establishing an ssh connection • 'sudo' or 'root' permissions on the VM 117 Friday, February 21, 14
  79. $ knife bootstrap <EXTERNAL_ADDRESS> --sudo -x chef -P chef -N

    "node1" "Bootstrap" the Target Instance Bootstrapping Chef on uvo1qrwls0jdgs3blvt.vm.cld.sr uvo1qrwls0jdgs3blvt.vm.cld.sr knife sudo password: Enter your password: ... ... uvo1qrwls0jdgs3blvt.vm.cld.sr Creating a new client identity for node1 using the validator key. uvo1qrwls0jdgs3blvt.vm.cld.sr resolving cookbooks for run list: [] uvo1qrwls0jdgs3blvt.vm.cld.sr Synchronizing Cookbooks: uvo1qrwls0jdgs3blvt.vm.cld.sr Compiling Cookbooks... uvo1qrwls0jdgs3blvt.vm.cld.sr [2014-01-28T11:03:14-05:00] WARN: Node node2 has an empty run list. uvo1qrwls0jdgs3blvt.vm.cld.sr Converging 0 resources uvo1qrwls0jdgs3blvt.vm.cld.sr Chef Client finished, 0 resources updated 119 Friday, February 21, 14
  80. local workstation managed node (VM) $ knife bootstrap IPADDRESS --sudo

    -x USERNAME -P PASSWORD -N node1 120 Friday, February 21, 14
  81. local workstation managed node (VM) $ knife bootstrap IPADDRESS --sudo

    -x USERNAME -P PASSWORD -N node1 SSH! 120 Friday, February 21, 14
  82. local workstation managed node (VM) $ knife bootstrap IPADDRESS --sudo

    -x USERNAME -P PASSWORD -N node1 chef_server_url validation_client_name validation_key SSH! 120 Friday, February 21, 14
  83. local workstation managed node (VM) $ knife bootstrap IPADDRESS --sudo

    -x USERNAME -P PASSWORD -N node1 Opscode Hosted Chef SSH! 120 Friday, February 21, 14
  84. local workstation managed node (VM) $ knife bootstrap IPADDRESS --sudo

    -x USERNAME -P PASSWORD -N node1 Opscode Hosted Chef SSH! bash -c ' install chef configure client run chef' 120 Friday, February 21, 14
  85. local workstation managed node (VM) chef-client $ knife bootstrap IPADDRESS

    --sudo -x USERNAME -P PASSWORD -N node1 Opscode Hosted Chef SSH! 120 Friday, February 21, 14
  86. What just happened? • Chef and all of its dependencies

    installed via an operating system-specific package ("omnibus installer") • Installation includes • The Ruby language - used by Chef • knife - Command line tool for administrators • chef-client - Client application • ohai - System profiler • ...and more 121 Friday, February 21, 14
  87. Verify Your Target Instance’s Chef-Client is Configured Properly $ ssh

    chef@<EXTERNAL_ADDRESS> chef@node1:~$ ls /etc/chef client.pem client.rb first-boot.json validation.pem chef@node1:~$ which chef-client /usr/bin/chef-client 122 Friday, February 21, 14
  88. View Node on Chef Server • Login to your Hosted

    Enterprise Chef 123 Friday, February 21, 14
  89. Node • The node is registered with Chef Server •

    The Chef Server displays information about the node • This information comes from Ohai - we'll see Ohai later..... 126 Friday, February 21, 14
  90. Lesson Objectives • After completing the lesson, you will be

    able to • Describe in detail what a cookbook is • Create a new cookbook • Explain what a recipe is • Describe how to use the package, service, and cookbook_file resources • Upload a cookbook to the Chef Server • Explain what a run list is, and how to set it for a node via knife • Explain the output of a chef-client run 129 Friday, February 21, 14
  91. What is a cookbook? • A cookbook is like a

    “package” for Chef recipes. • It contains all the recipes, files, templates, libraries, etc. required to configure a portion of your infrastructure • Typically they map 1:1 to a piece of software or functionality. 130 Friday, February 21, 14
  92. The Problem and the Success Criteria • The Problem: We

    need a web server configured to serve up our home page. • Success Criteria: We can see the homepage in a web browser. 131 Friday, February 21, 14
  93. Required steps • Install Apache • Start the service, and

    make sure it will start when the machine boots • Write out the home page • Please note in this course we're teaching Chef primitives, not web server management • This is probably not the Apache HTTP server configuration you would use in production 132 Friday, February 21, 14
  94. $ knife cookbook create apache Exercise: Create a new Cookbook

    ** Creating cookbook apache ** Creating README for cookbook: apache ** Creating CHANGELOG for cookbook: apache ** Creating metadata for cookbook: apache 133 Friday, February 21, 14
  95. $ ls -la cookbooks/apache Exercise: Explore the cookbook total 24

    drwxr-xr-x 13 opscode opscode 442 Jan 24 21:25 . drwxr-xr-x 5 opscode opscode 170 Jan 24 21:25 .. -rw-r--r-- 1 opscode opscode 412 Jan 24 21:25 CHANGELOG.md -rw-r--r-- 1 opscode opscode 1447 Jan 24 21:25 README.md drwxr-xr-x 2 opscode opscode 68 Jan 24 21:25 attributes drwxr-xr-x 2 opscode opscode 68 Jan 24 21:25 definitions drwxr-xr-x 3 opscode opscode 102 Jan 24 21:25 files drwxr-xr-x 2 opscode opscode 68 Jan 24 21:25 libraries -rw-r--r-- 1 opscode opscode 276 Jan 24 21:25 metadata.rb drwxr-xr-x 2 opscode opscode 68 Jan 24 21:25 providers drwxr-xr-x 3 opscode opscode 102 Jan 24 21:25 recipes drwxr-xr-x 2 opscode opscode 68 Jan 24 21:25 resources drwxr-xr-x 3 opscode opscode 102 Jan 24 21:25 templates 134 Friday, February 21, 14
  96. Exercise: Open a project drawer if you're using Sublime Text

    • If you're using Sublime, then File>Open the chef-repo directory you created earlier 135 Access the cookbook files from the left menu Friday, February 21, 14
  97. OPEN IN EDITOR: cookbooks/apache/recipes/default.rb # # Cookbook Name:: apache #

    Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # Exercise: Edit the default recipe 136 Friday, February 21, 14
  98. SAVE FILE! # # Cookbook Name:: apache # Recipe:: default

    # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "httpd" do action :install end Exercise: Add a package resource to install Apache to the default recipe OPEN IN EDITOR: cookbooks/apache/recipes/default.rb 137 Friday, February 21, 14
  99. package "httpd" do action :install end So the resource we

    just wrote... 138 Friday, February 21, 14
  100. package "httpd" do action :install end So the resource we

    just wrote... • Is a package resource 139 Friday, February 21, 14
  101. package "httpd" do action :install end So the resource we

    just wrote... • Is a package resource • Whose name is httpd 140 Friday, February 21, 14
  102. package "httpd" do action :install end So the resource we

    just wrote... • Is a package resource • Whose name is httpd • With an install action 141 Friday, February 21, 14
  103. Notice we didn’t say how to install the package •

    Resources are declarative - that means we say what we want to have happen, rather than how • Resources take action through Providers - providers perform the how • Chef uses the platform the node is running to determine the correct provider for a resource 142 Friday, February 21, 14
  104. Package Resource package "git" { yum install git apt-get install

    git pacman sync git pkg_add -r git Providers are determined by node's platform 143 Friday, February 21, 14
  105. SAVE FILE! ... # All rights reserved - Do Not

    Redistribute # package "httpd" do action :install end service "httpd" do action [ :enable, :start ] end Exercise: Add a service resource to ensure the service is started and enabled at boot OPEN IN EDITOR: cookbooks/apache/recipes/default.rb 144 Friday, February 21, 14
  106. service "httpd" do action [ :enable, :start ] end So

    the resource we just wrote... 145 Friday, February 21, 14
  107. service "httpd" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource 146 Friday, February 21, 14
  108. service "httpd" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource • Whose name is httpd 147 Friday, February 21, 14
  109. service "httpd" do action [ :enable, :start ] end So

    the resource we just wrote... • Is a service resource • Whose name is httpd • With two actions: • enable • start 148 Friday, February 21, 14
  110. • Body Level One • Body Level Two • Body

    Level Three • Body Level Four • Body Level Five Order Matters package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode "0644" notifies :restart, "service[haproxy]" end service "haproxy" do supports :restart => :true action [:enable, :start] end • Resources are executed in order 1st 2nd 3rd 149 Friday, February 21, 14
  111. SAVE FILE! ... service "httpd" do action [ :enable, :start

    ] end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end Exercise: Add a cookbook_file resource to copy the home page in place OPEN IN EDITOR: cookbooks/apache/recipes/default.rb 150 Friday, February 21, 14
  112. So the resource we just wrote... template "/var/www/html/index.html" do source

    "index.html.erb" mode "0644" end 151 Friday, February 21, 14
  113. So the resource we just wrote... • Is a template

    resource template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end 152 Friday, February 21, 14
  114. So the resource we just wrote... • Is a cookbook_file

    resource • Whose name is: /var/www/html/index.html template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end 153 Friday, February 21, 14
  115. So the resource we just wrote... • Is a cookbook_file

    resource • Whose name is: /var/www/html/index.html • With two parameters: • source of index.html.erb • mode of “0644” template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end 154 Friday, February 21, 14
  116. Full contents of the apache recipe # # Cookbook Name::

    apache # Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "httpd" do action :install end service "httpd" do action [ :enable, :start ] end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end 155 Friday, February 21, 14
  117. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <h1>Hello, world!</h1> Exercise: Add

    index.html to your cookbook’s files/default directory 156 Friday, February 21, 14
  118. $ knife cookbook upload apache Exercise: Upload the cookbook Uploading

    apache [0.1.0] Uploaded 1 cookbook. 157 Friday, February 21, 14
  119. The Run List • The Run List is the ordered

    set of recipes and roles that the Chef Client will execute on a node • Recipes are specified by “recipe[name]” • Roles are specified by “role[name]” 158 Friday, February 21, 14
  120. $ knife node run_list add node1 "recipe[apache]" Exercise: Add apache

    recipe to test node’s run list node1: run_list: recipe[apache] 159 Friday, February 21, 14
  121. chef@node1:~$ sudo chef-client Exercise: Run Chef Client Starting Chef Client,

    version 11.10.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[httpd] action install - install version 2.2.15-29.el6.centos of package httpd * service[httpd] action enable - enable service service[httpd] * service[httpd] action start - start service service[httpd] * template[/var/www/html/index.html] action create - create new file /var/www/html/index.html - update content in file /var/www/html/index.html from none to 17d291 160 Friday, February 21, 14
  122. Exercise: Verify that the home page works • Open a

    web browser • Type in the the URL for your test node 161 Friday, February 21, 14
  123. Congratulate yourself! • You have just written your first Chef

    cookbook! • (clap!) 162 Friday, February 21, 14
  124. And now a word from our sponsors! • We’re Hiring!

    getchef.com/careers • Twitter things... • #scale12x • @nathenharvey • #learnchef • #getchef • #hugops 163 Friday, February 21, 14
  125. Lesson Objectives • After completing the lesson, you will be

    able to • List all the steps taken by a chef-client during a run • Explain the basic security model of Chef • Explain the concepts of the Resource Collection 165 Friday, February 21, 14
  126. build node authenticate sync cookbooks load cookbooks converge node.save Yes

    chef-client success? 166 Friday, February 21, 14
  127. build node authenticate sync cookbooks load cookbooks converge node.save exception

    Yes No chef-client success? 166 Friday, February 21, 14
  128. build node authenticate sync cookbooks load cookbooks converge node.save notification

    handlers exception Yes No chef-client success? 166 Friday, February 21, 14
  129. Multiphase Execution - Compile Phase • During the compile phase,

    Chef 1. Loads all cookbooks from the run list 2. Reads each recipe to build the resource collection 168 Friday, February 21, 14
  130. Multiphase Execution - Execute Phase • During the execute phase

    chef takes the resource collection and for each resource it will 1. Check if the resource is in the required state • If 'yes' - do nothing • If 'no' - bring resource in line with required state 2. Move on to next resource 169 Friday, February 21, 14
  131. Resource Collection Phase 1 - Compile Phase 170 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection resource_collection = [ Recipe Friday, February 21, 14
  132. Resource Collection Phase 1 - Compile Phase 170 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection package["httpd"], resource_collection = [ Recipe Friday, February 21, 14
  133. Resource Collection Phase 1 - Compile Phase 170 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection package["httpd"], template["/var/www/html/index.html"], resource_collection = [ Recipe Friday, February 21, 14
  134. Resource Collection Phase 1 - Compile Phase 170 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Recipe Friday, February 21, 14
  135. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Friday, February 21, 14
  136. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Is httpd package currently installed? Do nothing Install httpd package Yes No Friday, February 21, 14
  137. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Is httpd package currently installed? Do nothing Install httpd package Yes No Is '/var/www/html/ index.html' identical to cookbook version? Do nothing Copy index.html file into place Yes No Next Friday, February 21, 14
  138. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Is httpd package currently installed? Do nothing Install httpd package Yes No Does 'index.html' have correct permissions? Do nothing Set permissions to 0644 Yes No Next Is '/var/www/html/ index.html' identical to cookbook version? Do nothing Copy index.html file into place Yes No Next Friday, February 21, 14
  139. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Is httpd package currently installed? Do nothing Install httpd package Yes No Does 'index.html' have correct permissions? Do nothing Set permissions to 0644 Yes No Next Is '/var/www/html/ index.html' identical to cookbook version? Do nothing Copy index.html file into place Yes No Next Is the service 'httpd' enabled to start on boot Do nothing Enable httpd to start on boot Yes No Next Friday, February 21, 14
  140. Resource Collection Phase 2 - Execute Phase 171 package "httpd"

    do action :install end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end service "httpd" do action [ :enable, :start ] end Resource Collection Recipe package["httpd"], template["/var/www/html/index.html"], service ["httpd"] ] resource_collection = [ Execution Is httpd package currently installed? Do nothing Install httpd package Yes No Does 'index.html' have correct permissions? Do nothing Set permissions to 0644 Yes No Next Is '/var/www/html/ index.html' identical to cookbook version? Do nothing Copy index.html file into place Yes No Next Is the service 'httpd' enabled to start on boot Do nothing Enable httpd to start on boot Yes No Next Yes Is the service 'httpd' currently started? Do nothing Start httpd No Next Friday, February 21, 14
  141. Recipe order is important! • Recipes are executed in the

    order they appear in the run list • These recipes are invoked in the following order 172 Run List: recipe[ntp::client], recipe[openssh::server], recipe[apache::server] Friday, February 21, 14
  142. Recipe order is important! • Recipes are executed in the

    order they appear in the run list • These recipes are invoked in the following order 172 1. recipe[ntp::client] 2. recipe[openssh::server] 3. recipe[apache::server] Run List: recipe[ntp::client], recipe[openssh::server], recipe[apache::server] Friday, February 21, 14
  143. template "/etc/ntp.conf" do ! source "ntp.conf.erb" ! owner "root" !

    mode "0644" end service "ntp" do ! action :start end package "ntp" do ! action :install end Resource Collection - Multiple Recipes 173 1. recipe[ntp::client] Resource Collection package[ntp], template[/etc/ntp.conf], service[ntp], resource_collection [ Friday, February 21, 14
  144. Resource Collection - Multiple Recipes 174 2. recipe[openssh::client] Resource Collection

    package[ntp], template[/etc/ntp.conf], service[ntp], resource_collection [ Friday, February 21, 14
  145. template "/etc/sshd/sshd_config" do ! source "sshd_config.erb" ! owner "root" !

    mode "0644" end service "openssh" do ! action :start end package "openssh" do ! action :install end Resource Collection - Multiple Recipes 174 2. recipe[openssh::client] Resource Collection package[ntp], template[/etc/ntp.conf], service[ntp], package[openssh], template[/etc/sshd/sshd_config], service[openssh], resource_collection [ Friday, February 21, 14
  146. Resource Collection - Multiple Recipes 175 3. recipe[httpd::server] Resource Collection

    package[ntp], template[/etc/ntp.conf], service[ntp], package[openssh], template[/etc/sshd/sshd_config], service[openssh], resource_collection [ Friday, February 21, 14
  147. service "httpd" do action [ :enable, :start ] end cookbook_file

    "/var/www/html/index.html" do source "index.html" mode "0644" end package "httpd" do action :install end Resource Collection - Multiple Recipes 175 3. recipe[httpd::server] Resource Collection package[ntp], template[/etc/ntp.conf], service[ntp], package[openssh], template[/etc/sshd/sshd_config], service[openssh], package[httpd], service[httpd], cookbook_file[/var/www/html/index.html] ] resource_collection [ Friday, February 21, 14
  148. The final resource collection • So the resources are invoked

    in the following order during the execute phase 176 package[ntp] template[/etc/ntp.conf] service[ntp] package[openssh] template[/etc/sshd/sshd_config] service[openssh] package[httpd] service[httpd] cookbook_file[/var/www/html/index.html] Friday, February 21, 14
  149. Multiphase Execution • Plain ruby is executed in the compile

    phase • Chef DSL is executed in the execute phase 177 %w[sites-available sites-enabled mods-available mods-enabled].each do |dir| directory "/var/www/#{dir}" do action :create mode '0755' owner 'root' group node['apache']['root_group'] end end Resource Collection Recipe Friday, February 21, 14
  150. Multiphase Execution • Plain ruby is executed in the compile

    phase • Chef DSL is executed in the execute phase 177 %w[sites-available sites-enabled mods-available mods-enabled].each do |dir| directory "/var/www/#{dir}" do action :create mode '0755' owner 'root' group node['apache']['root_group'] end end Resource Collection Recipe resource_collection [ directory["/var/www/sites-available"], directory["/var/www/sites-enabled"], directory["/var/www/mods-available"], directory["/var/www/mods-enabled"] ] Friday, February 21, 14
  151. Lesson Objectives • After completing the lesson, you will be

    able to • Explain what the Node object represents in Chef • Show details about a Node • Describe what Node Attributes are • Retrieve a node attribute 179 Friday, February 21, 14
  152. What is the Node object • A node is any

    physical, virtual, or cloud machines that is configured to be maintained by a Chef • The 'node object' is the representation of that physical node within Chef (e.g. in JSON) • When you are writing Recipes, the Node object is always available to you. 180 Friday, February 21, 14
  153. The Chef Node Object • Nodes are the objects that

    you manage with Chef • They have a few different properties • attributes • run_list • chef_environment Friday, February 21, 14
  154. The Chef Node Object • In client/server Chef, the Chef

    Server stores node object data • It becomes searchable through the API with knife and with recipes • Some of the data comes from ohai, which takes an inventory of the system and emits JSON data • You can add data to the node through attributes in cookbooks, roles, directly on a node, etc Friday, February 21, 14
  155. Node Attributes Syntax • Node attributes are hashes (of hashes)

    • Reference hash keys with square brackets and keys as strings •node["hostname"] •node["kernel"]["machine"] Friday, February 21, 14
  156. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, world!</p> <p>My name

    is <%= node['hostname'] %></p> Update the home page template Friday, February 21, 14
  157. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook. Friday, February 21, 14
  158. opscode@node1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.10.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[httpd] action install (up to date) * service[httpd] action enable (up to date) * service[httpd] action start (up to date) * template[/var/www/html/index.html] action create - update content in file /var/www/html/index.html from 17d291 to 77b6e3 --- /var/www/html/index.html 2014-02-21 08:18:54.710230641 -0500 +++ /tmp/chef-rendered-template20140221-4650-ykt042 2014-02-21 08:41:45.428088171 -0500 @@ -1,2 +1,3 @@ <h1>Hello, world!</h1> +<p>My name is centos63</p> Running handlers: Running handlers complete Friday, February 21, 14
  159. opscode@node1:~$ ohai Run Ohai on the node { "languages": {

    "ruby": { }, "perl": { "version": "5.14.2", "archname": "x86_64-linux-gnu-thread-multi" }, "python": { "version": "2.7.3", "builddate": "Aug 1 2012, 05:14:39" } }, <SNIP> Friday, February 21, 14
  160. $ knife node show node1 Show the node object Node

    Name: node1 Environment: _default FQDN: centos63.example.com IP: 10.160.201.90 Run List: recipe[apache] Roles: Recipes: apache, apache::default Platform: centos 6.4 Tags: Friday, February 21, 14
  161. $ knife node show node1 -a hostname Show specific node

    attribute node1: hostname: centos63 Friday, February 21, 14
  162. Lesson Objectives • After completing the lesson, you will be

    able to • Describe where and how attributes are set • Explain the attribute merge order and precedence rules • Declare an attribute with a recipe and set its value 193 Friday, February 21, 14
  163. What are Attributes? • Attributes represents information from your node

    as well as input variables in your cookbooks • Attributes hold static node data (e.g. platform, memory installed, disk space, etc) • Allows you to abstract out data in from cookbooks (e.g. directory paths, users, software version numbers) • All attributes are set on the "node object", and are indexed for search on the server 194 Friday, February 21, 14
  164. Attribute Sources • Attributes can be set at various levels

    (in increasing order of precedence) • Automagically on the node itself (by Ohai) • In roles • In environments • In cookbook recipes • In cookbook attribute files 195 Friday, February 21, 14
  165. Ohai - set automatically "languages": { "ruby": { }, "perl":

    { "version": "5.14.2", "archname": "x86_64- linux-gnu-thread-multi" }, "python": { "version": "2.6.6", "builddate": "Jul 10 2013, 22:48:45" }, "perl": { "version": "5.10.1", "archname": "x86_64- linux-thread-multi" } }, "network": { "interfaces": { "lo": { "mtu": "16436", "flags": [ "LOOPBACK", "UP","LOWER_UP" ], "encapsulation": "Loopback", "addresses": { "127.0.0.1": { "family": "inet", "netmask": "255.0.0.0", "scope": "Node" }, "::1": { "family": "inet6", "scope": "Node" } }, }, "eth0": { "type": "eth", "number": "0", "kernel": { "name": "Linux", "release": "3.2.0-32-virtual", "version": "#1 SMP Wed Oct 16 18:37:12 UTC 2013", "machine": "x86_64", "modules": { "isofs": { "size": "70066", "refcount": "2" }, "des_generic": { "size": "16604", "refcount": "0" } }, "os": "GNU/Linux" }, "os": "linux", "os_version": "2.6.32-358.23.2.el6.x86_64", "ohai_time": 1389105685.7735305, 196 Friday, February 21, 14
  166. Setting attributes in attribute files • Attributes can be set

    in the cookbook's attributes file • ./cookbooks/<cookbook>/attributes/default.rb • Format is • We'll look at precedence later.... precedence attribute name attribute value default['apache']['dir'] = "/etc/apache2" 197 Friday, February 21, 14
  167. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p>

    <p>My name is <%= node['hostname'] %></p> Update the home page template Friday, February 21, 14
  168. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook. Friday, February 21, 14
  169. opscode@node1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.10.4 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[httpd] action install (up to date) * service[httpd] action enable (up to date) * service[httpd] action start (up to date) * template[/var/www/html/index.html] action create - update content in file /var/www/html/index.html from 77b6e3 to 455d5b --- /var/www/html/index.html 2014-02-21 08:41:45.428088171 -0500 +++ /tmp/chef-rendered-template20140221-4957-1j4w9sg 2014-02-21 08:47:51.921529076 -0500 @@ -1,3 +1,3 @@ -<h1>Hello, world!</h1> +<h1>Hello, World!</h1> <p>My name is centos63</p> Friday, February 21, 14
  170. Checkpoint • We have a node attribute • node["apache"]["greeting"] •

    We've updated the index.html template to use this attribute. • This will be used again soon! Friday, February 21, 14
  171. Exercise: Verify new homepage works • Open a web browser

    • The homepage takes the attribute file value 203 Friday, February 21, 14
  172. Lesson Objectives • After completing the lesson, you will be

    able to • Use Data Bags for data-driven recipes • Use multiple recipes for a node's run list • Control execution of arbitrary commands with Chef's resource guards Friday, February 21, 14
  173. Data Bags are generic stores of information • Data bags

    are generic, arbitrary stores of information about the infrastructure. • Data Bag Items are JSON data • Our apache cookbook provides a good baseline • We'll drive site-specific virtual hosts with data bags Friday, February 21, 14
  174. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/bears.json { "id" : "bears",

    "port" : 80 } Add a Data Bag Item Friday, February 21, 14
  175. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/clowns.json { "id" : "clowns",

    "port" : 81 } Add a Data Bag Item Friday, February 21, 14
  176. $ knife upload data_bags/vhosts Upload the data bags Created data_bags/vhosts

    Created data_bags/vhosts/bears.json Created data_bags/vhosts/clowns.json Friday, February 21, 14
  177. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/default.rb service "httpd" do action

    [ :enable, :start ] end execute "mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.disabled" do only_if do File.exist?("/etc/httpd/conf.d/welcome.conf") end notifies :restart, "service[httpd]" end template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end Disable the default apache site Friday, February 21, 14
  178. template "/var/www/html/index.html" do source "index.html.erb" mode "0644" end OPEN IN

    EDITOR: /cookbooks/apache/recipes/default.rb SAVE FILE! Delete the template resource • Remove this resource from the recipe Friday, February 21, 14
  179. $ knife diff cookbooks/apache Diff the cookbook diff --knife cookbooks/apache/recipes/default.rb

    cookbooks/apache/recipes/default.rb --- cookbooks/apache/recipes/default.rb 2014-02-21 06:00:07.000000000 -0800 +++ cookbooks/apache/recipes/default.rb 2014-02-21 06:00:07.000000000 -0800 @@ -14,8 +14,11 @@ action [ :enable, :start ] end -template "/var/www/html/index.html" do - source "index.html.erb" - mode "0644" +execute "mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.disabled" do + only_if do + File.exist?("/etc/httpd/conf.d/welcome.conf") + end + notifies :restart, "service[httpd]" end Friday, February 21, 14
  180. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook. Friday, February 21, 14
  181. A new recipe for virtual hosts • We'll create an

    apache::vhosts recipe to manage the virtual hosts we created in data bag items • There's a number of new things to talk about in this recipe • We'll take this nice and slow :) Friday, February 21, 14
  182. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb data_bag("vhosts").each do |site| site_data

    = data_bag_item("vhosts", site) site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" end Create a vhosts recipe Friday, February 21, 14
  183. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb document_root = "/srv/apache/#{site_name}" template

    "/etc/httpd/conf.d/#{site_name}.conf" do source "custom-vhosts.erb" mode "0644" variables( :document_root => document_root, :port => site_data["port"] ) notifies :restart, "service[httpd]" end end Add a Virtual Hosts Configuration Template Friday, February 21, 14
  184. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end directory document_root do

    mode "0755" recursive true end end Add a directory resource Friday, February 21, 14
  185. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end template "#{document_root}/index.html" do

    source "index.html.erb" mode "0644" variables( :site_name => site_name, :port => site_data["port"] ) end end Index for each vhost Friday, February 21, 14
  186. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end template "#{document_root}/index.html" do

    source "index.html.erb" mode "0644" variables( :site_name => site_name, :port => site_data["port"] ) end end Index for each vhost https://gist.github.com/9134977 Friday, February 21, 14
  187. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/custom-vhosts.erb <% if @port !=

    80 -%> Listen <%= @port %> <% end -%> <VirtualHost *:<%= @port %>> ServerAdmin webmaster@localhost DocumentRoot <%= @document_root %> <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory <%= @document_root %>> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> </VirtualHost> Index for each vhost Friday, February 21, 14
  188. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/custom-vhosts.erb <% if @port !=

    80 -%> Listen <%= @port %> <% end -%> <VirtualHost *:<%= @port %>> ServerAdmin webmaster@localhost DocumentRoot <%= @document_root %> <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory <%= @document_root %>> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> </VirtualHost> Index for each vhost https://gist.github.com/2866454 Friday, February 21, 14
  189. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p>

    <p>My name is <%= node['hostname'] %></p> <p>We love <%= @site_name %></p> <p>Served from <%= node['ipaddress'] %>:<%= @port %></p> Update the index.html template Friday, February 21, 14
  190. $ knife diff cookbooks/apache Diff the cookbook diff --knife cookbooks/apache/templates/default/index.html.erb

    cookbooks/apache/templates/default/ index.html.erb --- cookbooks/apache/templates/default/index.html.erb 2014-02-21 06:02:53.000000000 -0800 +++ cookbooks/apache/templates/default/index.html.erb 2014-02-21 06:02:53.000000000 -0800 @@ -1,3 +1,5 @@ <h1>Hello, <%= node['apache']['greeting'] %>!</h1> <p>My name is <%= node['hostname'] %></p> +<p>We love <%= @site_name %></p> +<p>Served from <%= node['ipaddress'] %>:<%= @port %></p> diff --knife cookbooks/apache/templates/default/custom-vhosts.erb cookbooks/apache/templates/ default/custom-vhosts.erb new file --- /dev/null 2014-02-21 06:02:53.000000000 -0800 +++ cookbooks/apache/templates/default/custom-vhosts.erb 2014-02-21 06:02:53.000000000 -0800 Friday, February 21, 14
  191. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook. Friday, February 21, 14
  192. $ knife node run list add node1 "recipe[apache::vhosts]" Update the

    run list node1: run_list: recipe[apache] recipe[apache::vhosts] Friday, February 21, 14
  193. opscode@node1:~$ sudo chef-client Run the chef-client on your test node

    Starting Chef Client, version 11.10.4 resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 9 resources Recipe: apache::default * package[httpd] action install (up to date) * service[httpd] action enable (up to date) * service[httpd] action start (up to date) * execute[mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.disabled] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/httpd/conf.d/bears.conf] action create - create new file /etc/httpd/conf.d/bears.conf - update content in file /etc/httpd/conf.d/bears.conf from none to 416948 --- /etc/httpd/conf.d/bears.conf 2014-02-21 09:20:53.592830069 -0500 +++ /tmp/chef-rendered-template20140221-6294-y855dq 2014-02-21 09:20:53.594830068 -0500 Friday, February 21, 14
  194. Think about what we just did... • We had two

    virtual hosts... Friday, February 21, 14
  195. Think about what we just did... • We had two

    virtual hosts... • But we could arbitrarily add more... Friday, February 21, 14
  196. Think about what we just did... • We had two

    virtual hosts... • But we could arbitrarily add more... • Tigers on port 82, Lions on port 83, oh my! Friday, February 21, 14
  197. Checkpoint • Our cookbook has two recipes, default and vhosts

    • Additional data bags can be added, expanding our Virtual Hosting empire! Friday, February 21, 14
  198. Lesson Objectives • After completing the lesson, you will be

    able to • Explain what Roles are, and how they are used to provide clarity • Discuss the Role JSON DSL • Explain how merge order affects the precedence hierarchy 230 Friday, February 21, 14
  199. What is a Role? • So far, we’ve been just

    adding recipes directly to a single node • But that’s not how your infrastructure works - think about how you refer to servers • "It’s a web server" • "It’s a database server" • "It’s a monitoring server" 231 Friday, February 21, 14
  200. What is a Role? • Roles allow you to conveniently

    encapsulate the run lists and attributes required for a server to "be" what you already think it is • In practice, Roles make it easy to configure many nodes identically without repeating yourself each time 232 Friday, February 21, 14
  201. OPEN IN EDITOR: SAVE FILE! roles/webserver.json { "name" : "webserver",

    "default_attributes" : { "apache" : { "greeting" : "SCaLE12x" } }, "run_list" : [ "recipe[apache]", "recipe[apache::vhosts]" ] } Create a webserver role Friday, February 21, 14
  202. Components of a role • Roles must have a name

    • Roles may have a description • Roles may have a run_list, just like a node • Roles may set node attributes • default_attributes • override_attributes http://docs.opscode.com/essentials_roles.html Friday, February 21, 14
  203. $ knife role from file webserver.json Exercise: Create the role

    Updated Role webserver! 235 Friday, February 21, 14
  204. $ knife node run list remove node1 "recipe[apache::vhosts]" Remove the

    vhosts recipe node1: run_list: recipe[apache] Friday, February 21, 14
  205. $ knife node run list remove node1 "recipe[apache]" Remove the

    apache recipe node1: run_list: Friday, February 21, 14
  206. $ knife node run list add node1 "role[webserver]" Add the

    webserver role node1: run_list: role[webserver] Friday, February 21, 14
  207. Attributes can be set multiple places • This is for

    flexibility • Set a "sane default" that will be used in a cookbook • Override it easily in a role (higher priority) • In all, there are 15 places where attributes come from (!!) • In practice, you'll use 2-3 most of the time. • The others are there when you need them. • http://docs.opscode.com/chef_overview_attributes.html Friday, February 21, 14
  208. Data Bags Are Not Attributes • Important: data bag items

    are not attributes • Data bags are a separate API end point • Data bags are not tied to a specific node or role in the infrastructure • Not even necessarily tied to anything, just data you want to store Friday, February 21, 14
  209. Checkpoint • We now have a webserver role. • We

    could apply just this role on more nodes to scale out our Virtual Hosting service. • Roles are a great way to assign attributes for specific purposes Friday, February 21, 14
  210. opscode@node1:~$ sudo chef-client Run chef-client * template[/srv/apache/bears/index.html] action create -

    update content in file /srv/apache/bears/index.html from d8f475 to 60fcf9 --- /srv/apache/bears/index.html 2014-02-21 09:20:53.870834436 -0500 +++ /tmp/chef-rendered-template20140221-6541-m57zkc 2014-02-21 09:28:53.857331666 -0500 @@ -1,4 +1,4 @@ -<h1>Hello, World!</h1> +<h1>Hello, SCaLE12x!</h1> <p>My name is centos63</p> <p>We love bears</p> <p>Served from 10.160.201.90:80</p> * template[/etc/httpd/conf.d/clowns.conf] action create (up to date) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create - update content in file /srv/apache/clowns/index.html from 828461 to d9b384 --- /srv/apache/clowns/index.html 2014-02-21 09:20:53.883834816 -0500 Friday, February 21, 14
  211. Chef's Search Feature • Search ties together the infrastructure topology

    • We can now search the Chef Server for all the "webserver" nodes • This is relevant for a variety of reasons • Load balancing several front ends (search from nginx, or haproxy for example) • Monitoring HTTP (search from nagios cookbook) • Graphing traffic (search from munin cookbook) Friday, February 21, 14
  212. However, we only have one node... • We don't have

    an environment that is really setup for doing search • But we will talk about what kind of search we do with the command-line, or in a recipe Friday, February 21, 14
  213. $ knife search node "role:webserver" Search for webservers with knife

    1 items found Node Name: node1 Environment: _default FQDN: centos63.example.com IP: 10.160.201.90 Run List: role[webserver] Roles: webserver Recipes: apache, apache::vhosts, apache::default Platform: centos 6.4 Tags: Friday, February 21, 14
  214. Search from a recipe webservers = search(:node, "role:webserver") # or

    search(:node, "role:webserver").each do |web| #things do do on all web servers... end Friday, February 21, 14
  215. Available search indexes • The Chef Server indexes JSON data

    for the major API endpoints: • node • client • environment • role • All data bags are also indexed for search... • vhosts (our example) Friday, February 21, 14
  216. $ knife search vhosts "port:81" Search for webservers with knife

    1 items found chef_type: data_bag_item data_bag: vhosts id: clowns port: 81 Friday, February 21, 14
  217. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb search(:vhosts).each do |site_data| site_name

    = site_data["id"] document_root = "/srv/apache/#{site_name}" Refactor the vhosts recipe Friday, February 21, 14
  218. $ knife diff cookbooks/apache View the diff diff --knife cookbooks/apache/recipes/vhosts.rb

    cookbooks/apache/recipes/ vhosts.rb --- cookbooks/apache/recipes/vhosts.rb 2014-02-21 06:34:42.000000000 -0800 +++ cookbooks/apache/recipes/vhosts.rb 2014-02-21 06:34:42.000000000 -0800 @@ -1,5 +1,4 @@ -data_bag("vhosts").each do |site| - site_data = data_bag_item("vhosts", site) +search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" Friday, February 21, 14
  219. $ knife cookbook upload apache Upload the cookbook Uploading apache

    [0.1.0] Uploaded 1 cookbook. Friday, February 21, 14
  220. opscode@node1:~$ sudo chef-client Run chef-client Starting Chef Client, version 11.10.4

    resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 9 resources Recipe: apache::default * package[httpd] action install (up to date) * service[httpd] action enable (up to date) * service[httpd] action start (up to date) * execute[mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.disabled] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/httpd/conf.d/bears.conf] action create (up to date) * directory[/srv/apache/bears] action create (up to date) * template[/srv/apache/bears/index.html] action create (up to date) * template[/etc/httpd/conf.d/clowns.conf] action create (up to date) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create (up to date) Running handlers: Running handlers complete Chef Client finished, 0/9 resources updated in 7.422708817 seconds Friday, February 21, 14
  221. Our cookbook... • Chef cookbook "apache" with two recipes: •

    default (manages apache package and service) • vhosts (iterates over data bags and renders vhost configuration) • However, Chef publishes an "apache2" cookbook that manages much much more, including all apache2 configuration, modules, sites Debian style w/ a2ensite/a2enmod, etc Friday, February 21, 14
  222. Our role • webserver role in JSON • Sets a

    default attribute • Sets a run list • Uploaded with knife upload Friday, February 21, 14
  223. Our data bag • The vhosts data bag serves as

    an example • Directory structure follows the data bag API end point and "knife upload" makes it easy to upload everything • Create additional vhosts to see how this is dynamically expanded easily Friday, February 21, 14
  224. Chef Server • Hosted Enterprise Chef • Free to manage

    up to five nodes Friday, February 21, 14
  225. Using A Different Chef Server • Configure knife by modifying

    the chef_server_url and the node_name values in ./chef/knife.rb • Get the validation client key from the Chef Server (this differs by implementation): • Open Source: •/etc/chef-server/chef-validator.pem • Hosted Chef / Private Chef • Download after creating an organization Friday, February 21, 14
  226. $ cat .chef/knife.rb The knife.rb configuration # See http://docs.opscode.com/config_rb_knife.html for

    more information on knife configuration options current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name "nathenharveyrmr1" client_key "#{current_dir}/nathenharveyrmr1.pem" validation_client_name "nhrmr1-validator" validation_key "#{current_dir}/nhrmr1-validator.pem" chef_server_url "https://api.opscode.com/organizations/nhrmr1" cache_type 'BasicFile' cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) cookbook_path ["#{current_dir}/../cookbooks"] Friday, February 21, 14
  227. Chef Resources (Core Chef) • Chef client comes with 24+

    different resources • Packages, files, services, users, symlinks, registry keys, and more • Each resource has one or more providers • Some resources have platform-specific providers (e.g., package, service, user, group) http://docs.opscode.com/resource.html Friday, February 21, 14
  228. Managing files and directories • file • cookbook_file • remote_file

    • template • directory • remote_directory • link http://docs.opscode.com/resource.html Friday, February 21, 14
  229. Managing packages • package • apt_package • chef_gem • dpkg_package

    • easy_install_package • freebsd_package • gem_package • ips_package • macports_package • pacman_package • portage_package • rpm_package • smartos_package • solaris_package • yum_package http://docs.opscode.com/resource.html Friday, February 21, 14
  230. RubyGem packages • chef_gem - install a RubyGem into Chef's

    Ruby environment to be used in a Chef recipe • gem_package - install a RubyGem to be used by the system or an application http://docs.opscode.com/resource.html Friday, February 21, 14
  231. Services • service is used to manage services using the

    common init systems available • Each platform has it's own provider • arch, debian, freebsd, gentoo, "init", insserv • invokercd, macosx (launchd), redhat, "simple" • solaris (SMF), upstart, windows http://docs.opscode.com/resource.html Friday, February 21, 14
  232. Other Chef Resources • cron • deploy (revision, timestamped) •

    env (windows) • group • ifconfig (RHEL) • log • mdadm • mount • ohai • registry_key (windows) • route • scm (git, subversion) • user http://docs.opscode.com/resource.html Friday, February 21, 14
  233. There when you need them... • execute (we used this)

    • script (bash, perl, python, csh, ruby interpreters) • windows_script (batch, powershell) - Chef 11.6.0! • ruby_block http://docs.opscode.com/resource.html Friday, February 21, 14
  234. community.opscode.com • apache2, nginx • ark • build-essential • chef-client

    • chruby • cron • line • java • jenkins • minitest-handler • mysql, postgresql • openssh • omnibus_updater • partial_search • runit • whitelist-node-attrs Friday, February 21, 14
  235. Opscode Cookbooks with Chef Resources • Chef has several cookbooks

    that include new custom Chef Resources • apt (apt_repository) • aws (aws_ebs_volume, aws_elastic_ip, and more) • yum (yum_repository) • windows (windows_package, windows_feature, and more) • homebrew (homebrew_package) • runit (runit_service) • many more! http://docs.opscode.com/chef/lwrps_custom.html Friday, February 21, 14
  236. Getting Cookbooks from the Community Site • Knife commands: •knife

    cookbook site download • knife cookbook site install (integrates with git) • Dependency resolvers (a la Ruby's bundler): •librarian-chef •berkshelf Friday, February 21, 14
  237. Knife Plugins • http://docs.opscode.com/community_plugin_knife.html • Cloud plugins (ec2, openstack, rackspace,

    google azure, hpcloud, cloudstack, eucalyptus, and more) • knife-server • knife-solo • knife-preflight • knife-essentials • https://github.com/jkeiser/knife-essentials Friday, February 21, 14
  238. Chef Handlers • http://docs.opscode.com/community_plugin_report_handler.html • IRC • Campfire • HipChat

    • DataDog • Splunk Storm • Graylog2 • Graphite Friday, February 21, 14
  239. Workflow Helpers • Berkshelf: berkshelf.com • Librarian-chef: github.com/applicationsonline/ librarian-chef •

    Knife Spork: github.com/jonlives/knife-spork • Vagrant: vagrantup.com (also a testing tool) Friday, February 21, 14
  240. Cookbook Testing • Vagrant: vagrantup.com • Test Kitchen: kitchen.ci •

    Foodcritic: foodcritic.io • ChefSpec: code.sethvargo.com/chefspec/ • RSpec: rspec.info • serverspec: serverspec.org • bats: github.com/sstephenson/bats Friday, February 21, 14
  241. • April 15 - 17 in San Francisco • Keynotes

    by Target, GE Capital, Jez Humble, Rachel Chalmers • 50+ Sessions over 2 days • Community Concert Friday, February 21, 14
  242. Training Workshops • Introduction to Chef - Managing Linux •

    Introduction to Chef - Managing Windows • Chef - Intermediate Topics • Team Workflows with Chef • Testing Your Automation Code • Introduction to Kanban for Operations & DevOps • Awesome Postmortems • Git Foundations & Intermediate and Advanced Git and Github Tips and Tricks • Ruby for Chefs Friday, February 21, 14
  243. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife- plugins, and more • Best Practices for working with Chef 282 Friday, February 21, 14
  244. v2.0.0 Thank You! What questions do you have? 283 Nathen

    Harvey @nathenharvey Kennon Kwok @kennonkwok [email protected] github.com/nathenharvey [email protected] github.com/kennonkwok Friday, February 21, 14