$30 off During Our Annual Pro Sale. View Details »

Chef Workshop - SCaLE 11x - Sunday

Nathen Harvey
February 24, 2013

Chef Workshop - SCaLE 11x - Sunday

Slides for Chef Workshop on Sunday of SCaLE 11x

Nathen Harvey

February 24, 2013
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. Chef Introductory Workshop training@opscode.com http://opscode.com/training http://bit.ly/VFYRfR

  2. Introductions

  3. Nathen Harvey • Technical Community Manager • Co-host Food Fight

    Show • http://foodfightshow.org • @nathenharvey
  4. Introduce yourselves

  5. Objectives and Expectations

  6. System Administration with Chef: Agenda • Overview of Chef •

    Your First Recipe • Configure Your Workstation • Bootstrap a Node • Use a Community Cookbook • Further Resources
  7. Workshop Objectives • Describe Chef’s architecture. • Use Community Cookbooks

    • Be familiar with Chef's various tools • Know how to get further help
  8. Overview of Chef What is this thing again?

  9. Evolving towards Configuration Management • Just build it • Keep

    notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Snapshot & Clone
  10. Chef is an automation platform for developers & systems engineers

    to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
  11. http://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/ Applications

  12. Infrastructure http://www.flickr.com/photos/sbh/462754460/

  13. Collection of Resources http://www.flickr.com/photos/philliecasablanca/3354734116/ • Networking • Files • Directories

    • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configuration • Other Stuff
  14. Code Sample Acting in Concert http://www.flickr.com/photos/glowjangles/4081048126/

  15. Code Sample To Provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/

  16. And it Evolves http://www.flickr.com/photos/16339684@N00/2681435235/

  17. See Node Application Server

  18. Application Server Application Database See Nodes

  19. Application Server Application Databases See Nodes Grow

  20. Application Servers Application Databases See Nodes Grow

  21. Application Servers Application Databases Load Balancer See Nodes Grow

  22. Application Servers Application Databases Load Balancers See Nodes Grow

  23. Application Servers Application Database Cache Load Balancers Application Databases See

    Nodes Grow
  24. Application Servers Application Database Cache Load Balancers Application Databases Tied

    Together with Configuration
  25. Application Servers Application Database Cache Load Balancers Floating IP? Application

    Databases Infrastructure is a Snowflake
  26. Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache

    Database Evolving Complexity
  27. DC1 DC3 DC2 Complexity Grows Quickly

  28. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/

  29. Golden Images are not the answer • Gold is heavy

    • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
  30. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Typical

    Infrastructure
  31. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Move SSH off port 22 • Lets put it on 2022 New Compliance Mandate!
  32. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    edit /etc/ssh/sshd_config 1 2 3 4 5 6 6 Golden Image Updates
  33. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually 12 Instance Replacements
  34. • Don’t break anything! • Bob just got fired =(

    5 Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3 Done in Maintenance Windows
  35. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite •

    Invalid configs! Different IP Addresses?
  36. Configuration Desperation Code Sample http://www.flickr.com/photos/francoforeshock/5716969942/

  37. • But you already guessed that, didn’t you? Chef Solves

    this Problem
  38. http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any

    other code base • Reconstruct business from code repository, data backup, and bare metal resources. Chef is Infrastructure as Code
  39. http://www.flickr.com/photos/ssoosay/5126146763/ • Chef generates configurations directly on nodes from their

    run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control Programs
  40. • Define Policy • Say what, not how • Pull

    not Push Code Sample http://www.flickr.com/photos/bixentro/2591838509/ Declarative Interface to Resources
  41. package "ntp" do action :install end service "ntpd" do action

    [:enable,:start] end template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “time.example.com”) notifies :restart, “service[ntpd]” end That looks like this
  42. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite So

    when this
  43. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes

    this
  44. Jboss App Memcache Postgres Slaves Postgres Master Nagios Graphite This

    can happen automatically
  45. Nagios Graphite Jboss App Memcache Postgres Slaves • Load balancer

    config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition Count the resources
  46. Getting Started

  47. Getting Started • Your first recipe • Workstation Setup •

    Chef Server Account • Chef Repository • Remote target managed node
  48. Pre-requisites • Install Chef Client on your workstation • opscode.com/chef/install

    • Register for FREE Hosted Chef trial • http://www.opscode.com/hosted-chef/
  49. Your First Recipe • Resource are the building blocks of

    Chef • Recipes are a collection of resources
  50. Chef Resources • Have a type. • Have a name.

    • Have parameters. • Take action to put the resource in the declared state. • Can send notifications to other resources. package "haproxy" do action :install end template "/etc/haproxy/haproxy.cfg" do source "haproxy.cfg.erb" owner "root" group "root" mode 0644 notifies :restart, "service[haproxy]" end service "haproxy" do supports :restart => true action [:enable, :start] end
  51. The Problem and the Success Criteria • The Problem: We

    need a web server configured to serve up our home page. • Success Criteria: We can see the homepage in a web browser.
  52. Apache Recipe • Write a simple recipe file • SSH

    and vim • Write locally and scp • Apply with chef-apply
  53. apache.rb package "apache2" do action :install end

  54. apache.rb file "/var/www/index.html" do mode "0644" content "<h1>Hello, SCALE!</h1>" end

  55. apache.rb service "apache2" do action [:start,:enable] end

  56. Apply the Apache Recipe $ sudo chef-apply apache.rb [sudo] password

    for opscode: Recipe: (chef-apply cookbook)::(chef-apply recipe) * package[apache2] action install - install version 2.2.22-1ubuntu1 of package apache2 * file[/var/www/index.html] action create - update content in file /var/www/index.html from 94850c to 599510 --- /var/www/index.html 2013-02-24 14:56:23.445076249 +0000 +++ /tmp/chef-diff20130224-3262-1h031ui 2013-02-24 14:56:26.065076250 +0000 @@ -1,4 +1 @@ -<html><body><h1>It works!</h1> -<p>This is the default web page for this server.</p> -<p>The web server software is running but no content has been added, yet.</p> -</body></html> +<h1>Hello, SCALE!</h1> * service[apache2] action start (up to date) * service[apache2] action enable (up to date)
  57. Hello!

  58. Congratulate yourself! • You have just written your first Chef

    recipe! • (clap!)
  59. Code Sample Landscape of Chef-managed Infrastructure

  60. Workstation Setup • Install Chef (if not already installed) •

    https://www.opscode.com/ chef/install/
  61. Your Chef Server for this class... • Set up Chef

    Server Account • Opscode Hosted Chef • https://manage.opscode.com
  62. Sign-up for Hosted Chef

  63. Create an Organization

  64. Create New Organization Organization Short Name must be GLOBALLY unique!

  65. Download the Validation Key and Knife Config

  66. Get a New User Key • Only if you don’t

    have your user key with you today!
  67. Setup Your Chef Repository > cd [THE DIR FOR THIS

    WORKSHOP] > mkdir .chef
  68. Copy Chef Server Files # copy your user key, validation

    key and knife config: > cp ~/Downloads/ORGNAME-validator.pem .chef > cp ~/Downloads/USERNAME.pem .chef > cp ~/Downloads/knife.rb .chef > ls .chef ORGNAME-validator.pem USERNAME.pem knife.rb
  69. Verify Knife > knife --version Chef: 11.4.0 > knife client

    list ORGNAME-validator Your version may differ, that's okay!
  70. Bootstrap the Target Instance

  71. Target Instances ec2-based Instance • ec2-STUDENT_ID.compute-1.amazonaws.com • Ubuntu 12.04 •

    SSH • Username: opscode • Password: opscode
  72. "Bootstrap" the Target Instance > knife bootstrap IPADDRESS --sudo -x

    opscode -P opscode Bootstrapping Chef on IPADDRESS IPADDRESS knife sudo password: Enter your password:
  73. Opscode Hosted Chef local workstation managed node (VM) chef-client knife

    bootstrap IPADDRESS --sudo -x USERNAME -P PASSWORD chef_server_url validation_client_name validation_key SSH! bash -c ' install chef configure client run chef'
  74. Chef 101 Terminology

  75. chef-client runs on your systems

  76. chef-client talks to a Chef Server

  77. API Clients authenticate with RSA keys The server has the

    public key
  78. Configured, or managed systems are called Nodes

  79. Knife is the command-line user's tool for Chef.

  80. Anatomy of a Chef Run

  81. build node authenticate sync cookbooks load cookbooks converge node.save notification

    handlers exception Yes No chef-client success? expanded run list (recipes) Ohai! node_name platform platform_version
  82. /etc/chef/ client.pem? /etc/chef/ validation.pem? 401! Request API Client Sign Requests

    client.pem Yes No No Yes
  83. Current Status - Managed Node

  84. Current Status: > knife node list target1 > knife client

    list target1 ORGNAME-validator > knife node show target1 Node Name: ip-XXX.ec2.internal Environment: _default FQDN: ip-XXX.ec2.internal IP: IPADDRESS Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags:
  85. Knife's commands have built-in help > knife node show --help

    > knife help node
  86. What did Knife Bootstrap Create? > ssh opscode@target opscode@target1:~$ ls

    /etc/chef client.pem client.rb first-boot.json validation.pem
  87. /etc/chef/client.rb $ cat /etc/chef/client.rb log_level :auto log_location STDOUT chef_server_url "https://chef.local/organizations/ORGNAME"

    validation_client_name "ORGNAME-validator" # Using default node name (fqdn)
  88. /etc/chef/first-boot.json $ cat /etc/chef/first-boot.json {"run_list":[]"]} $ chef-client -h | grep

    -i json -j JSON_ATTRIBS, Load attributes from a JSON file or URL --json-attributes
  89. Private Keys • Remember from the authentication cycle: Chef Server

    requires keys to authenticate. • client.pem - private key for API client • validation.pem - private key for ORGNAME-validator
  90. Chef 101 Terminology

  91. chef-client runs on your systems

  92. chef-client talks to a Chef Server

  93. API Clients authenticate with RSA keys The server has the

    public key
  94. Configured, or managed systems are called Nodes

  95. Nodes have a Run List The list of roles or

    recipes to apply in order
  96. Recipes are lists of resources Resources are applied in the

    order they're written in recipes
  97. Cookbooks are packages for Recipes

  98. Knife is the command-line user's tool for Chef.

  99. Exercise: base role

  100. Objectives • Understand what a role is • Know how

    to create a new role • Know how to upload a role to the Chef Server
  101. Exercise: Re-run the Chef Client $ mkdir roles $ vim

    roles/base.rb name "base" description "Base role applied to all nodes" run_list( ) default_attributes( )
  102. Exercise: Re-run the Chef Client $ knife role from file

    roles/base.rb Updated Role base!
  103. The pattern for each exercise is a common Chef workflow

    • Download cookbooks from Chef Community Site with Knife. • Extract the cookbook's .tar.gz into cookbooks directory. • Review the code you're going to run as root. • Upload the cookbook to the Chef Server. • Apply the cookbook to your node(s) with a role. • Edit role's run list (base, monitoring) • Modify attributes as required
  104. Download, extract, upload > knife cookbook site download COOKBOOK >

    tar -zxvf COOKBOOK*.tar.gz -C cookbooks > less cookbooks/COOKBOOK/README.md > less cookbooks/COOKBOOK/recipes/default.rb > knife cookbook upload COOKBOOK
  105. Exercise: sudo cookbook

  106. Exercise: sudo cookbook Policy statement: User privileges will be managed

    through sudoers entries. New concepts: • Attribute priority • Setting attributes in Cookbooks and Roles • Using attributes in a template • Ruby array iteration • Package resource • File backups
  107. #protip: Log in and su to root!

  108. sudo cookbook • Download the sudo cookbook. • Extract it

    to the cookbooks directory. • Upload it to the Chef Server. • Add "recipe[sudo]" to the run list. • Modify sudo-specific attributes in the base role. • Run Chef on the target managed node.
  109. Exercise: chef-client cookbook > knife cookbook site download sudo >

    tar -zxvf sudo*.tar.gz -C cookbooks > knife cookbook upload sudo
  110. Opscode Hosted Chef Chef Repository .chef/knife.rb cookbooks/ data_bags/ roles/ local

    workstation Chef Community Site knife cookbook site download knife cookbook upload cookbook tar -zxvf cb.tar.gz -C cookbooks cookbooks/COOKBOOK ├── metadata.rb ├── recipes │ └── default.rb └── templates └── default └── my-tmpl.erb
  111. Update the base role name "base" description "Base role applied

    to all nodes." run_list( "recipe[sudo]" ) default_attributes( "authorization" => { "sudo" => { "users" => ["opscode"], "groups" => ["admin","sudo"], "passwordless" => true } } )
  112. Exercise: Re-run the Chef Client $ knife role from file

    roles/base.rb Updated Role base!
  113. cookbooks/sudo/attributes/default.rb default['authorization']['sudo']['groups'] = [] default['authorization']['sudo']['users'] = [] default['authorization']['sudo']['passwordless'] = false

    default['authorization']['sudo']['include_sudoers_d'] = false default['authorization']['sudo']['agent_forwarding'] = false
  114. cookbooks/sudoers/recipes/default.rb package 'sudo' do action :install end if node['authorization']['sudo']['include_sudoers_d'] directory

    '/etc/sudoers.d' { ... } cookbook_file '/etc/sudoers.d/README' { ... } end template '/etc/sudoers' do source 'sudoers.erb' mode '0440' owner 'root' group 'root' variables(:sudoers_groups => node['authorization']['sudo']['groups'], :sudoers_users => node['authorization']['sudo']['users'], :passwordless => node['authorization']['sudo']['passwordless'], :include_sudoers_d => node['authorization']['sudo']['include_sudoers_d'] :agent_forwarding => node['authorization']['sudo']['agent_forwarding']) end
  115. cookbooks/sudo/templates/default/sudoers.erb root ALL=(ALL) ALL <% @sudoers_users.each do |user| -%> <%=

    user %> ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL <% end -%> # Members of the sysadmin group may gain root privileges %sysadmin ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL <% @sudoers_groups.each do |group| -%> # Members of the group '<%= group %>' may gain root privileges %<%= group %> ALL=(ALL) <%= "NOPASSWD:" if @passwordless %>ALL <% end -%> <%= '#includedir /etc/sudoers.d' if @include_sudoers_d %>
  116. File Content Updates • "file", "template", "cookbook_file" and "remote_file" •

    Default backup location is /var/chef/ backup, configurable with "file_backup_path" in /etc/chef/client.rb • 5 backups are kept by default, change this with the "backup" parameter in the resource.
  117. Running chef-client

  118. Running the Chef Client • Automatically • cron • daemon

  119. knife ssh $ knife ssh "role:base"

  120. knife ssh $ knife ssh role:base "sudo chef-client" -x opscode

    -P opscode
  121. Further Resources

  122. Further Resources • http://opscode.com/ • http://community.opscode.com/ • http://docs.opscode.com • http://wiki.opscode.com/

    • http://lists.opscode.com • http://youtube.com/user/Opscode
  123. Food Fight Show • http://foodfightshow.org • The Podcast Where DevOps

    Chef Do Battle • Regular updates about new Cookbooks, Knife-plugins, and more • Best Practices for working with Chef
  124. Get Involved Locally

  125. More Local User Groups • http://wiki.opscode.com/display/chef/ Community+Events

  126. More Training in LA • SOCAL-CHEF saves you $200 •

    http://opscode.eventbrite.com/
  127. #ChefConf 2013 Tex OPSCODE-SCALE - Save 10%

  128. Exercise: Database Connection

  129. Exercise: Database Connection Policy statement: The database connection should be

    dynamically generated based on a search and encrypted credentials New concepts: • Creating Cookbooks • Search • Environments • Encrypted Data bags
  130. Exercise: Database Connection • Create an Environment • Update your

    node’s Environment • Create an encrypted data bag item with database credentials • Create a cookbook • Write a file that uses • Search for the host • Encrypted Data Bag for the Credentials
  131. database.yml staging: host: foo.example.com username: yourusername password: yourpassword

  132. Additional Topics Version Control Ruby Testing Recipes Get Involved! Chef

    Development
  133. Version Control • USE SOMETHING. • Distributed Version Control •

    Git, GitHub, BitBucket • http://git-scm.com • https://github.com • https://bitbucket.org • Workflows, CI
  134. Ruby is worth learning • Recipe DSL • Libraries, "LWRPs"

    and more • Knife plugins • Report/exception handlers • chef-shell
  135. Testing Recipes • Chef 10.14+, "why run" mode • Test

    Kitchen (RubyGem) • Vagrant • http://vagrantup.com • Minitest - cookbook, handler • Cucumber - cucumber-chef • http://www.cucumber-chef.org/
  136. Get Involved • Community Site: • community.opscode.com • IRC: #chef,

    #chef-hacking • irc.freenode.net • Mailing list: • lists.opscode.com • ChefConf, Community Summits, User Groups, Hack days and more
  137. Chef Development • Apache 2 Software License • Continually growing

    number of contributors! • Development repositories: • http://github.com/opscode • http://github.com/opscode-cookbooks