Save 37% off PRO during our Black Friday Sale! »

Chef & Immutable Infrasturcture

Aa0358f6740f1bf02911e5300e08e800?s=47 Richard Lee
September 26, 2014

Chef & Immutable Infrasturcture

For Rails Pacific workshop.


Richard Lee

September 26, 2014


  1. Chef & Immutable Infrastructure by Richard Lee Photo by @hownowdesign

    on flickr
  2. Welcome to this workshop!

  3. "Richard Lee".inspect 4 CTO & Cofounder at (Polydice, inc.)

    4 Rails, DevOps & iOS 4 GitHub, Twitter & everywhere: @dlackty 4 Email: Feel free to contact me for anything!
  4. Before we started

  5. Prerequisite 1. Have you install Vagrant yet? 2. Have you

    install ChefDK yet? 4 Not only Chef 4 But many other tools 4 Can be installed via gem install as well though Go download it or ask staff for USB flash drive.
  6. Vagrant installation Go to Vagrant website and download packages.

    and you also need VirtualBox from its website
  7. ChefDK installation Go get SDK on ChefDK website chef-dk/

    To verify the installation, type following command chef verify
  8. Agenda We'll have five exercises, each for 30 minutes. 1.

    Set up VirtualBox & Chef 2. Create your first cookbook 3. Write test for your cookbook 4. Add cookbook dependencies using Berksfile 5. Prepare environment for your Rails app
  9. Set up VirtualBox & Chef Photo by @hownowdesign on flickr

  10. Test Kitchen Test Kitchen is an integration tool for developing

    and testing infrastructure code and software on isolated target platforms.
  11. Let's get started Type this on shell: git init workshop

    cd workshop kitchen init and it will generate .kitchen, .kitchen.yml, and also .gitignore
  12. Check .kitchen.yml 4 Driver is the way to set up

    base instance 4 Vagrant for local testing 4 EC2 / Digital Ocean or others for production 4 Provisioner is the tool for environment setup process 4 Chef / Puppet / Ansible
  13. Check .kitchen.yml (cont’d) 4 Platforms 4 It’s obvious, right? 4

    Use ubuntu-14.04 for this workshop 4 Suites 4 Different set of recipes you want to run 4 e.g. App, Database, Cache, LoadBalancer
  14. So… you’ll need to download VM image The SPoF (Single

    Point of Failure) of this workshop. We’ll use it later (and tell you how to download it) We've downloaded it and put it into USB sticks. Feel free to ask us to install faster. vagrant box add PATH --name opscode-ubuntu-14.04
  15. Then let’s run it up! Just use the command: kitchen

    create ubuntu Argument can be a regex for available instances. You can check all available instance via kitchen list
  16. Time to talk about tools Photo by @hownowdesign on flickr

  17. Vagrant Commonly misunderstood as a command line tool for VirtualBox.

    1. It’s a general tool 2. Many plugins available 3. Support provision tools
  18. Test Kitchen supported platforms kitchen driver discover Just name a

    few: 1. kitchen-{azure, cloudstack, digitalocean, ec2, gce, backspace, openstack, docker} 2. kitchen-ssh for almost every server!
  19. Login to instance kitchen login ubuntu Here you go! and

    you can open to see running instances. You can use kitchen destroy to remove it.
  20. Exercise 1 Complete! Congrats! We have a foundation now.

  21. Create your first cookbook Photo by @hownowdesign on flickr

  22. Cookbook basics 4 cookbook is a fundamental unit for a

    scenario 4 e.g. nginx / elasticsearch / mysql 4 cookbook has_many recipes 4 e.g. nginx::default, nginx::ssl, nginx::status
  23. Common use case Usually you will have a cookbook representing

    “your app”. e.g. We have our app “icook” cookbook opne sourced on GitHub “polydice/cookbooks”
  24. Define cookbook Like .gemspec is for Ruby gems, metadata.rb is

    for Chef cookbook. Create an metadata.rb with following content: name "workshop" version "0.1.0"
  25. Add recipe To add workshop::default cookbook, create the following file/directory

    structure. mkdir recipes touch recipes/default.rb
  26. Chef recipe in Ruby Chef is written in Ruby, and

    you can use arbitrary Ruby syntax in recipe. if node["platform"] == "ubuntu" # Do ubuntu thing end
  27. Chef DSLs However, there’re some “Recipe DSL” methods like platform?

    or platform_family? if platform_family? "debian" # Do debian thing end Check for more information
  28. Resources You can think of Chef resources as wrapper of…

    system resources. To name a few built-in ones: 4 directory, file, user, group - create things 4 package - for system package 4 bash - to run random shell script (well, be careful) 4 cron - to update crontab
  29. Learn Chef resources This might be the first obstacle for

    Chef beginners. As usual, please check Chef official doc for more info:
  30. Put something into our recipe For some packages for our

    Rails app, open recipes/ default.rb and put: package "git" package "graphicsmagick" log “OK! We now complete exercise 2”
  31. Run list In Chef, we define run list, which contains

    a series of recipes that will be executed in order. So update .kitchen.yml: suites: - name: default run_list: workshop::default attributes:
  32. Now let’s run again Use the following command: kitchen converge

    This will help you: 1. Install chef on the instance 2. Copy cookbooks to the instance 3. Execute recipes in run list
  33. Verify by hand Login to the instance: kitchen login and

    check: vagrant@default-ubuntu-1404:~$ git --version git version 1.9.1
  34. Exercise 2 Complete! Congrats for your first recipe!

  35. Write test for your cookbooks Photo by @hownowdesign on flickr

  36. Immutable infrastructure Immutable means not changeable, and there’re benefits: 1.

    Reduce inconsistency 2. Improve the trust into your deployment process 3. The whole process is repeatable, hence 4 It’s easier to recover, scale 4 It’s testable
  37. Introduce Serverspec Server spec is a set of RSpec matchers

    for infrastructure testing. Again, check the document online for example usages. resource_types.html
  38. Serverspec examples There’re few examples for Severspec: describe command("whoami") do

    it { should return_stdout "root" } end describe file("/etc/sudoers") do it { should"owner") } it { should"group") } end
  39. Let’s write our specs Create folder and file: mkdir -p

    test/integration/default/serverspec touch test/integration/default/severspec/packages_spec.rb Be careful about spelling. Test Kitchen uses your directory name to select testing framework.
  40. And put something into files Put something like below: require

    'serverspec' include Serverspec::Helper::Exec include Serverspec::Helper::DetectOS describe package("git") do it { should be_installed } end describe package("graphicsmagick") do it { should be_installed } end
  41. Run it! Let’s run it! kitchen verify and as usually,

    Test Kitchen will help you: 1. Set up test framework 2. Copy test files 3. Run the tests
  42. To do a complete test To double confirm your recipe

    and test are in good status, use the following command: kitchen test and it does everything we discussed before: {destroy, create, converge, verify} to make sure it works.
  43. Exercise 3 Complete! Congrats! It’s now well tested.

  44. Add cookbook dependencies using Berksfile Photo by @hownowdesign on flickr

  45. Berksfile is Gemfile for Chef As we need Bundler for

    Ruby Gems, Berkshelf is made for cookbook dependency management. Developed by Riot Games, company behind League of Legends.
  46. Creating Berksfile Put the following codes into Berksfile: source ""

    metadata It’s similar to Gemfile, right? Then install
  47. 2 ways to add dependencies Again, like Ruby Gems, you

    can add dependencies in Berksfile or metadata.rb. My personal suggestion: 4 Put related and real dependencies in metadata.rb 4 Put something else in Berksfile
  48. Recipe configuration How could we do recipe configuration? Usually there’re

    2 ways: 1. node attributes 2. data bags tl;dr: Most of recipes use node attributes now. Data bags are deprecated.
  49. Reading README When you use a new cookbook, first read

    its README file for: 1. What recipes available 2. What node attributes are able to be used to configure Take nginx as example:
  50. Be cautious of different version of ! There’re usually several

    versions of “nginx” cookbook available, and you need to be careful the version you use might provides different set of recipes. Use Berksfile to specify: cookbook 'mysql', path: '../mysql-cookbook' cookbook 'mysql', git: 'git://' cookbook 'nginx', github: 'dlackty/mysql', branch: 'something'
  51. Let’s do this Edit metadata.rb: name "workshop" version "0.1.0" depends

    "nginx", "~> 2.7.0" then you can put nginx::default into run list. Still remember how?
  52. Another way to include recipe To combine few recipes, put

    this into our recipe: include_recipe("nginx::default") then run kitchen converge again.
  53. To do configuration Update .kitchen.yml: suites: - name: default run_list:

    workshop::default attributes: nginx: gzip: off
  54. Exercise 4 Complete! Congrats! Now you’re all set.

  55. Prepare environment for your Rails app Photo by @hownowdesign on

  56. All right, you have a good start now Now it’s

    your time. 1. Go pick up few cookbook 2. Run it on the machine 3. Try different drivers
  57. Go pick up few cookbook There’re few steps that you

    can consider: 1. Search 2. Search e.g. MySQL, Redis, Elsticsearch and others
  58. A good start point You can find some cookbook is

    designed for Rails, and includes several required dependencies teohm/rackbox-cookbook is a good start point.
  59. Try different drivers Test Kitchen provides a good plugin system:

    1. Drivers - to spin up new instances 2. Bussers - to run test
  60. Take EC2 as example From driver: name: ec2 aws_access_key_id:

    KAS... aws_secret_access_key: 3UK... aws_ssh_key_id: id_rsa-aws ssh_key: /path/to/id_rsa-aws security_group_ids: ["sg-1a2b3c4d"] region: us-east-1 availability_zone: us-east-1b subnet_id: subnet-6d6...
  61. Random topics Some related things for Chef in production

  62. Deployment Usually for Chef beginners, you will get confused with

    Chef & deployment tools like Capistrano. Don’t panic. 1. Try to avoid using Chef as deployment tool at first 2. Consider using Chef as a way to set up Capistrano friendly environment
  63. How about docker? Just another way to create base instances.

    You still need to “set it up”. It’s can be used with Vagrant & Test Kitchen.
  64. chef-solo and chef-server What we do today is named chef-solo,

    which indicates that it doesn’t require a central server. chef-server isn’t recommended for most people because: 1. The server is complex 2. One more SPoF 3. Many functionalities are duplicate of other tools
  65. Chef in production To adopt Chef successfully in production, usually

    there’re 2 ways. 1. Use Chef-enabled environment like AWS OpsWorks or Engine Yard 2. Use Chef to provision images for later usage
  66. AWS OpsWorks It’s a free platform provided by AWS built

    upon EC2 & related products. 1. It provides a builtin set of cookbook for Rails / PHP / Node.js 2. It has API to run Chef programmatically 3. Deploy vis GitHub
  67. Thank you! Your feedback is highly appreciated