Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chef & Immutable Infrasturcture

Richard Lee
September 26, 2014

Chef & Immutable Infrasturcture

For Rails Pacific workshop.

Richard Lee

September 26, 2014
Tweet

More Decks by Richard Lee

Other Decks in Programming

Transcript

  1. Chef & Immutable
    Infrastructure
    by Richard Lee
    Photo by @hownowdesign on flickr

    View full-size slide

  2. Welcome to this workshop!

    View full-size slide

  3. "Richard Lee".inspect
    4 CTO & Cofounder at iCook.tw
    (Polydice, inc.)
    4 Rails, DevOps & iOS
    4 GitHub, Twitter & everywhere:
    @dlackty
    4 Email: [email protected]
    Feel free to contact me for
    anything!

    View full-size slide

  4. Before we started

    View full-size slide

  5. Prerequisite
    1. Have you install Vagrant yet?
    2. Have you install ChefDK yet?
    4 Not only Chef
    4 But many other tools
    4 Can be installed via gem install as well though
    Go download it or ask staff for USB flash drive.

    View full-size slide

  6. Vagrant installation
    Go to Vagrant website and
    download packages.
    https://www.vagrantup.com/
    and you also need VirtualBox from
    its website
    https://www.virtualbox.org/

    View full-size slide

  7. ChefDK installation
    Go get SDK on ChefDK website
    https://downloads.getchef.com/
    chef-dk/
    To verify the installation, type
    following command
    chef verify

    View full-size slide

  8. Agenda
    We'll have five exercises, each for 30 minutes.
    1. Set up VirtualBox & Chef
    2. Create your first cookbook
    3. Write test for your cookbook
    4. Add cookbook dependencies using Berksfile
    5. Prepare environment for your Rails app

    View full-size slide

  9. Set up VirtualBox & Chef
    Photo by @hownowdesign on flickr

    View full-size slide

  10. Test Kitchen
    Test Kitchen is an integration tool
    for developing and testing
    infrastructure code and
    software on isolated target
    platforms.
    http://kitchen.ci/

    View full-size slide

  11. Let's get started
    Type this on shell:
    git init workshop
    cd workshop
    kitchen init
    and it will generate .kitchen, .kitchen.yml, and
    also .gitignore

    View full-size slide

  12. Check .kitchen.yml
    4 Driver is the way to set up base instance
    4 Vagrant for local testing
    4 EC2 / Digital Ocean or others for production
    4 Provisioner is the tool for environment setup process
    4 Chef / Puppet / Ansible

    View full-size slide

  13. Check .kitchen.yml (cont’d)
    4 Platforms
    4 It’s obvious, right?
    4 Use ubuntu-14.04 for this workshop
    4 Suites
    4 Different set of recipes you want to run
    4 e.g. App, Database, Cache, LoadBalancer

    View full-size slide

  14. So… you’ll need to download VM image
    The SPoF (Single Point of Failure) of this workshop.
    We’ll use it later (and tell you how to download it)
    We've downloaded it and put it into USB sticks. Feel free
    to ask us to install faster.
    vagrant box add PATH --name opscode-ubuntu-14.04

    View full-size slide

  15. Then let’s run it up!
    Just use the command:
    kitchen create ubuntu
    Argument can be a regex for available instances. You can
    check all available instance via kitchen list

    View full-size slide

  16. Time to talk about tools
    Photo by @hownowdesign on flickr

    View full-size slide

  17. Vagrant
    Commonly misunderstood as a
    command line tool for VirtualBox.
    1. It’s a general tool
    2. Many plugins available
    3. Support provision tools

    View full-size slide

  18. Test Kitchen supported platforms
    kitchen driver discover
    Just name a few:
    1. kitchen-{azure, cloudstack, digitalocean, ec2,
    gce, backspace, openstack, docker}
    2. kitchen-ssh for almost every server!

    View full-size slide

  19. Login to instance
    kitchen login ubuntu
    Here you go! and you can open VirtualBox.app to see
    running instances.
    You can use kitchen destroy to remove it.

    View full-size slide

  20. Exercise 1 Complete!
    Congrats! We have a foundation now.

    View full-size slide

  21. Create your first cookbook
    Photo by @hownowdesign on flickr

    View full-size slide

  22. Cookbook basics
    4 cookbook is a fundamental unit for a scenario
    4 e.g. nginx / elasticsearch / mysql
    4 cookbook has_many recipes
    4 e.g. nginx::default, nginx::ssl, nginx::status

    View full-size slide

  23. Common use case
    Usually you will have a cookbook representing “your
    app”.
    e.g. We have our app “icook” cookbook opne sourced on
    GitHub “polydice/cookbooks”

    View full-size slide

  24. Define cookbook
    Like .gemspec is for Ruby gems, metadata.rb is for Chef
    cookbook.
    Create an metadata.rb with following content:
    name "workshop"
    version "0.1.0"

    View full-size slide

  25. Add recipe
    To add workshop::default cookbook, create the
    following file/directory structure.
    mkdir recipes
    touch recipes/default.rb

    View full-size slide

  26. Chef recipe in Ruby
    Chef is written in Ruby, and you can use arbitrary Ruby
    syntax in recipe.
    if node["platform"] == "ubuntu"
    # Do ubuntu thing
    end

    View full-size slide

  27. Chef DSLs
    However, there’re some “Recipe DSL” methods like
    platform? or platform_family?
    if platform_family? "debian"
    # Do debian thing
    end
    Check http://docs.getchef.com/chef/dsl_recipe.html
    for more information

    View full-size slide

  28. Resources
    You can think of Chef resources as wrapper of…
    system resources. To name a few built-in ones:
    4 directory, file, user, group - create things
    4 package - for system package
    4 bash - to run random shell script (well, be careful)
    4 cron - to update crontab

    View full-size slide

  29. Learn Chef resources
    This might be the first obstacle for Chef beginners.
    As usual, please check Chef official doc for more info:
    http://docs.getchef.com/chef/resources.htm

    View full-size slide

  30. Put something into our recipe
    For some packages for our Rails app, open recipes/
    default.rb and put:
    package "git"
    package "graphicsmagick"
    log “OK! We now complete exercise 2”

    View full-size slide

  31. Run list
    In Chef, we define run list, which contains a series of
    recipes that will be executed in order. So
    update .kitchen.yml:
    suites:
    - name: default
    run_list: workshop::default
    attributes:

    View full-size slide

  32. Now let’s run again
    Use the following command:
    kitchen converge
    This will help you:
    1. Install chef on the instance
    2. Copy cookbooks to the instance
    3. Execute recipes in run list

    View full-size slide

  33. Verify by hand
    Login to the instance:
    kitchen login
    and check:
    vagrant@default-ubuntu-1404:~$ git --version
    git version 1.9.1

    View full-size slide

  34. Exercise 2 Complete!
    Congrats for your first recipe!

    View full-size slide

  35. Write test for your
    cookbooks
    Photo by @hownowdesign on flickr

    View full-size slide

  36. Immutable infrastructure
    Immutable means not changeable, and there’re benefits:
    1. Reduce inconsistency
    2. Improve the trust into your deployment process
    3. The whole process is repeatable, hence
    4 It’s easier to recover, scale
    4 It’s testable

    View full-size slide

  37. Introduce Serverspec
    Server spec is a set of RSpec
    matchers for infrastructure
    testing.
    Again, check the document online
    for example usages.
    http://serverspec.org/
    resource_types.html

    View full-size slide

  38. Serverspec examples
    There’re few examples for Severspec:
    describe command("whoami") do
    it { should return_stdout "root" }
    end
    describe file("/etc/sudoers") do
    it { should be_readable.by("owner") }
    it { should be_readable.by("group") }
    end

    View full-size slide

  39. Let’s write our specs
    Create folder and file:
    mkdir -p test/integration/default/serverspec
    touch test/integration/default/severspec/packages_spec.rb
    Be careful about spelling. Test Kitchen uses your
    directory name to select testing framework.

    View full-size slide

  40. And put something into files
    Put something like below:
    require 'serverspec'
    include Serverspec::Helper::Exec
    include Serverspec::Helper::DetectOS
    describe package("git") do
    it { should be_installed }
    end
    describe package("graphicsmagick") do
    it { should be_installed }
    end

    View full-size slide

  41. Run it!
    Let’s run it!
    kitchen verify
    and as usually, Test Kitchen will help you:
    1. Set up test framework
    2. Copy test files
    3. Run the tests

    View full-size slide

  42. To do a complete test
    To double confirm your recipe and test are in good
    status, use the following command:
    kitchen test
    and it does everything we discussed before: {destroy,
    create, converge, verify} to make sure it works.

    View full-size slide

  43. Exercise 3 Complete!
    Congrats! It’s now well tested.

    View full-size slide

  44. Add cookbook dependencies
    using Berksfile
    Photo by @hownowdesign on flickr

    View full-size slide

  45. Berksfile is Gemfile for Chef
    As we need Bundler for Ruby Gems, Berkshelf is made
    for cookbook dependency management.
    Developed by Riot Games, company behind League of
    Legends.

    View full-size slide

  46. Creating Berksfile
    Put the following codes into Berksfile:
    source "https://supermarket.getchef.com"
    metadata
    It’s similar to Gemfile, right?
    Then install

    View full-size slide

  47. 2 ways to add dependencies
    Again, like Ruby Gems, you can add dependencies in
    Berksfile or metadata.rb.
    My personal suggestion:
    4 Put related and real dependencies in metadata.rb
    4 Put something else in Berksfile

    View full-size slide

  48. Recipe configuration
    How could we do recipe configuration? Usually there’re
    2 ways:
    1. node attributes
    2. data bags
    tl;dr: Most of recipes use node attributes now. Data
    bags are deprecated.

    View full-size slide

  49. Reading README
    When you use a new cookbook, first read its README
    file for:
    1. What recipes available
    2. What node attributes are able to be used to
    configure
    Take nginx as example:
    https://supermarket.getchef.com/cookbooks/nginx

    View full-size slide

  50. Be cautious of different version of !
    There’re usually several versions of “nginx” cookbook
    available, and you need to be careful the version you
    use might provides different set of recipes.
    Use Berksfile to specify:
    cookbook 'mysql', path: '../mysql-cookbook'
    cookbook 'mysql', git: 'git://github.com/opscode-cookbooks/mysql.git'
    cookbook 'nginx', github: 'dlackty/mysql', branch: 'something'

    View full-size slide

  51. Let’s do this
    Edit metadata.rb:
    name "workshop"
    version "0.1.0"
    depends "nginx", "~> 2.7.0"
    then you can put nginx::default into run list. Still
    remember how?

    View full-size slide

  52. Another way to include recipe
    To combine few recipes, put this into our recipe:
    include_recipe("nginx::default")
    then run kitchen converge again.

    View full-size slide

  53. To do configuration
    Update .kitchen.yml:
    suites:
    - name: default
    run_list: workshop::default
    attributes:
    nginx:
    gzip: off

    View full-size slide

  54. Exercise 4 Complete!
    Congrats! Now you’re all set.

    View full-size slide

  55. Prepare environment for
    your Rails app
    Photo by @hownowdesign on flickr

    View full-size slide

  56. All right, you have a good start now
    Now it’s your time.
    1. Go pick up few cookbook
    2. Run it on the machine
    3. Try different drivers

    View full-size slide

  57. Go pick up few cookbook
    There’re few steps that you can consider:
    1. Search supermarket.chef.com
    2. Search GitHub.com
    e.g. MySQL, Redis, Elsticsearch and others

    View full-size slide

  58. A good start point
    You can find some cookbook is designed for Rails, and
    includes several required dependencies
    teohm/rackbox-cookbook is a good start point.

    View full-size slide

  59. Try different drivers
    Test Kitchen provides a good plugin system:
    1. Drivers - to spin up new instances
    2. Bussers - to run test

    View full-size slide

  60. Take EC2 as example
    From http://rubydoc.info/gems/kitchen-ec2
    driver:
    name: ec2
    aws_access_key_id: KAS...
    aws_secret_access_key: 3UK...
    aws_ssh_key_id: id_rsa-aws
    ssh_key: /path/to/id_rsa-aws
    security_group_ids: ["sg-1a2b3c4d"]
    region: us-east-1
    availability_zone: us-east-1b
    subnet_id: subnet-6d6...

    View full-size slide

  61. Random topics
    Some related things for Chef in
    production

    View full-size slide

  62. Deployment
    Usually for Chef beginners, you will get confused with
    Chef & deployment tools like Capistrano. Don’t panic.
    1. Try to avoid using Chef as deployment tool at first
    2. Consider using Chef as a way to set up Capistrano
    friendly environment

    View full-size slide

  63. How about docker?
    Just another way to create base instances. You still
    need to “set it up”.
    It’s can be used with Vagrant & Test Kitchen.

    View full-size slide

  64. chef-solo and chef-server
    What we do today is named chef-solo, which indicates
    that it doesn’t require a central server.
    chef-server isn’t recommended for most people
    because:
    1. The server is complex
    2. One more SPoF
    3. Many functionalities are duplicate of other tools

    View full-size slide

  65. Chef in production
    To adopt Chef successfully in production, usually
    there’re 2 ways.
    1. Use Chef-enabled environment like AWS OpsWorks or
    Engine Yard
    2. Use Chef to provision images for later usage

    View full-size slide

  66. AWS OpsWorks
    It’s a free platform provided by
    AWS built upon EC2 & related
    products.
    1. It provides a builtin set of
    cookbook for Rails / PHP /
    Node.js
    2. It has API to run Chef
    programmatically
    3. Deploy vis GitHub

    View full-size slide

  67. Thank you!
    Your feedback is highly appreciated

    View full-size slide