Chef & Immutable Infrasturcture

For Rails Pacific workshop.

Transcript

1. Chef & Immutable Infrastructure by Richard Lee Photo by @hownowdesign on flickr
2. Welcome to this workshop!
3. "Richard Lee".inspect 4 CTO & Cofounder at iCook.tw (Polydice, inc.) 4 Rails, DevOps & iOS 4 GitHub, Twitter & everywhere: @dlackty 4 Email: dlackty@gmail.com Feel free to contact me for anything!
4. Before we started
5. Prerequisite 1. Have you install Vagrant yet? 2. Have you install ChefDK yet? 4 Not only Chef 4 But many other tools 4 Can be installed via gem install as well though Go download it or ask staff for USB flash drive.
6. Vagrant installation Go to Vagrant website and download packages. https://www.vagrantup.com/ and you also need VirtualBox from its website https://www.virtualbox.org/
7. ChefDK installation Go get SDK on ChefDK website https://downloads.getchef.com/ chef-dk/ To verify the installation, type following command chef verify
8. Agenda We'll have five exercises, each for 30 minutes. 1. Set up VirtualBox & Chef 2. Create your first cookbook 3. Write test for your cookbook 4. Add cookbook dependencies using Berksfile 5. Prepare environment for your Rails app
9. Set up VirtualBox & Chef Photo by @hownowdesign on flickr
10. Test Kitchen Test Kitchen is an integration tool for developing and testing infrastructure code and software on isolated target platforms. http://kitchen.ci/
11. Let's get started Type this on shell: git init workshop cd workshop kitchen init and it will generate .kitchen, .kitchen.yml, and also .gitignore
12. Check .kitchen.yml 4 Driver is the way to set up base instance 4 Vagrant for local testing 4 EC2 / Digital Ocean or others for production 4 Provisioner is the tool for environment setup process 4 Chef / Puppet / Ansible
13. Check .kitchen.yml (cont’d) 4 Platforms 4 It’s obvious, right? 4 Use ubuntu-14.04 for this workshop 4 Suites 4 Different set of recipes you want to run 4 e.g. App, Database, Cache, LoadBalancer
14. So… you’ll need to download VM image The SPoF (Single Point of Failure) of this workshop. We’ll use it later (and tell you how to download it) We've downloaded it and put it into USB sticks. Feel free to ask us to install faster. vagrant box add PATH --name opscode-ubuntu-14.04
15. Then let’s run it up! Just use the command: kitchen create ubuntu Argument can be a regex for available instances. You can check all available instance via kitchen list
16. Time to talk about tools Photo by @hownowdesign on flickr
17. Vagrant Commonly misunderstood as a command line tool for VirtualBox. 1. It’s a general tool 2. Many plugins available 3. Support provision tools
18. Test Kitchen supported platforms kitchen driver discover Just name a few: 1. kitchen-{azure, cloudstack, digitalocean, ec2, gce, backspace, openstack, docker} 2. kitchen-ssh for almost every server!
19. Login to instance kitchen login ubuntu Here you go! and you can open VirtualBox.app to see running instances. You can use kitchen destroy to remove it.
20. Exercise 1 Complete! Congrats! We have a foundation now.
21. Create your first cookbook Photo by @hownowdesign on flickr
22. Cookbook basics 4 cookbook is a fundamental unit for a scenario 4 e.g. nginx / elasticsearch / mysql 4 cookbook has_many recipes 4 e.g. nginx::default, nginx::ssl, nginx::status
23. Common use case Usually you will have a cookbook representing “your app”. e.g. We have our app “icook” cookbook opne sourced on GitHub “polydice/cookbooks”
24. Define cookbook Like .gemspec is for Ruby gems, metadata.rb is for Chef cookbook. Create an metadata.rb with following content: name "workshop" version "0.1.0"
25. Add recipe To add workshop::default cookbook, create the following file/directory structure. mkdir recipes touch recipes/default.rb
26. Chef recipe in Ruby Chef is written in Ruby, and you can use arbitrary Ruby syntax in recipe. if node["platform"] == "ubuntu" # Do ubuntu thing end
27. Chef DSLs However, there’re some “Recipe DSL” methods like platform? or platform_family? if platform_family? "debian" # Do debian thing end Check http://docs.getchef.com/chef/dsl_recipe.html for more information
28. Resources You can think of Chef resources as wrapper of… system resources. To name a few built-in ones: 4 directory, file, user, group - create things 4 package - for system package 4 bash - to run random shell script (well, be careful) 4 cron - to update crontab
29. Learn Chef resources This might be the first obstacle for Chef beginners. As usual, please check Chef official doc for more info: http://docs.getchef.com/chef/resources.htm
30. Put something into our recipe For some packages for our Rails app, open recipes/ default.rb and put: package "git" package "graphicsmagick" log “OK! We now complete exercise 2”
31. Run list In Chef, we define run list, which contains a series of recipes that will be executed in order. So update .kitchen.yml: suites: - name: default run_list: workshop::default attributes:
32. Now let’s run again Use the following command: kitchen converge This will help you: 1. Install chef on the instance 2. Copy cookbooks to the instance 3. Execute recipes in run list
33. Verify by hand Login to the instance: kitchen login and check: vagrant@default-ubuntu-1404:~$ git --version git version 1.9.1
34. Exercise 2 Complete! Congrats for your first recipe!
35. Write test for your cookbooks Photo by @hownowdesign on flickr
36. Immutable infrastructure Immutable means not changeable, and there’re benefits: 1. Reduce inconsistency 2. Improve the trust into your deployment process 3. The whole process is repeatable, hence 4 It’s easier to recover, scale 4 It’s testable
37. Introduce Serverspec Server spec is a set of RSpec matchers for infrastructure testing. Again, check the document online for example usages. http://serverspec.org/ resource_types.html
38. Serverspec examples There’re few examples for Severspec: describe command("whoami") do it { should return_stdout "root" } end describe file("/etc/sudoers") do it { should be_readable.by("owner") } it { should be_readable.by("group") } end
39. Let’s write our specs Create folder and file: mkdir -p test/integration/default/serverspec touch test/integration/default/severspec/packages_spec.rb Be careful about spelling. Test Kitchen uses your directory name to select testing framework.
40. And put something into files Put something like below: require 'serverspec' include Serverspec::Helper::Exec include Serverspec::Helper::DetectOS describe package("git") do it { should be_installed } end describe package("graphicsmagick") do it { should be_installed } end
41. Run it! Let’s run it! kitchen verify and as usually, Test Kitchen will help you: 1. Set up test framework 2. Copy test files 3. Run the tests
42. To do a complete test To double confirm your recipe and test are in good status, use the following command: kitchen test and it does everything we discussed before: {destroy, create, converge, verify} to make sure it works.
43. Exercise 3 Complete! Congrats! It’s now well tested.
44. Add cookbook dependencies using Berksfile Photo by @hownowdesign on flickr
45. Berksfile is Gemfile for Chef As we need Bundler for Ruby Gems, Berkshelf is made for cookbook dependency management. Developed by Riot Games, company behind League of Legends.
46. Creating Berksfile Put the following codes into Berksfile: source "https://supermarket.getchef.com" metadata It’s similar to Gemfile, right? Then install
47. 2 ways to add dependencies Again, like Ruby Gems, you can add dependencies in Berksfile or metadata.rb. My personal suggestion: 4 Put related and real dependencies in metadata.rb 4 Put something else in Berksfile
48. Recipe configuration How could we do recipe configuration? Usually there’re 2 ways: 1. node attributes 2. data bags tl;dr: Most of recipes use node attributes now. Data bags are deprecated.
49. Reading README When you use a new cookbook, first read its README file for: 1. What recipes available 2. What node attributes are able to be used to configure Take nginx as example: https://supermarket.getchef.com/cookbooks/nginx
50. Be cautious of different version of ! There’re usually several versions of “nginx” cookbook available, and you need to be careful the version you use might provides different set of recipes. Use Berksfile to specify: cookbook 'mysql', path: '../mysql-cookbook' cookbook 'mysql', git: 'git://github.com/opscode-cookbooks/mysql.git' cookbook 'nginx', github: 'dlackty/mysql', branch: 'something'
51. Let’s do this Edit metadata.rb: name "workshop" version "0.1.0" depends "nginx", "~> 2.7.0" then you can put nginx::default into run list. Still remember how?
52. Another way to include recipe To combine few recipes, put this into our recipe: include_recipe("nginx::default") then run kitchen converge again.
53. To do configuration Update .kitchen.yml: suites: - name: default run_list: workshop::default attributes: nginx: gzip: off
54. Exercise 4 Complete! Congrats! Now you’re all set.
55. Prepare environment for your Rails app Photo by @hownowdesign on flickr
56. All right, you have a good start now Now it’s your time. 1. Go pick up few cookbook 2. Run it on the machine 3. Try different drivers
57. Go pick up few cookbook There’re few steps that you can consider: 1. Search supermarket.chef.com 2. Search GitHub.com e.g. MySQL, Redis, Elsticsearch and others
58. A good start point You can find some cookbook is designed for Rails, and includes several required dependencies teohm/rackbox-cookbook is a good start point.
59. Try different drivers Test Kitchen provides a good plugin system: 1. Drivers - to spin up new instances 2. Bussers - to run test
60. Take EC2 as example From http://rubydoc.info/gems/kitchen-ec2 driver: name: ec2 aws_access_key_id: KAS... aws_secret_access_key: 3UK... aws_ssh_key_id: id_rsa-aws ssh_key: /path/to/id_rsa-aws security_group_ids: ["sg-1a2b3c4d"] region: us-east-1 availability_zone: us-east-1b subnet_id: subnet-6d6...
61. Random topics Some related things for Chef in production
62. Deployment Usually for Chef beginners, you will get confused with Chef & deployment tools like Capistrano. Don’t panic. 1. Try to avoid using Chef as deployment tool at first 2. Consider using Chef as a way to set up Capistrano friendly environment
63. How about docker? Just another way to create base instances. You still need to “set it up”. It’s can be used with Vagrant & Test Kitchen.
64. chef-solo and chef-server What we do today is named chef-solo, which indicates that it doesn’t require a central server. chef-server isn’t recommended for most people because: 1. The server is complex 2. One more SPoF 3. Many functionalities are duplicate of other tools
65. Chef in production To adopt Chef successfully in production, usually there’re 2 ways. 1. Use Chef-enabled environment like AWS OpsWorks or Engine Yard 2. Use Chef to provision images for later usage
66. AWS OpsWorks It’s a free platform provided by AWS built upon EC2 & related products. 1. It provides a builtin set of cookbook for Rails / PHP / Node.js 2. It has API to run Chef programmatically 3. Deploy vis GitHub
67. Thank you! Your feedback is highly appreciated