Chef or how to make computers do the work for us

My KRUG (Kraków Ruby Users Group) presentation about automating boring tasks with Opscode's Chef.

Transcript

1. C H E F or how to make computers do the work for us Marcin Kulik, Lunar Logic Polska KRUG 2011/11/08
2. Everyday we're dealing with mechanical, repetitive tasks... we can automate.
3. What is Chef?
4. Automation tool written in Ruby
5. DSL
6. Created by Opscode
7. "Chef is an open source systems integration framework built to bring the benefits of configuration management to your entire infrastructure." "You write source code to describe how you want each part of your infrastructure to be built, then apply those descriptions to your servers." "The result is a fully automated infrastructure: when a new server comes on line, the only thing you have to do is tell Chef what role it should play in your architecture."
8. Why do you need it?
9. Economics + Efficiency + Scalability
10. Terms
11. Node remote server, local machine...
12. Role web server, database server, ruby dev workstation...
13. Cookbook mysql, ssh-access, dotfiles...
14. Recipe install mysql-server, create database, add user...
15. Resource file, dir, user, package, service, gem, virtual host...
16. Run list list of recipes to run in order
17. { "run_list": [ "recipe[mysql]", "recipe[git]", "recipe[ruby19]" ] }
18. Cookbook structure |-- config | |-- node.json | `-- solo.rb |-- cookbooks | |-- book1 | | |-- attributes | | |-- files | | |-- metadata.rb | | |-- recipes | | | |-- default.rb | | | `-- source.rb | | `-- templates | |-- book2 | | |-- attributes | | | `-- default.rb | | |-- files | | |-- recipes | | | `-- default.rb | | `-- templates
19. | | `-- templates | | `-- default | | `-- authorized_keys.erb | |-- book3 | | |-- attributes | | |-- files | | | `-- default | | | `-- secret-key | | |-- recipes | | | `-- default.rb | | `-- templates |-- config | |-- node.json | `-- solo.rb |-- cookbooks | |-- book1 | | |-- attributes | | |-- files | | |-- metadata.rb | | |-- recipes | | | |-- default.rb | | | `-- libs.rb | | `-- templates Installation
20. $ gem install chef
21. Modes of operation
22. Cookbooks stored in central repository (free cookbooks hosting by Opscode: https://manage.opscode.com/)
23. $ sudo chef-client
24. Cookbooks stored on the node
25. $ sudo chef-solo -c /path/to/cfg.rb -j /path/to/node-data.json
26. Use cases
27. Configure new machine (in the cloud with Knife) Amazon EC2, Engine Yard, Linode, BrightBox...
28. Manage config of existing company servers Client demo apps (directory, vhost, god config), developers' ssh keys...
29. Bootstrap workstation! rvm + ruby 1.9, git, mysql, vim/emacs...
30. Enough with theory!
31. Lunar Station https://github.com/LunarLogicPolska/lunar-station
32. Lunar Station is a set of Chef cookbooks and a bash script (???) for bootstrapping developers machines at Lunar Logic Polska.
33. You need ruby to run Chef
34. (We assume) you use RVM No need for system ruby for ruby devs nowadays
35. bootstrap.sh
36. detects platform (Ubuntu, Fedora, OSX) installs compilers and other RVM dependencies installs RVM & ruby 1.9 & chef gem downloads latest Lunar Station cookbooks runs chef-solo
37. $ curl -skL http://bit.ly/lunar-station | bash Initializing Lunar Workstation... >> Fedora Linux detected. >> Checking for RVM... >> Fetching latest version of Lunar Station cookbooks... >> Starting chef-solo run... [Mon, 07 Nov 2011 22:19:54 +0100] INFO: *** Chef 0.10.4 *** [Mon, 07 Nov 2011 22:19:54 +0100] INFO: Setting the run_list to ...
38. Nodes
39. # linux-rubydev.json { "run_list": [ "role[rubydev]" ] }
40. # osx-rubydev.json { "run_list": [ "role[osx]", "role[rubydev]" ] }
41. Roles
42. # base.rb run_list 'recipe[repos]', 'recipe[curl]', 'recipe[wget]', 'recipe[git]', 'recipe[libxml2]', 'recipe[ack]', 'recipe[vim]', 'recipe[ctags]', 'recipe[skype]', 'recipe[firefox]' , 'recipe[google-chrome]'
43. # rubydev.rb run_list 'role[base]', 'recipe[mysql]'
44. # osx.rb run_list "recipe[homebrew]"
45. Cookbooks
46. repos cookbook
47. # cookbooks/repos/recipes/default.rb case node[:platform] when 'fedora' path = "/tmp/rpmfusion-free-release-stable.noarch.rpm" bash "download rpmfusion free package" do code "wget http://download1.rpmfusion.org/.../" + "rpmfusion-free-release-stable.noarch.rpm -O #{path}" not_if { File.exist?(path) } end package "rpmfusion-free-release-stable" do source path options "--nogpgcheck" end when 'ubuntu' ... end
48. end # cookbooks/repos/recipes/default.rb case node[:platform] when 'fedora' ... when 'ubuntu' bash "enable multiverse repo" do code "head -n 1 /etc/apt/sources.list | " + "sed 's/main universe/multiverse/' " + ">> /etc/apt/sources.list" not_if "egrep '^deb.+multiverse' /etc/apt/sources.list" end end
49. vim cookbook
50. # cookbooks/vim/recipes/default.rb case node[:platform] when "ubuntu" package "vim" package "vim-gnome" when "fedora" package "vim-enhanced" package "vim-X11" when 'mac_os_x' package "macvim" end
51. skype cookbook
52. # cookbooks/skype/recipes/default.rb case node[:platform] when 'ubuntu' include_recipe 'init::ubuntu' # for partner repo package 'skype' when 'mac_os_x' dmg_package "Skype" do source "http://www.skype.com/go/getskype-macosx.dmg" action :install end when 'fedora' ... end
53. Lunar Kitchen
54. Source of LLP servers configuration data and a set of Chef cookbooks
55. chef-solo invoked on remote machines no chef server
56. Each server we configure has its corresponding node configuration file in nodes/ directory of kitchen project that specifies run_list and few other settings
57. # nodes/deneb.json { "run_list": [ "recipe[ssh_access]" ], "ssh_access": [ "marcin.kulik", "anna.lesniak", ...], "opened_ports": { "tcp": [80, 443, 22, 8080], "udp": [] }, ...
58. How do we run chef-solo on remote machine?
59. Capistrano!
60. # See the list of configured servers: $ cap -T # Make the changes happen on the server: $ cap configure:deneb
61. How does Capfile look like?
62. set :user, 'chef' NODE_LIST = Dir["nodes/*.json"].map do |nodefile| File.basename(nodefile, '.json') end NODE_LIST.each do |node| role node.to_sym, node end NODE_CONFIG = <<-EOS file_cache_path '/tmp/chef-solo' cookbook_path '/tmp/chef-solo/cookbooks' role_path '/tmp/chef-solo/roles' EOS ...
63. ... namespace :configure do NODE_LIST.each do |node| desc "Configure #{node}" task node.to_sym, :roles => node.to_sym do run "if [ ! -e /tmp/chef-solo ]; then mkdir /tmp/chef-sol upload("cookbooks", "/tmp/chef-solo/", :via => :scp, :rec upload("roles", "/tmp/chef-solo/", :via => :scp, :recursi upload("nodes/#{node}.json", "/tmp/chef-solo/node.json", put(NODE_CONFIG, "/tmp/chef-solo/solo.rb") run "rvmsudo chef-solo " + "-c /tmp/chef-solo/solo.rb " + "-j /tmp/chef-solo/node.json" end end end
64. SSH access
65. ├── Capfile ├── config ├── cookbooks ├── nodes ├── README.md ├── roles └── ssh_keys ├── anna.lesniak ├── artur.bilski ├── ... └── marcin.kulik
66. # cookbooks/access/recipes/default.rb username = 'dev' ssh_keys = node[:ssh_access].map do |f| File.read("/tmp/chef-solo/ssh_keys/#{f}") end template "/home/#{username}/.ssh/authorized_keys" do source "authorized_keys.erb" owner username group 'users' mode "0600" variables :ssh_keys => ssh_keys end
67. # cookbooks/access/templates/authorized_keys.erb # Generated by Chef, do not edit! <%= @ssh_keys.join("\n") %>
68. Tips
69. Learn step by step EC2 + Chef + Knife + Opscode... = Fuuuuuuuuuuuuuuuuuuuuu
70. Start with chef-solo
71. Run on local machine Easy to troubleshoot problems
72. Use Vagrant http://vagrantup.com/ Great for testing cookbooks - doesn't pollute your system
73. Q?
74. Thanks! marcin.kulik@llp.pl | @sickill | https://github.com/sickill