Getting Started with Chef for Rails

Presentation given at the Northern Virginia Ruby Users Group.

Overview of Chef
Building a project in Chef
Provision a Server for Your Rails App
Additional resources

Transcript

1. The Joy of Cooking: Whip up a Rails Environment with Chef Nathen Harvey @nathenharvey Northern Virginia Ruby Users Group
2. Nathen • Technical Community Manager, Opscode • Co-organizer • DevOpsDC • MongoDC • Formerly - Web Operations at CustomInk
3. Agenda • Overview of Chef • Building a project in Chef • Provision a Server for Your Rails App • Additional resources • Obligatory Shameless Plugs • Please Ask Questions Along the Way
4. Overview of Chef What is this thing again?
5. Evolution of Configuration Management • Just build it • Keep notes in server.txt • Move notes to the wiki • Custom scripts (in scm?!) • Configuration Management framework
6. When should I use a CM Framework? • After you outgrow Heroku
7. Chef
8. Applications http://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/
9. http://www.flickr.com/photos/sbh/462754460/ Infrastructure
10. Collection of Resources • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configurations • Stuff http://www.flickr.com/photos/stevekeys/3123167585/
11. Acting in concert http://www.flickr.com/photos/glowjangles/4081048126/
12. http://www.flickr.com/photos/28309157@N08/3743455858/ To provide a Service
13. http://www.flickr.com/photos/16339684@N00/2681435235/ And it evolves
14. Application See Node
15. Application Application Database See Nodes
16. Application App Databases See Nodes Grow
17. App Servers App Databases See Nodes Grow
18. App LB App Servers App Databases See Nodes Grow
19. App LBs App Servers App Databases See Nodes Grow
20. App LBs App Servers App DB Cache App DBs See Nodes Grow
21. App LBs App Servers App DB Cache App DBs Stitched together with configs
22. App LB App Servers App DB Cache App DBs Floating IP? Your Infrastructure is a snow flake
23. App LBs App Servers NoSQL DB slaves Cache DB Cache DBs Complexity increases quickly
24. Complexity increases very quickly DC1 DC3 DC2
25. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
26. Rails App Memcache Postgres Slaves Postgres Master Typical Boring Infrastructure Nagios Graphite
27. Golden Images are not the answer • Gold is heavy • Hard to transport • Hard to mold • Easy to lose configuration detail http://www.flickr.com/photos/garysoup/2977173063/
28. Rails App Memcache Postgres Slaves Postgres Master New Compliance Mandate Nagios Graphite • Move SSH off port 22 • Lets put it on 2022
29. Rails App Memcache Postgres Slaves Postgres Master 6 Golden Image Updates Nagios Graphite • edit /etc/ssh/sshd_config 1 2 3 4 5 6
30. Rails App Memcache Postgres Slaves Postgres Master 12 Instance Replacements Nagios Graphite • Delete, launch 1 2 3 4 5 6 7 8 9 10 11 12 • Repeat • Typically manually
31. Done in Maintenance Windows • Don’t break anything! • Bob just got fired =( 5 Rails App Memcache Postgres Slaves Postgres Master Nagios Graphite 1 2 4 5 6 7 8 9 10 11 12 3
32. Rails App Memcache Postgres Slaves Postgres Master Different IP Addresses? Nagios Graphite • Invalid configs!
33. http://www.flickr.com/photos/francoforeshock/5716969942/ Configuration Desperation
34. Chef Solves This Problem • But you already guessed that, didn’t you?
35. • Generate configurations directly on nodes • Reduce management complexity • Version control the programs http://www.flickr.com/photos/ssoosay/5126146763/ Programs!
36. Declarative Interface to Resources • Define policy • Say what, not how • Pull not Push http://www.flickr.com/photos/bixentro/2591838509/
37. Chef is Infrastructure as Code http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources.
38. package "ntp" do action :install end service "ntpd" do action [:enable,:start] end template "/etc/ntpd.conf" do source "ntpd.conf.erb" owner "root" group "root" mode 0644 action :create variables(:time_server => “time.example.com”) notifies :restart, “service[ntpd]” end That looks like this
39. Rails App Memcache Postgres Slaves Postgres Master So when this Nagios Graphite
40. Rails App Memcache Postgres Slaves Postgres Master Nagios Graphite Becomes this
41. Rails App Memcache Postgres Slaves Postgres Master Nagios Graphite This can happen automatically
42. Nagios Graphite Count the resources Rails App Memcache Postgres Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • Postgres firewall • Postgres authZ config • 12+ resource changes for 1 node addition
43. Building a Project in Chef OK, so how do I build something with Chef?
44. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec
45. Resources package "tmux" do action :install end directory "/u/apps/awesome" do owner "apache" group "apache" action :create recursive true end
46. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec • Write recipes
47. Recipe include_recipe "app_user" app_name = node["app_name"] app_user = node["app_user"] app_group = node["app_group"] %w(releases shared).each do |dir| directory "/u/apps/#{app_name}/#{dir}" do mode "0755" owner app_user group app_group recursive true end end
48. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec • Write recipes • Package recipes in cookbooks
49. Cookbooks |-- ldirectord | |-- README.md | |-- attributes | | `-- default.rb | |-- metadata.rb | |-- recipes | | `-- default.rb | `-- templates | `-- default | `-- site.cf.erb
50. Cookbooks |-- monit | |-- README.md | |-- attributes | | `-- default.rb | |-- files | | `-- ubuntu | | `-- monit.default | |-- metadata.rb | |-- recipes | | `-- default.rb | `-- templates | `-- default | `-- monitrc.erb
51. Recipes and Cookbooks • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity http://www.flickr.com/photos/shutterhacks/4474421855/
52. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec • Write recipes • Package recipes in cookbooks • Apply recipes to nodes
53. Nodes • Representation of a host • runs the Chef client • has attributes • has a list of recipes to be applied
54. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec • Write recipes • Package recipes in cookbooks • Apply recipes to nodes • Group things into roles
55. Roles • mechanism for easily composing sets of functionality • have attributes and a list of recipes to be applied
56. Run Lists Server Server Server Server chef-server API chef-client recipe[rails_app] node rails_app default.rb
57. Run Lists Server Server Server Server chef-server API chef-client “role[base]”, “role[my_awesome_app]” node ntp default.rb sudo default.rb rails_app default.rb
58. Server Server Server Server chef-server API chef-client “role[webserver]” node ntp client.rb openssh server.rb apache default.rb rails_app default.rb Roles chef-client “role[database]” node ntp client.rb openssh server.rb postgresql server.rb
59. Roles name "base" description "Base of all nodes" default_attributes( "newrelic" => { "license_key" => "cbb1f5..." } ) run_list( "recipe[base_config]", "recipe[users]", "recipe[groups]", "recipe[sudo]" )
60. Building a Chef Project • First, come up with your policy / specification • Abstract the resources in your spec • Write recipes • Package recipes in cookbooks • Apply recipes to nodes • Group things into roles
61. Provision Rails with Chef Let’s do this!
62. Landscape of Chef-managed Infrastructure
63. Rails Application • Apache • Passenger • PostgreSQL • Rails Application
64. © Opscode, 2011 – Confidential – DO NOT DISTRIBUTE Chef Provides a Model for Reuse That Works 64 700+ Cookbooks “Yesterday we started open sourcing some of our Opscode Chef work created at bestbuy.com; bit.ly/ yDV9Hl #Splunk #opschef #expectmor”
65. Knife • Chef’s secret weapon • Command-line interface for interacting with • Chef Server API • Local Chef Repository • Community Site • and more...
66. Download Community Cookbooks $ knife cookbook site download passenger_apache2 $ knife cookbook site download mysql
67. Create a Cookbook $ knife cookbook create rails_app ** Creating cookbook rails_app ** Creating README for cookbook: rails_app ** Creating CHANGELOG for cookbook: rails_app ** Creating metadata for cookbook: rails_app
68. Write Our Rails App’s Recipes • Application Recipe • Database Recipe
69. Application Recipe • Set-up some directories %w(releases shared shared/system shared/pids shared/logs shared/config).each do |dir| directory "#{deploy_to}/#{app_name}/#{dir}" do action :create owner app_user group app_group mode "0664" recursive true end end
70. Application Recipe • Configure Apache / Passenger web_app app_name do docroot "#{deploy_to}/current/public" server_name "#{app_name}.#{node["domain"]}" server_aliases [ app_name, "localhost", node["hostname"] ] rails_env "production" end
71. Attributes
72. node[“domain”] • cookbook • default[“domain”] = “localhost” • environments • test •“domain” : “test.awesomeco.com” • production •“domain” : “awesomeco.com"
73. node[“hostname”] • The node knows it’s own hostname • Not something we need to set in our code
74. "memory": { "swap": { "cached": "0kB", "total": "4128760kB", "free": "4128760kB" }, "total": "2055676kB", "free": "1646524kB", "buffers": "35032kB", "cached": "210276kB", "active": "125336kB", "inactive": "142884kB", "dirty": "8kB", "writeback": "0kB", "anon_pages": "22976kB", "mapped": "8416kB", "slab": "121512kB", "slab_reclaimable": "41148kB", "slab_unreclaim": "80364kB", "page_tables": "1784kB", "nfs_unstable": "0kB", "bounce": "0kB", "commit_limit": "5156596kB", "committed_as": "74980kB", "vmalloc_total": "34359738367kB", "vmalloc_used": "274512kB", "vmalloc_chunk": "34359449936kB" }, Ohai! "block_device": { "ram0": { "size": "32768", "removable": "0" }, "ram1": { "size": "32768", "removable": "0" }, "ram2": { "size": "32768", "removable": "0" }, "hostname": "server-1", "fqdn": "server-1.example.com", "domain": "example.com", "network": { "interfaces": { "eth0": { "type": "eth", "number": "0", "encapsulation": "Ethernet", "addresses": { "00:0C:29:43:26:C5": { "family": "lladdr" }, "192.168.177.138": { "family": "inet", "broadcast": "192.168.177.255", "netmask": "255.255.255.0" }, "fe80::20c:29ff:fe43:26c5": { "family": "inet6", "prefixlen": "64", "scope": "Link" } },
75. Attributes
76. Database Recipe mysql_connection_info = { :host => "localhost", :username => 'root', :password => node['mysql']['server_root_password'] } mysql_database app_name do connection mysql_connection_info action :create end
77. Database Recipe mysql_database_user node["database"]["user"] do connection mysql_connection_info password node["database"]["pw"] database_name node["database"]["name"] host "%" action :grant end
78. Upload Cookbooks to Chef Server
79. Create Roles • mechanism for easily composing sets of functionality • have attributes and a list of recipes to be applied • Define as Ruby or JSON • base-ubuntu • rails-web • rails-database
80. Role as Ruby name "base-ubuntu" description "all Ubuntu servers" run_list( "recipe[apt]", "recipe[build-essential]" )
81. Role as JSON { "name": "rails-web", "description": "Web Server for our Rails App", "default_attributes": { }, "run_list": [ "recipe[rails_app::web]" ] }
82. Run Chef Client Server Server Server Server chef-server API chef-client “role[base-ubuntu]”, “role[rails-web]” node apt default.rb build-essential default.rb rails-app web.rb
83. Running the Chef Client • Automatically • cron • daemon
84. knife ssh $ knife ssh "roles:rails-web" "sudo chef- client"
85. knife ssh $ knife ssh "roles:rails-web" "sudo chef-client"
86. Review • Server provisioned and communicating with Chef Server API • Apache & Passenger installed with a default configuration • MySQL installed and running
87. Deploying Your App • Pull - Use Chef • Push - Use Capistrano
88. Capistrano role :web, "web01","web02","web03"
89. Capistrano + Chef webservers = [] web_query = Chef::Search::Query.new web_query.search(:node, 'role:dcrug_web') do |h| webservers << h["fqdn"] end role :web, *webservers
90. But wait, there’s more! • Encrypted Data Bags • Environments • Lightweight Resources and Providers • Exception and Report Handlers
91. Additional Resources Where to go next
92. Want More? • community.opscode.com • wiki.opscode.com • docs.opscode.com • #chef on irc.freenode.net
93. Food Fight Show • The podcast where DevOps Chefs Do Battle • foodfightshow.org
94. Get Involved Locally! • meetup.com/DevOpsDC • meetup.com/Arlington-Ruby • Feb 13 • zero-to-Rails Application
95. Learning Chef Series
96. Learn More • Use discount code ‘MEETUP’ to save 10%! • opscode.eventbrite.com
97. #ChefConf 2013
98. Find Me • @nathenharvey • http://nathenharvey.com • nharvey@opscode.com • Thank You! • What Questions Do You Have?