The Joy of Cooking: Deploying Drupal with Chef

An introduction to Chef and information on deploying Drupal with Chef. This presentation was originally given at DrupalCon 2013.

Transcript

2. Nathen •Technical Community Manager at Opscode •Co-host of the Food Fight Show Podcast •Meetup Organizer
3. Opscode
4. Chef is an automation platform for developers & systems engineers to continuously define, build, and manage infrastructure. CHEF USES: Recipes and Cookbooks that describe Infrastructure as Code. Chef enables people to easily build & manage complex & dynamic applications at massive scale • New model for describing infrastructure that promotes reuse • Programmatically provision and configure • Reconstruct business from code repository, data backup, and bare metal resources “ ” Chef
5. http://www.flickr.com/photos/steffenz/337700069/ http://www.flickr.com/photos/kky/704056791/ Applications
6. Infrastructure http://www.flickr.com/photos/sbh/462754460/
7. Collection of Resources http://www.flickr.com/photos/philliecasablanca/3354734116/ • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Tasks • Packages • Software • Services • Configuration • Other Stuff
8. Acting in Concert http://www.flickr.com/photos/glowjangles/4081048126/
9. To Provide a Service http://www.flickr.com/photos/28309157@N08/3743455858/
10. And it Evolves http://www.flickr.com/photos/16339684@N00/2681435235/
11. Application Server See Node
12. Application Server Application Database See Nodes
13. Application Server Application Databases See Nodes Grow
14. Application Servers Application Databases See Nodes Grow
15. Application Servers Application Databases Load Balancer See Nodes Grow
16. Application Servers Application Databases Load Balancers See Nodes Grow
17. Application Servers Application Database Cache Load Balancers Application Databases See Nodes Grow
18. Application Servers Application Database Cache Load Balancers Application Databases Tied Together with Configuration
19. Application Servers Application Database Cache Load Balancers Floating IP? Application Databases Infrastructure is a Snowflake
20. Load Balancers Application Servers NoSQL Database Slaves ApplicationCache Database Cache Database Evolving Complexity
21. Configuration Management http://www.flickr.com/photos/philliecasablanca/3354734116/
22. New Requirements •A new developer joined the team, get her set-up with the application •Add New Relic monitoring to the application •Add a new module to the development site •Do not store the unencrypted admin password in the git repository
23. Configuration Desperation http://www.flickr.com/photos/francoforeshock/5716969942/
24. • But you already guessed that, didn’t you? Chef Solves This Problem
25. http://www.flickr.com/photos/louisb/4555295187/ • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. Chef is Infrastructure as Code
26. http://www.flickr.com/photos/ssoosay/5126146763/ • Chef generates configurations directly on nodes from their run list • Reduce management complexity through abstraction • Store the configuration of your programs in version control Programs
27. •Define Policy •Say what, not how •Pull not Push http://www.flickr.com/photos/bixentro/2591838509/ Declarative Interface to Resources
28. That Looks Like This package "apache2" template "/etc/apache2/apache2.conf" do soucre "apache2.conf.erb" owner "root" group "root" mode "0644" variables(:allow_override => "All") notifies :reload, "service[apache2]" end service "apache2" do action [:enable,:start] supports :reload => true end
29. • Recipes are collections of Resources • Cookbooks contain recipes, templates, files, custom resources, etc • Code re-use and modularity • Hundreds already on Community.opscode.com http://www.flickr.com/photos/shutterhacks/4474421855/ Recipes and Cookbooks
30. http://www.flickr.com/photos/kathycsus/2686772625 • IP addresses • Hostnames • FQDNs • Search for nodes with Roles • Find configuration data Search
31. pool_members = search("node","role:awesome_site) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode "0644" variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end Pass Results To Templates
32. # Set up application listeners here. listen application 0.0.0.0:80 balance roundrobin <% @pool_members.each do |member| -%> server <%= member[:hostname] %> <%= member[:ipaddress] %>: weight 1 maxconn 1 check <% end -%> <% if node["haproxy"]["enable_admin"] -%> listen admin 0.0.0.0:22002 mode http stats uri / <% end -%> Pass Results To Templates
33. Drupal App Memcache MySQL Slaves MySQL Master Nagios Graphite So when this...
34. Drupal App Memcache MySQL Slaves MySQL Master Nagios Graphite ...becomes this...
35. Drupal App Memcache MySQL Slaves MySQL Master Nagios Graphite ...this can happen automatically
36. Nagios Graphite Drupal App Memcache MySQL Slaves • Load balancer config • Nagios host ping • Nagios host ssh • Nagios host HTTP • Nagios host app health • Graphite CPU • Graphite Memory • Graphite Disk • Graphite SNMP • Memcache firewall • MySQL firewall • 11+ resource changes for 1 node addition Count the Resources
37. Deploying Drupal with Chef
38. Chef Components
39. Our Workflow •Build Drupal locally •Deploy Drupal on EC2 •Iterate
40. Local Development •Git repository •Virtual Machine •Chef •knife •Chef Server
41. Git Repository $ mkdir -p ~/drupalcon/chef-repo $ cd ~/drupalcon/chef-repo $ git init
42. Building Drupal Locally •Deploy in a local virtual machine managed by Vagrant, provisioned with Chef
43. Vagrantfile Vagrant.configure("2") do |config| config.vm.hostname = "drupalcon-dev" config.vm.box = "opscode-ubuntu-12.04" config.vm.network :private_network, ip: "33.33.33.10" config.vm.network :forwarded_port, guest: 80, host: 8080 end
44. Chef
45. Chef Server
46. Register VM with Chef Vagrant.configure("2") do |config| ... config.vm.provision :chef_client do |chef| chef.chef_server_url = "https://api.opscode.com/organizations/nhdrupalcon" chef.validation_key_path = ".chef/nhdrupalcon-validator.pem" chef.validation_client_name = "nhdrupalcon-validator" end end $ vagrant provision
47. Chef Components
48. Cookbooks •Download from community.opscode.com •Write our own •Upload to the Chef Server
49. Community Site •950+ Cookbooks •Codify tribal knowledge •Reference
50. Write our own Cookbook $ knife cookbook create awesome_site ** Creating cookbook awesome_site ** Creating README for cookbook: awesome_site ** Creating CHANGELOG for cookbook: awesome_site ** Creating metadata for cookbook: awesome_site
51. Write our own Cookbook include_recipe "drupal" web_app "drupal" do template "drupal.conf.erb" docroot node['drupal']['dir'] server_name server_fqdn server_aliases node['fqdn'] end
52. Data •Separate data from policy •Policy: Site has a document root •Data: /var/www/drupal •Policy: Apache has a Server Alias •Data: Fully-qualified domain name of the server web_app "drupal" do template "drupal.conf.erb" docroot node['drupal']['dir'] server_name server_fqdn server_aliases node['fqdn'] end
53. Where does data come from? •Attributes •Data Bags •Encrypted Data Bags •Search
54. Attributes •Specific details about a node •Defined by: •The state of the node •Cookbooks •Roles •Environments
55. Ohai "languages": { "ruby": { }, "perl": { "version": "5.14.2", "archname": "x86_64-linux-gnu-thread-multi" }, "python": { "version": "2.7.3", "builddate": "Aug 1 2012, 05:14:39" }, "php": { "version": "5.3.10-1ubuntu3.6", "builddate": "(cli) (built: Mar" } }, "kernel": { "name": "Linux", "release": "3.2.0-32-virtual", "version": "#51-Ubuntu SMP Wed Sep 26 21:53:42 UTC 2012", "machine": "x86_64", "modules": { "isofs": { "size": "40257", "refcount": "0" }, "acpiphp": { "size": "24231", "refcount": "0" } }, "os": "GNU/Linux" }, "os": "linux", "os_version": "3.2.0-32-virtual", "ohai_time": 1369328621.3456137, "network": { "interfaces": { "lo": { "mtu": "16436", "flags": [ "LOOPBACK", "UP", "LOWER_UP" ], "encapsulation": "Loopback", "addresses": { "127.0.0.1": { "family": "inet", "prefixlen": "8", "netmask": "255.0.0.0", "scope": "Node" }, "::1": { "family": "inet6", "prefixlen": "128", "scope": "Node" } }, "state": "unknown" }, "eth0": { "type": "eth", "number": "0", "mtu": "1500",
56. Attributes •Cookbooks, Roles, and Environments can also set attribute data •Extremely flexible mechanism for configuration •Cookbook: •Environment: default['drupal']['modules'] = ['views', 'webform'] default_attributes({ "drupal" => {"modules" => ["views","webform","token"]}})
57. Data Bags •Global Data •Stored in JSON •Accessible from the Chef Server •Can be encrypted { "id": "nharvey", "groups": ["sysadmin"], "uid": 2001, "shell": "/bin/bash", "comment": "Nathen Harvey <nharvey@opscode.com>", "nagios": { "email": "you@example.com" } }
58. Search pool_members = search("node","awesome_site) template "/etc/haproxy/haproxy.cfg" do source "haproxy-app_lb.cfg.erb" owner "root" group "root" mode "0644" variables :pool_members => pool_members.uniq notifies :restart, "service[haproxy]" end
59. Upload our Cookbooks •We’re using •Cookbooks from the community site •A cookbook we wrote •Upload the cookbooks using knife $ knife cookbook upload -a
60. Deploy Drupal
61. Run List •Ordered list of roles or recipes to be run on a node •The run list represents the policy to be applied •chef-client •Executes on the node on a regular basis •Receives the run list from the Chef Server •Brings the node in-line with the policy
62. Run List Server Server Server Server chef-server API chef-client “role[webserver]” node ntp client.rb openssh server.rb apache default.rb drupal default.rb chef-client “role[database]” node ntp client.rb openssh server.rb mysql server.rb
63. Run chef-client config.vm.provision :chef_client do |chef| chef.chef_server_url = "https://api.opscode.com/organizations/nhdrupalcon" chef.validation_key_path = ".chef/nhdrupalcon-validator.pem" chef.validation_client_name = "nhdrupalcon-validator" chef.add_recipe "awesome_site" end $ vagrant provision
64. Next Steps •Share the Vagrantfile with your teammates •Develop to a releasable state •Deploy to production
65. Production Deploy •We’ll use Amazon’s EC2 for our initial deploy •...but we could deploy to any server •Using the same cookbooks as our development environment, deploy to EC2 $ knife ec2 server create -r "recipe[awesome_site]" -f m1.medium -I ami-641c8e0d -N ec2_drupal
66. Current State •Local virtual machine for development running Drupal •“Production” instance on EC2
67. Next Steps •Move Database to a different node •Add a Load Balancer and additional application servers •Add monitoring •Add log aggregation •etc.
68. More about Chef
69. Community
70. Open Source •Apache 2 Software License •Continually growing number of contributors! •1400+ individuals, 200+ companies •Development repositories: •github.com/opscode •github.com/opscode-cookbooks •Regular code reviews via Google+ Hangout
71. Get Started on Your Own •http://learnchef.com •http://docs.opscode.com •http://lists.opscode.com •#chef on Freenode IRC
72. Wrap-up & Special Offer
73. Infrastructure as Code •Repeatability •Automation •Agility •Scalability •Reassurance •Disaster Recovery •Happiness
74. Thanks! •Promet Source •Will Milton •github.com/promet/drupal_cookbook •curl -L bit.ly/10McZSZ | bash; vagrant up •Marius Ducea •community.opscode.com/cookbooks/drupal
75. Chef Training •Chef Introductory Workshop •Tomorrow in Corvallis •FREE to all DrupalCon Attendees •http://bit.ly/learnchef-drupalcon
76. Thank You! •What Questions Do You Have? •@nathenharvey •Free Training Tomorrow (register by 9PM) •bit.ly/learnchef-drupalcon •Feedback on this session: portland2013.drupal.org/node/2963